Switch Net 4 neural network

[dafhi]

i assumed 'width' meant array size. i only glanced at the blog b/c i read slowly.

i highly recommend GreenInk's mutator concept for simplicity

the idea is to save a portion of the network which will be changed

in .mutate(),
pIdx() stores location
previous() stores value

[edit] if you want, you could post here and i'll do my best to add a mutator. the concept is simple enough. i'll also add (depending on how well i grasp your code) synthetic data generator of some sort.

[Luxan]

After referring to the Mathwork site, and my local installation
of GNU Octave, I determined that my version of the Fast Walsh Hadamard
Transform and that presented by the originator of this discussion,
are correct.

The following code verifies this ; I haven't included my routine
WHT() within this.

Examining the use of the xwht() type function, in the original code,
the expansion of parameters appears to happen within the switch network.

A cost function is easy to write, adjusting the weights to minimise
this; is more involved.

Code: Select all

/'

  wht_wn4.bas
  sciwiseg@gmail.com 

  fast Walsh Hadamard Transform and possible wnet4.
  
  https://au.mathworks.com/help/signal/ug/walshhadamard-transform.html
  
       0 1 2 3 4 5  6 7 
  x = [4 2 2 0 0 2 -2 0]
  y = fwht(x)    "sequency"  format
  y = [1 1 0 1 0 0 1 0]
  
  x1 = ifwht(y)
  x1 = [4 2 2 0 0 2 -2 0]



  Using GNU Octave , with signal package loaded and this script :
  
  x = [4 2 2 0 0 2 -2 0]
  y = fwht(x)
  y = fwht(x,8)
  y = fwht(x,8,"hadamard")
  
  returns :
  
  x =    4   2   2   0   0   2  -2   0

  y =    1   1   0   1   0   0   1   0

  y =    1   1   0   1   0   0   1   0

  y =    1   0   1   0   1   1   0   0
      

'/
declare sub flipper(A() as single)
declare sub genrand1(g1() as single)
declare sub genrand2(g2() as single)
declare sub prt_1ds(g1() as single)
declare sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
'
declare sub xwht( vec() as single, hs_pow2_shl as long = 0 )
'
dim as long L
dim as integer n,m,i
dim as single x

L=3
n=2^L
m=2^(L-2)

dim vec(0 to n-1) as single

'
' ......................................................................
'
restore xdat
for i=0 to n-1
   read vec(i) 
next i
'
print " Input "
for i=0 to n-1
   print vec(i);" , ";
next i
print
'
xwht(vec() , 0 )
'
print " xwht(x)  output "
for i=0 to n-1
   print vec(i)/n;" , ";
next i
print
'
print " Expected output "
restore ydat
for i=0 to n-1
   read x
   print x;" , ";
next i
print
print
'
' ......................................................................
'
dim Wn(0 to 3,0 to m-1) as single 
dim Wp(0 to 3,0 to m-1) as single 
'
'
'
genrand2(Wp())
genrand2(Wn())
genrand1(vec())
flipper(vec())
'
xwht(vec() , 0 )
swnet4(vec(),Wp(),Wn(),n,m)
xwht(vec() , 0 )
'
'
print " xwht(), swnet4(), xwht()  output "
prt_1ds(vec())
'
'
end
'
'=======================================================================
'
'
  '' Fast Walsh Hadamard Transform
  '' This routine is from another author .
  '' As illustrated  
  '' Input vec() , output vec()
  sub xwht( vec() as single, hs_pow2_shl as long = 0 ) '' 1 or 2 for Partial Fast W.H.T.
    static as long n 
    n = ubound(vec)+1
    var hs = 1 shl hs_pow2_shl
    'print " xwht, n , hs ";n;" , ";hs
    while (hs < n) 
      var i = 0
      while (i < n) 
        var j = i + hs
        while (i < j) 
          var a = vec(i)
          var b = vec(i + hs)
          vec(i) = a + b
          vec(i + hs) = a - b
          i += 1
        wend
        i += hs
      wend
      hs += hs
    wend
  end sub
' _________________________________________________________________
'
'
sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
'
'   2 way switches and net composed of Wn and Wp adjustable weights 
'
static as integer i,j,k,p
static as single z,s
static as single a(0 to 4)
static as single y(0 to n-1)
'
            k=0
            for p=0 to m-1 
                j=0
                for i=k to k+3  
                    z=x(i)
                    if z<0 then s=z*Wn(j,p) else s=z*Wp(j,p) end if
                    a(j)=s
                    j=j+1
                next i
                     j=0
                for i=k to k+3 
                    y(i)=y(i) + a(j)
                    j=j+1
                next i
                     k=k+4
            next p  
'
            for i=0 to n-1
                x(i)=y(i)
            next i
'
'
end sub
'
' _________________________________________________________________
'
'
sub genrand1(g1() as single)
'
'   Generate signed random values for 1d array
'
static as integer i,n
static as single z
'
        n=ubound(g1)
        for i=0 to n-1 
             z=rnd
          if z<0.5 then  z=-z else z=z end if
             g1(i)=z
        next i
end sub
'
' _________________________________________________________________
'
sub genrand2(g2() as single)
'
'        Generate signed random values for 2d array
'
static as integer i, j, n, m
static as single z
'
n=ubound(g2,1)
m=ubound(g2,2)
'
  for j=0 to m 
    for i=0 to n 
        z=rnd
        if z < 0.5 then z=-z  end if
        g2(i,j)=z
    next i
  next j
'
'    
end sub
'
' _________________________________________________________________
'
sub flipper(A() as single)
'
'          Random sign flips, use with input .  
'
static as integer n,i
static as single s
'  
        n=ubound(A)
    for i=0 to n
         s=rnd
        if s<0.5 then s=-1 else s=1 end if
       A(i)=s*A(i)
    next i
'
'
end sub
' _________________________________________________________________
'
'
sub prt_1ds(g1() as single)
'
'    print 1d, single precision, array .
'
static as integer n,i
'
'  
        n=ubound(g1)
    for i=0 to n
        print g1(i);" , ";
    next i
    print
'
end sub
'
' _________________________________________________________________
'
xdat:
data 4, 2, 2, 0, 0, 2, -2, 0
ydat:
data 1, 0, 1, 0, 1, 1,  0, 0

[Luxan]

This program illustrates a few well known procedures that are possible
before and after a transform, these aren't limited to a doubling of the
size of the vector.

The question is, did the originator use any of these.

Code: Select all

'
'  ft_pc.bas
'
'  General function transform procedures for discrete, and possibly
' continuous, data .
'
'  sciwiseg@gmail.com
'
'
screen 12
'line(0,0)-(639,479),11,b
window screen(-0.1,-0.1)-(100.1,100.1)
'view screen (0,0)-(639,479)
line(-0.1,-0.1)-(100.1,100.1),14,b

dim as integer i,j,n
dim as single x,x1,y,y1,a,b
'
n=8
a=5
'
for i=0 to n-1
    locate i+2,2
    print i
next i
'
for i=0 to n-1
    y=i*3.4 + 3
    x=a
    y1=y+3.4
    x1=x+5
    line(x,y)-(x1,y1),12,bf
    line(x,y)-(x1,y1),11,b
next i
'
locate 23,5
print "x()"
'
' ...........................................................
'
n=16
a=20
'
for i=0 to n-1
    locate i+2,13
    print i
next i
'
for i=0 to n-1
    y=i*3.4 + 3
    x=a
    y1=y+3.4
    x1=x+5
    if i<(n/2) then line(x,y)-(x1,y1),12,bf end if
    line(x,y)-(x1,y1),11,b
next i
'
locate 23,15
print "x(), zero extended"
'
' ......................................................................
'
n=8
a=60
'
for i=0 to n-1
    locate i+2,45
    print i
next i
'
for i=0 to n-1
    y=i*3.4 + 3
    x=a
    y1=y+3.4
    x1=x+5
    line(x,y)-(x1,y1),12,bf 
    line(x,y)-(x1,y1),11,b
next i
'
locate 23,47
print "  x()"
'
' ......................................................................
'
n=16
a=75
'
for i=0 to n-1
    locate i+2,56
    print i
next i
'
j=0
for i=0 to n-1
    y=i*3.4 + 3
    x=a
    y1=y+3.4
    x1=x+5
    if j mod 2 = 0 then line(x,y)-(x1,y1),12,bf end if
    line(x,y)-(x1,y1),11,b
    j=j+1
next i
'
locate 23,58
print "x(), zero insert"
'
' ......................................................................
'
sleep  6000
line(-0.1,-0.1)-(100.1,100.1),0,bf
line(-0.1,-0.1)-(100.1,100.1),14,b
'
n=8
a=5
'
for i=0 to n-1
    locate i+2,2
    print i
next i
'
for i=0 to n-1
    y=i*3.4 + 3
    x=a
    y1=y+3.4
    x1=x+5
    line(x,y)-(x1,y1),i+1,bf
    line(x,y)-(x1,y1),11,b
next i
'
locate 23,5
print "x()"
'
' ...........................................................
'
n=16
a=20
'
for i=0 to n-1
    locate i+2,13
    print i
next i
'

for i=0 to n-1
    y=i*3.4 + 3
    x=a
    y1=y+3.4
    x1=x+5
    
    line(x,y)-(x1,y1),int(i/2)+1,bf
    line(x,y)-(x1,y1),11,b
    
next i
'
locate 23,15
print "x(), replicate"
'
' ......................................................................
'


sleep

end

[Luxan]

A slight update.
It's slow going with the coding, so many unrelated notions
on my mind and too many late nights.

I haven't thoroughly read the articles.
I have found that microfiber cloth is effective at cleaning
my eye ware.

Code: Select all

'
'  swnet4f.bas
'                     Switchnet 4, preliminary 
'
'       sciwiseg@gmail.com
'

declare function round(in as double, places as ubyte = 2) as string
declare function max( a as double, b as double ) as double
declare function costL2(vec() as single, tar() as single ) as double
declare sub xwht( vec() as single, hs_pow2_shl as long = 0 )
'
declare sub genrand1(g1() as single)
declare sub genrand2(g2() as single)
declare sub flipper(A() as single)
declare sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
declare sub prt_1ds(g1() as single)

declare sub UWCopy(Av() as single, Bv() as single, m as integer)
declare sub mutate(Wn() as single, Wp() as single, m as integer)

declare function bp_ramp(x as single) as single
declare sub apply_activation(g1() as single)
'
'
dim as integer L,n,m,i,j,k
dim as double cost, cost2
'
L=3
n=2^L
m=2^(L-2) ' using 4 elements per bank .
print
print " L= ";L;" n= ";n;" m= ";m
print
dim as single x(0 to n-1)
dim as single y(0 to n-1)
dim Wn(0 to 3,0 to m-1) as single 
dim Wp(0 to 3,0 to m-1) as single 
'
dim Un(0 to 3,0 to m-1) as single 
dim Up(0 to 3,0 to m-1) as single 
'
'
'
genrand2(Wp())
genrand2(Wn())
genrand1(x())
'
for i=0 to n-1
   y(i)=x(i) ' [0,1]
next i
'  
flipper(x())
xwht( x() ,  0 )
swnet4(x(),Wp(),Wn(),n,m)
apply_activation(x())
xwht( x() ,  0 )
'
prt_1ds(x())
cost = costL2(x() , y())
print
print " Initial cost = "; cost
'
' .............................. repeats ...............................
'
for k=1 to 256
'
for i=0 to n-1
   x(i)=y(i)
next i
'                          copy, mutate; before wht & swnet4
UWCopy(Wp() , Up() , m )
UWCopy(Wn() , Un() , m )
mutate(Wn() , Wp(), m)
'
flipper(x())
xwht( x() ,  0 )
swnet4(x(),Wp(),Wn(),n,m)
apply_activation(x())
xwht( x() ,  0 )
cost2 = costL2(x() , y())
'
if cost2>cost then
   UWCopy(Up() , Wp() , m )
   UWCopy(Un() , Wn() , m )
 '  print " cost2 > cost "
else
 '  print k;" cost : "; cost2
   cost = cost2   
end if
'
next k
print " final   cost = ";cost
'
end
'
' ================================================================
'
sub genrand1(g1() as single)
'
'   Generate signed random values for 1d array
'
static as integer i,n
static as single z
'
        n=ubound(g1)
        for i=0 to n-1 
             z=rnd
          if z<0.5 then  z=-z else z=z end if
             g1(i)=z
        next i
end sub
'
' _________________________________________________________________
'
sub genrand2(g2() as single)
'
'        Generate signed random values for 2d array
'
static as integer i, j, n, m
static as single z
'
n=ubound(g2,1)
m=ubound(g2,2)
'
  for j=0 to m 
    for i=0 to n 
        z=rnd
        if z < 0.5 then z=-z  end if
        g2(i,j)=z
    next i
  next j
'
'    
end sub
'
' _________________________________________________________________
'
sub flipper(A() as single)
'
'          Random sign flips, use with input .  
'
static as integer n,i
static as single s
'  
        n=ubound(A)
    for i=0 to n
         s=rnd
        if s<0.5 then s=-1 else s=1 end if
       A(i)=s*A(i)
    next i
'
'
end sub
' _________________________________________________________________
'
'
sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
'
'   2 way switches and net composed of Wn and Wp adjustable weights 
'
static as integer i,j,k,p
static as single z,s
static as single a(0 to 4)
static as single y(0 to n-1)
'
            k=0
            for p=0 to m-1 
                j=0
                for i=k to k+3  
                    z=x(i)
                    if z<0 then s=z*Wn(j,p) else s=z*Wp(j,p) end if
                    a(j)=s
                    j=j+1
                next i
                     j=0
                for i=k to k+3 
                    y(i)=y(i) + a(j)
                    j=j+1
                next i
                     k=k+4
            next p  
'
            for i=0 to n-1
                x(i)=y(i)
            next i
'
'
end sub
' _________________________________________________________________
'
'
sub prt_1ds(g1() as single)
'
'    print 1d, single precision, array .
'
static as integer n,i
'
'  
        n=ubound(g1)
    for i=0 to n
        print g1(i);" , ";
    next i
    print
'
end sub
' _________________________________________________________________
'
'
sub mutate(Wn() as single, Wp() as single, m as integer)
'
'     Randomly alter a few weights, retain if cost less else retain 
'   previous few weights .
'
static as integer i,j,k,p,q
static as single nv,pv
static as double s
'
'  Assuming this selection criteria produces only a few changes.
'
Randomize ,1
'
q=int(m*rnd)
for p=0 to m-1 
   for j=0 to 3
    nv = Wn(j,p)
      s=rnd
  if (s<0.5) then nv=nv*(1+csng(s)) else nv=nv end if
     pv = Wp(j,p)
     s=rnd
  if (s<0.5) then pv=pv*(1+csng(s)) else pv=pv end if
    Wn(j,i)=nv
    Wp(j,i)=pv
   next j
next p   
 
 
'
'
end sub
' _________________________________________________________________
'
'
sub UWCopy(Av() as single, Bv() as single, m as integer)
'
'   Copy values from Av to Bv
'
static as integer j,p

'
'  Assuming this selection criteria produces only a few changes.
'
 for p=0 to m-1 
   for j=0 to 3
     Bv(j,p) = Av(j,p) 
   next j
 next p   
'
end sub
' _________________________________________________________________
'
'
function bp_ramp(x as single) as single

'  bipolar ramp function .
static as single y
      if abs(x) >1 then y=x/abs(x) else y=x end if
      return y

end function
'
sub apply_activation(g1() as single)
'
'  Apply activation function to x()
'
static as integer n,i
static as single y
'
'  
        n=ubound(g1)
    for i=0 to n
        y=g1(i)
        y=bp_ramp(y)
         g1(i) = y
    next i
'
end sub

' _________________________________________________________________
'
'
'    Next 3 functions, From another author
function round(in as double, places as ubyte = 2) as string
  dim as long _mul = 10 ^ places
  return str(cdbl(int(in * _mul + .5) / _mul))
End Function
'
' ----------------------------------------------------------------------
'
function max( a as double, b as double ) as double
  return iif( a > b, a, b)
end function
'
' ----------------------------------------------------------------------
'
'' Sum of squared difference
' Inputs   vec() , tar
' Outputs  cost
function costL2(vec() as single, tar() as single ) as double
  static as integer i
  dim as double cost
  for i as long= 0 to ubound(vec)-1
    var e = vec(i) - tar(i)
    cost += e*e
  next i
  return cost
end function
'
' ----------------------------------------------------------------------
'
'
  '' Fast Walsh Hadamard Transform
  '' This routine is from another author .
  '' As illustrated  
  '' Input vec() , output vec()
  sub xwht( vec() as single, hs_pow2_shl as long = 0 ) '' 1 or 2 for Partial Fast W.H.T.
    static as long n , i
    n = ubound(vec)+1
    var hs = 1 shl hs_pow2_shl
    'print " xwht, n , hs ";n;" , ";hs
    while (hs < n) 
      var i = 0
      while (i < n) 
        var j = i + hs
        while (i < j) 
          var a = vec(i)
          var b = vec(i + hs)
          vec(i) = a + b
          vec(i + hs) = a - b
          i += 1
        wend
        i += hs
      wend
      hs += hs
    wend
    '   scale
    for i=0 to n
      vec(i)=vec(i)/n
    next i
  end sub
' _________________________________________________________________
'
'

[dafhi]

will have a look, thanks. i train hyperparams to find faster network convergence, had a realization that some hyperparam sets are obviously bad so i break loop early. not quite ready for release tho

here is a great video on 'mutation' Christoph Adami - TED talk

[edit] here's my hyperparam thing

Code: Select all

/'  hyperparam search for switchNet4 demo - 2023 Sep 13.1 - by dafhi

  proof-of-concept hyper-parameter trainer.
  
  my hyper-parameters represent linear interpolation
  
    i linearly interpolate 3 tihngs for this demonstration
  1. mutator size
  2. swap chance
  3. flip chance
  
    the hyper-parameters are 3 variables, for each of the above:
  base + percentage * delta.
  
  so that's 9 variables in total which get mutated until fast-ish
  network performance is discovered.
  
    mutation formula:
  sub hyperparam.new_val
    var big_nudge = lo+rnd*(hi-lo)
    var nudge = curr * (1 + .025*(rnd-.5))
    curr = iif( rnd < .11, big_nudge, nudge )
    clamp( curr )
    
  you can see network performance after some hyperparameter training,
  comment out  #define train_hyperparams, below

'/


#define train_hyperparams


'#include "../ba_ecs.bas"
'#include once "util.bas"
/' -- util.bas - 2023 Aug 17 - by dafhi

'/
'#include "boilerplate.bas"
/' -- boilerplate.bas - 2023 May 12 - by dafhi

  + ------------------------ +
  |  freebasic  |  c++       |
  + ----------- + ---------- +
  |  true = -1  |  true = 1  |
  |  0.99 =  1  |  0.99 = 0  | .. i hope that covers it
  +------------- ----------- +

'/

#define sng         as single
#define dbl         as double

function min( a as double, b as double ) as double
  return iif( a < b, a, b)
end function

function max( a as double, b as double ) as double
  return iif( a > b, a, b)
end function

function clamp( in dbl, hi dbl = 1, lo dbl = 0) dbl
  return min( max(in, lo), hi ) '' June 12
End Function

union uptrs
  as any ptr          a
  as ubyte ptr        b
  as ushort ptr       sho
  as ulong ptr        ul
  as ulongint ptr     uli
  As Single Ptr       s
  as double ptr       d
End Union

' -------------- boilerplate.bas

' ------- util.bas continued ..
#macro sw( a, b, tmp )
  tmp = a: a = b: b = tmp
#endmacro
'
' -------- util.bas

' -------- hyperparams proof-of-concept continued ..
'
#include "string.bi" '' format() [rounding]

'' switch net 4
'' https://editor.p5js.org/congchuatocmaydangyeu7/sketches/IIZ9L5fzS
  
'' translation w/o hyperparameter trainer:
'' https://freebasic.net/forum/viewtopic.php?p=299998#p299998

type SwNet4
  '' vecLen must be 4,8,16,32.....
  declare sub setup( vecLen as long, as long )
  declare sub recall( () as single, byref as single ptr ) '' () arrays
  declare sub _abcd( () as single, as long )
  declare sub _flips_backup
  declare sub _flips_restore
  as long   depth
  as single scale
  sng params( any )
  as ubyte  _flips( any ) 
  as ubyte  _flip_undo( any )
  as single _a,_b,_c,_d
  as long   _paramIdx, _j_base
end type

sub SwNet4.setup( vecLen as long, _depth as long)
  depth = _depth
  scale = 1 / sqr( vecLen shr 2 )

  redim params ( 8 * vecLen * depth - 1 )
  var j = 0
  for i as long = 0 to ubound(this.params) step 8
    this.params(i+j) = this.scale
    this.params(i+4+j) = this.scale
    j=(j+1) and 3
  next

  redim _flips( vecLen - 1 )
  for  i as long=0 to vecLen-1
    this._flips(i)= rnd
  next
  redim _flip_undo ( ubound(_flips) )

end sub

  '' Fast Walsh Hadamard Transform
  sub wht( vec() as single, hs_pow2_shl as long = 0 ) '' 1 or 2 for Partial Fast W.H.T.
    dim as long n = ubound(vec)+1
    var hs = 1 shl hs_pow2_shl
    while (hs < n) 
      var i = 0
      while (i < n) 
        var j = i + hs
        while (i < j) 
          var a = vec(i)
          var b = vec(i + hs)
          vec(i) = a + b
          vec(i + hs) = a - b
          i += 1
        wend
        i += hs
      wend
      hs += hs
    wend
  end sub

sub SwNet4._abcd( result() as single, j as long )
  _paramIdx += 8        
  dim sng x=result( j+_j_base )
  #if 1
  if(x<0)then
    _a+=x*params(_paramIdx)
    _b+=x*params(_paramIdx+1)
    _c+=x*params(_paramIdx+2)
    _d+=x*params(_paramIdx+3)     
  else
    _a+=x*params(_paramIdx+4)
    _b+=x*params(_paramIdx+5)
    _c+=x*params(_paramIdx+6)
    _d+=x*params(_paramIdx+7)
  endif
  #else
  if(x<0)then
    _a+=x*param_ecs.val_read(_paramIdx)
    _b+=x*param_ecs.val_read(_paramIdx+1)
    _c+=x*param_ecs.val_read(_paramIdx+2)
    _d+=x*param_ecs.val_read(_paramIdx+3)     
  else
    _a+=x*param_ecs.val_read(_paramIdx+4)
    _b+=x*param_ecs.val_read(_paramIdx+5)
    _c+=x*param_ecs.val_read(_paramIdx+6)
    _d+=x*param_ecs.val_read(_paramIdx+7)
  endif
  #endif
  _j_base += 1
end sub

sub SwNet4.recall( result() as single, byref inVec as single ptr )

  for i as long = 0 to ubound(result)
    result(i) = inVec[i] * (scale/9) * iif( this._flips(i)and 1, 1, -1 )
  next
  wht( result() )
  _paramIdx = -8
  for i as long = 0 to this.depth - 1
    for j as long = 0 to ubound(result) step 4
      _j_base = 0
      _a=0:_b=0:_c=0:_d=0
      _abcd result(), j
      _abcd result(), j
      _abcd result(), j
      _abcd result(), j
      result(j)=_a
      result(j+1)=_b
      result(j+2)=_c
      result(j+3)=_d
    next
    const pow2_shl = 2 '' August 1
    wht( result(), pow2_shl )
  next
end sub
  
sub SwNet4._flips_backup
  for i as long = 0 to ubound(_flips)
    _flip_undo(i) = _flips(i)
  next
end sub

sub SwNet4._flips_restore
  for i as long = 0 to ubound(_flips)
    _flips(i) = _flip_undo(i)
  next
end sub


function costL2(vec() as single, byref tar as single ptr) as double
  dim dbl cost
  for i as long= 0 to ubound(vec)-1
    var e = vec(i) - tar[i]
    cost += e*e
  next
  return cost
end function


type hyperparam
  declare constructor
  declare constructor( sng, sng )
  declare operator cast sng
  declare operator cast as string
  declare operator let( sng )
  sng     hi, lo, curr
  declare sub new_val
  declare sub val_from_file
end type

constructor hyperparam
end constructor

constructor hyperparam( _lo sng, _hi sng )
  hi = _hi
  lo = _lo
end constructor

operator hyperparam.cast sng
  return curr
end operator

operator hyperparam.cast as string
  return str(curr)
end operator

operator hyperparam.let( s sng)
  curr = s
end operator

sub hyperparam.val_from_file
  get #1,, curr
end sub

sub hyperparam.new_val
    var big_nudge = lo+rnd*(hi-lo)
    var nudge = curr * (1 + .025*(rnd-.5))
  curr = iif( rnd < .14, big_nudge, nudge )
  curr = clamp(curr,hi,lo)
end sub

type hypers_base
  declare sub       dbg( sng, as string )
  declare sub       quick_vals( sng, sng, sng )
  declare sub       print
  declare sub       from_file
  as hyperparam     bas, delt, dcay
  sng               delt0 '' for copying pre-decay delta
end type

sub hypers_base.dbg( smax sng, msg as string )
  if bas + delt > smax orelse dcay > 1 then
   ? msg
   ? bas + delt
   ? dcay
   : sleep
  endif
end sub

sub hypers_base.quick_vals( b sng, del sng, dcy sng )
  bas = b
  delt = del
  dcay = dcy
end sub

sub hypers_base.print
  ? bas'format(bas, ".##")
  ? delt'format(delt0, ".##")
  ? dcay'format(dcay, ".###")
end sub

sub hypers_base.from_file
  bas.val_from_file
  delt.val_from_file
  dcay.val_from_file
end sub

type hyper_set
  as hypers_base   muta_size
  as hypers_base   swap_chance
  as hypers_base   flip_chance
  dbl             cost
end type

sub backup_delta( byref t as hyper_set )
  t.muta_size.delt0 = t.muta_size.delt
  t.swap_chance.delt0 = t.swap_chance.delt
  t.flip_chance.delt0 = t.flip_chance.delt
end sub

sub restore_decayed_delta( byref t as hyper_set )
  t.muta_size.delt = t.muta_size.delt0
  t.swap_chance.delt = t.swap_chance.delt0
  t.flip_chance.delt = t.swap_chance.delt0
end sub

sub print_hypers( byref t as hyper_set )
  ? " muta_size": t.muta_size.print
  ? " swap_chance": t.swap_chance.print
  ? " flip_chance": t.flip_chance.print
end sub


type Mutator
  declare constructor( as long, as long, as single )
  declare sub mutate( byref as SwNet4, byref as hyper_set )
  declare sub undo( byref as SwNet4 )
  as long   size, precision
  as single limit
  as single previous( any)
  as long   pIdx( any)
end type


constructor mutator( _size as long, precis as long, limit  as single )
  size = _size
  redim this.previous( size-1 )
  redim this.pIdx( size-1 )
  this.precision = precis
  this.limit = limit
end constructor

sub Mutator.mutate( byref net as SwNet4, byref hypers as hyper_set )
  
  dim sng     sc = ubound(net.params) + .499 '' c++ .999
  dim as long rpos, rpos2, c
  dim sng     vm, m
  
  '' previous() and pIdx() detail a small set for mutation

  for i as long= 0 to ubound( this.pidx ) ' relatively small array

    '' to continue GreenInk's sub-random idea, i try swapping - dafhi
    if rnd < hypers.swap_chance.bas + rnd*hypers.swap_chance.delt andalso i < size-1 _
      then

        rpos = rnd * sc
      this.pIdx(i) = rpos '' muta location
      this.previous(i) = net.params(rpos) '' save pre-mutate
      i += 1
        rpos2 = rnd * sc
      this.pIdx(i) = rpos2 '' muta location
      this.previous(i) = net.params(rpos2) '' save pre-mutate
      
        static as typeof(net.params) tmp '' custom swap
      sw( net.params(rpos), net.params(rpos2), tmp )
      
    else

      rpos = rnd * sc   ' random elem
      this.pIdx(i) = rpos '' muta location
      this.previous(i) = net.params(rpos) '' save pre-mutate
      m = 2 * this.limit * exp(rnd*-this.precision)
      vm = net.params(rpos) + iif(rnd<.5,m,-m)
      if (vm > this.limit)orelse(vm < -this.limit) then continue for
      net.params(rpos) = vm
    endif
    
  next
  
  hypers.swap_chance.delt *= hypers.swap_chance.dcay '' new

  '' new: mutate flips (Aug 23
  net._flips_backup
  sc = ubound(net._flips)+.499 '' c++ .999
  c = ubound(net._flips)+1
  var u = c*( hypers.flip_chance.bas + hypers.flip_chance.delt) - 1
  for i as long = 0 to u
    dim as long dest_index = rnd*sc
    net._flips( dest_index ) = rnd
  next

  hypers.flip_chance.delt *= hypers.flip_chance.dcay
  
end sub

sub Mutator.undo( byref net as SwNet4 )

  for i as long = ubound(pIdx) to 0 step -1
    net.params(pIdx(i))= previous(i)
  next
 
  net._flips_restore
end sub


'#include "swnet_hypers_demo.bas"
  
  namespace demo

    '' Test with Lissajous curves
    dim as ulong c1 = rgb(0,0,0),c2 = rgb(255,255,0)
    dim sng ex(8,255)
    dim sng work(255)
    dim dbl parentCost

    dim as long w,h
    dim as SwNet4 parentNet
    
const success_frames_scalar = 1.02
  

sub visualize
  cls ' clearscreen
   
  locate 2,1
  ? "Training Data"
  
  for i as long = 0 to 7
    for j as long= 0 to 255 step 2
      var y=44 + 18 * ex(i,j + 1)
      pset (25 + i * 40 + 18 * ex(i,j), y), c2
    next
  next
  
  locate 10,1
  ? "Recall"
  
  for i as long = 0 to 7
    parentNet.recall( work(), @ex(i,0) )
    for j as long = 0 to  255 step 2
      pset(25 + i * 40 + 18 * work(j), 104 + 18 * work(j + 1)), c2
    next
  next
end sub

sub random_hypers( byref t as hyper_set )
  const chance = 1.
  if rnd < chance then t.muta_size.delt.new_val
  if rnd < chance then t.muta_size.dcay.new_val
  if rnd < chance then t.muta_size.bas.new_val
  if rnd < chance then t.swap_chance.delt.new_val
  if rnd < chance then t.swap_chance.dcay.new_val
  if rnd < chance then t.swap_chance.bas.new_val
  if rnd < chance then t.flip_chance.delt.new_val
  if rnd < chance then t.flip_chance.dcay.new_val
  if rnd < chance then t.flip_chance.bas.new_val
end sub
  
  
sub best_to_file( byref t as hyper_set )
  restore_decayed_delta t
    open "hypers.txt" for output as #1
  put #1,, t.muta_size.bas.curr
  put #1,, t.muta_size.delt.curr
  put #1,, t.muta_size.dcay.curr
  put #1,, t.swap_chance.bas.curr
  put #1,, t.swap_chance.delt.curr
  put #1,, t.swap_chance.dcay.curr
  put #1,, t.flip_chance.bas.curr
  put #1,, t.flip_chance.delt.curr
  put #1,, t.flip_chance.dcay.curr
  write #1, ""
  write #1, "size base " + str(t.muta_size.bas.curr)
  write #1, "delta "+ str(t.muta_size.delt.curr)
  write #1, "delta decay "+ str(t.muta_size.dcay.curr)
  write #1, "swap base " + str(t.swap_chance.bas.curr)
  write #1, "swap delta "+ str(t.swap_chance.delt.curr)
  write #1, "swap decay "+ str(t.swap_chance.dcay.curr)
  write #1, "flip base " + str(t.flip_chance.bas.curr)
  write #1, "flip delta "+ str(t.flip_chance.delt.curr)
  write #1, "flip decay "+ str(t.flip_chance.dcay.curr)
  close
end sub
  
  dim as double     perf
  dim sng           cost_best(2)
  dim as hyper_set  hypers_best


sub save_new_stats( hardening as boolean, byref t as hyper_set )

  for i as long = ubound(cost_best) to 1 step -1
    cost_best(i) = cost_best(i-1)
  next
  cost_best(0) = parentCost

  restore_decayed_delta t
  hypers_best = t
end sub

sub training_info( hardening as boolean )

  locate 22
  ? "best: "; format(cost_best(0), ".##")

  locate 24
  if hardening then
    ? "hardening result from file"
  else
    ? "bad set if > 1 -->  "; format(perf, ".##"); " "
  endif
end sub

  dim as long       epoch
  dim as single     outer_frame, _outer_frame
  dim as hyper_set  hyp(4)
  
  
sub run_epoch( byref t as hyper_set, hardening as boolean = false )
  
  parentNet.setup 256,2
  
  '' after a bug hunt, i realized i need to save / restore 'delta'
  '' because after a run, delta has decayed and if net was good,
  '' i save to hypers_best

  backup_delta t

  demo.parentCost = 1/0
  
  t.muta_size.dbg 255.499, "muta size"
  t.swap_chance.dbg 1, "swap"
  t.flip_chance.dbg 1, "swap"
  
  _outer_frame = outer_frame
  #ifdef train_hyperparams
      if not hardening then _outer_frame = outer_frame * .7 + _
    rnd * outer_frame * .5
  #endif
  
  for frame as long = 1 to _outer_frame
  
  var precision = 35
  var limit = 2*parentNet.scale
  
  '' mutator with hyperparams.  2023 Aug 8 - by dafhi
  dim as Mutator mut = type( t.muta_size.bas + rnd*t.muta_size.delt, precision, limit )
  t.muta_size.delt.curr *= t.muta_size.dcay.curr '' reduce max over time

  for i as long = 0 to 6 '' 100 originally.  reduced for some cpu sleep
    mut.mutate( parentNet, t )
    dim dbl cost = 0
    for j as long = 0 to 7
      parentNet.recall( work(), @ex(j,0) )
      cost += costL2( work(), @ex(j,0) )
    next
    if (cost < parentCost) then
      parentCost = cost
    else
      mut.undo( parentNet )
    endif
  next
  
  #ifdef train_hyperparams
    if hardening then
      visualize
    else
      var break_early = .8
        perf = (parentCost / cost_best(0)) ^ break_early * _
      (frame / _outer_frame) ^ 1.2
      if perf  > .99 then parentCost = 1/0: exit for
    endif
    training_info hardening
  #else
    visualize
  #endif
  
  locate 21
  ?"Cost: "; ; format(parentCost, ".##"); " "
  locate 19
  ? "frame "; frame; " of"; _outer_frame; " "
  
  sleep 1 '' large sleep val to keep laptop cool-ish

  dim as string kstr = lcase(inkey)
  select case kstr
  case ""
  case chr(27)
    end
  case "z","x","c","v", " "
    parentCost *= (frame / _outer_frame) ^ .5
    exit for
  end select

  next frame
  
  t.cost = parentCost

  #ifdef train_hyperparams
    if parentCost < cost_best(0) then

      save_new_stats hardening, t
      if not hardening then
        best_to_file t
      endif

      visualize

      if not hardening then outer_frame = _outer_frame

    endif
    
  #endif
  
end sub ' ---------- run_epoch

sub print_cost( byref t as hyper_set, y as long )
  locate 26+y
  print t.cost
end sub


sub run_array
  run_epoch hyp(0)
  hyp(0) = hypers_best
  random_hypers hyp(0)
end sub


sub setup

  dim as hyper_set ptr p = @hyp(0)

  '' hyperparam ranges
  p->muta_size.bas = type(4,43)
  p->muta_size.delt = type(65,160)
  p->muta_size.dcay = type(.91,.99999)
  
  p->swap_chance.bas = type( .001, .4)
  p->swap_chance.delt = type(.001, .5)
  p->swap_chance.dcay = type(.89, .99999)
  
  p->flip_chance.bas = type( .001, .4)
  p->flip_chance.delt = type(.001, .5)
  p->flip_chance.dcay = type(.88, .99999)
  
  w = 400
  h = 400
  
  screenres w,h,32
  
  const tau = 8*atn(1)
  for i as long = 0 to 127
    '' Training data
    dim sng t = (i * tau) / 127
    ex(0,2 * i) = sin(t)
    ex(0,2 * i + 1) = sin(2 * t)
    ex(1,2 * i) = sin(2 * t)
    ex(1,2 * i + 1) = sin(t)
    ex(2,2 * i) = sin(2 * t)
    ex(2,2 * i + 1) = sin(3 * t)
    ex(3,2 * i) = sin(3 * t)
    ex(3,2 * i + 1) = sin(2 * t)
    ex(4,2 * i) = sin(3 * t)
    ex(4,2 * i + 1) = sin(4 * t)
    ex(5,2 * i) = sin(4 * t)
    ex(5,2 * i + 1) = sin(3 * t)
    ex(6,2 * i) = sin(2 * t)
    ex(6,2 * i + 1) = sin(5 * t)
    ex(7,2 * i) = sin(5 * t)
    ex(7,2 * i + 1) = sin(2 * t)
  next
  
    open "hypers.txt" for input as #1
  
  p->muta_size.from_file
  p->swap_chance.from_file
  p->flip_chance.from_file

  close
  
  var file_found = p->muta_size.bas > 0

  cost_best(0) = 1/0
  #ifdef train_hyperparams
    outer_frame = 659
    if file_found then
        var hardening = true
        for i as long = 1 to 2
          run_epoch *p, hardening
        next
        visualize
        outer_frame *= .93
    else
      random_hypers *p
      hypers_best = *p
    endif
  #else
    outer_frame = 5499
  #endif
  
end sub

end namespace


randomize

demo.setup

#ifdef train_hyperparams
  dim as long epoch_max = 9999
#else
  dim as long epoch_max = 1
#endif

for demo.epoch = 1 to epoch_max
  locate 18
  ? "epoch "; demo.epoch; " of "; epoch_max
  
  demo.run_array
  
  sleep 24
  
next

locate 30
?"done!"

sleep

[Luxan]

The video is interesting, reminds me a little of Conway's Game of Life.

[dafhi]

i notice WHT similarity to FFT. my grasp of FFT is weak but i understand how both highlight that 'typical' NN's are overkill.

working out a variant of switch net in my head where input layer has equal size layer #2, connections mutate-shuffled

[Luxan]

FFTs have some versatility , even a size 8 represents a fair amount of calculation and connectedness .

Reading what's on the internet, deep learning involves larger NN's ;
my matrix approach to NN might qualify if that's the instance.

[Luxan]

In an attempt to re familiarise myself with NNs I examined some
introductory examples from the internet.
There the notion of regularisation is discussed; this can take many
forms, including normalising the weights, selectively subtracting a random
amount, nullifying certain connections.
In this instance an activation function, and its derivative were used with
back propagation.

Your approach is a little different.

[dafhi]

GreenInk described years ago the idea
1. mutate
2. keep vs. revert

and i stuck with it.

my hyper-parameters interest spans a few years. discovering optimizations and realizing oversight from previous projects.

[Luxan]

There was a question about the FFT.

The FFT is similar to the WHT, except there are
complex additions and multiplications by a complex
value, known as the twiddle factor, this might be
interpreted as being like a fixed weight.
Just as the WHT has a decimation in frequency arrangement,
so does the FFT.

If you run this code you'll see that even a small FFT, or WHT,
is already considerably larger than the first NN illustrated.
The first NN was effectively trained, after 10000 iterations,
to simulate an exclusive or gate and, perhaps, that's all.
The WHT [FFT] version is almost like a deep neural network in
comparison.

There are various properties of the FFT that might be useful.
For instance, by having extra values in the last inputs that
are zero ; we accomplish interpolation at the last pass.
This provides extra, related , values for the main switched
NN ; possibly guarding somewhat against over fitting .

I'm gradually pondering your linear hyper parameter approach.

Code: Select all

' nn2_draw2a.bas

'  Draw Neural Network structure from selection variables.


declare sub nn_illustrate(numInputs as integer,numHiddenNodes as integer,numOutputs as integer,numTrainingSets as integer)


' NN 
dim as integer numInputs = 3
dim as integer numHiddenNodes = 4 ' 2
dim as integer numOutputs = 1
dim as integer numTrainingSets = 8

dim as integer numHiddenLayers=1
'
' ______________________________________________________________________
'
nn_illustrate(numInputs ,numHiddenNodes ,numOutputs ,numTrainingSets)

print"done"
sleep
end
'
' ======================================================================
'
sub nn_illustrate(numInputs as integer,numHiddenNodes as integer,numOutputs as integer,numTrainingSets as integer)
'
'  Illustrate the structure of the Neural Network .
'
dim as integer i,j
'
screen 12
locate 2,30
print" Neural Network Structure "
locate 3,4
print "Inputs"
for i=0 to numInputs - 1
   locate i+4,1
   print i
   line(30,i*15+50)-(60,(i+1)*15+50),14,b
next i
'
' HiddenNodes
'
locate 3,12
color 11,0
print "Hidden Nodes"
for i=0 to numHiddenNodes - 1
 '  locate i+4,1
 '  print i
   line(90,i*15+50)-(120,(i+1)*15+50),11,b
next i
'
' Outputs
'
locate 3,41
print "Outputs"
for i=0 to numOutputs-1
   line(320,i*15+50)-(350,(i+1)*15+50),10,b
next i 
'

'
' HiddenNodes Biases
'
locate 8,17
color 13,0
print "Hidden Nodes Biases"
for i=0 to numHiddenNodes - 1
 '  locate i+4,1
 '  print i
   line(125,i*15+50)-(155,(i+1)*15+50),13,b
next i
'
' Outputs Biases
'
locate 8,46
print "Outputs Biases"
for i=0 to numOutputs-1
   line(355,i*15+50)-(385,(i+1)*15+50),13,b
next i 
'
'
'  Training sets .
'
for j=0 to numTrainingSets-1
  locate 13,j*5+5
  print j
  for i=0 to numInputs - 1
     locate i+14,1
     print i
     line(30+j*40,i*15+210)-(60+j*40,(i+1)*15+210),14,b
  next i
  
next j
'
'
dim as integer k
k= numInputs*15+15
for j=0 to numTrainingSets-1
 ' locate 13+numInputs,j*5+5
 ' print j
  for i=0 to numOutputs - 1
     locate numInputs+15,1
     print i
     line(30+j*40,k+i*15+210)-(60+j*40,k+(i+1)*15+210),10,b
  next i
next j
'
locate 12,6
print " Training Set"
line(25,170)-( numTrainingSets*40+30,(numInputs+1+numOutputs)*15+5+210),7,b

'
locate 30,54
color 12,0
print " press any key to continue"
sleep
'
' 
'   Structure of length 8 fft , or wht .
'
cls
dim as integer n

n=8
locate 2,30
color 15,0
print" FFT or WHT Structure, N=8 "
locate 3,4
print "Inputs"
for i=0 to n - 1
   locate i+4,1
   print i
   line(30,i*15+50)-(60,(i+1)*15+50),14,b
next i
'
locate 3,10
color 10,0
print " Bit Reversed"
for i=0 to n - 1
  ' locate i+4,1
  ' print i
   line(65,i*15+50)-(95,(i+1)*15+50),10,b
next i
'
locate 12,14
color 11,0
print " Pass 1"
for i=0 to n - 1
   line(120,i*15+50)-(150,(i+1)*15+50),11,b
next i
'
locate 12,22
color 11,0
print " Pass 2"
for i=0 to n - 1
   line(175,i*15+50)-(205,(i+1)*15+50),11,b
next i
'
locate 12,29
color 11,0
print " Pass 3"
for i=0 to n - 1
   line(230,i*15+50)-(260,(i+1)*15+50),11,b
next i
'
locate 3,38
color 15,0
print"Switched NN "
i=0
   line(290,i*15+50)-(380,(n)*15+50),15,b


'
locate 30,54
color 12,0
print " press any key to continue"
sleep

dim as ulong SCREEN_EXIT=&h80000000
Screen 0, , , SCREEN_EXIT' (with SCREEN_EXIT=&h80000000)
' The previous command returns to native console/terminal window.

'screen
'
end sub

[dafhi]

xor gate .. yeah okay. one of my past projects i attempted 'all the gates'

input

00
01
10
11 .. sparsenet

.. [edit] ..

my hyper-parameter comments need updating. linear interp on its own could be replaced by one variable.
my interp is .. base + [decaying] delta

[Luxan]

This code will illustrate some of the concepts I perceive
as being used within switchnet4.

The switch section selectively uses the positive or negative
Neural Network weights. These weights reside within banks of size
four.

The main banks contain the Neural Network weights for each layer,
separated into negative [ cyan ] and positive [ red ] banks.
The stored banks are copied from the main banks.
Then conditionally written back to the main banks, depending upon
the difference between the output values and the expected output values.

The section labeled "extra" is to be determined.



Now I have a few questions.
How much of the main banks is randomly altered, is it just a few banks
of size four; or all of the main banks.
Are these alterations dependent upon whether the positive or negative
banks in use at any particular epoch.

In the code for SwNet4(), firstly wht( result() ) is used, then
after some calculations, wht( result(), pow2_shl ) is used.
What is the effect of this, are we sampling from a larger data set.

Code: Select all

'  sciwiseg@gmail.com

'   Rectangle drawing 


screen 12
window screen (0,0)-(100,100)



type pt
     x as single
     y as single
end type

dim as integer n,i,cm,j
n=64


dim pts(0 to n-1) as pt
dim chrome(0 to n-1) as integer
'
' Read data
'
restore datax
for i=0 to n-1
  read pts(i).x
next i
restore datay
for i=0 to n-1
  read pts(i).y
next i
restore colorp
for j=0 to (n/2)-1
  read chrome(j)
next j  
'
'  Display connected pts
'
dim as single x,y,x1,y1
' 
'  Line [target,] [[STEP]|(x1, y1)]-[STEP] (x2, y2) [, [color][, [B|BF][, style]]]
' or
'  Line - (x2, y2) [, [color][, [B|BF][, style]]]
' 

for i=0 to n-1 step 2 'n-1
    x=pts(i).x
    y=pts(i).y  
   cm=chrome(int(i/2))
   x1=pts(i+1).x
   y1=pts(i+1).y
 if x1 > 0 and x > 0 then line(x,y)-(x1,y1),cm,b
   
next i
'
' ______________________________________________________________________
'
color 10,0
locate 2,2
print "in";
color 13,0
locate 2,8
print "wht"
color 14,0
locate 2,12
print " switch"
color 15,0
locate 2,21
print "main banks"
color 6,0
locate 2,32
print " extra"
color 13,0
locate 2, 40
print "wht"
color 9,0
locate 2,46
print "out"
color 8,0
locate 18,3
print "stored banks"
color 11,0
locate 14,50
print "- banks"
color 4,0
locate 15,50
print "+ banks"



' within SwNet4()
' wht( result() )
' wht( result(), pow2_shl )
color 13,0
locate 12,1
print "wht(result())"
locate 12,38
print "wht(result(), pow2_shl )"
sleep
end
'
' ======================================================================
'
' ----------------------------------- x --------------------------------
'
datax:
'input
data 2,5
' wht
data 7,15
' switch
data 17,20
' block
data 22,39
' - bank
data 24,27,24,27,29,32,29,32,34,37,34,37
' + bank
data 24,27,24,27,29,32,29,32,34,37,34,37
' xtra
data 41,44
' wht 2
data 46,54
' output
data 56,59
'
' -------------------------- banks 2 ----------------------------------
'
' block
data 22,39
' - bank
data 24,27,24,27,29,32,29,32,34,37,34,37
' + bank
data 24,27,24,27,29,32,29,32,34,37,34,37
'
' ------------------------------ y ------------------------------------
'
datay:
' input
data 10,30
' wht
data 10,30
' switch
data 10,52
' block
data 8,53
' - bank 
data 10,18,21,30,10,18,21,30,10,18,21,30
' + bank 
data 32,40,43,51,32,40,43,51,32,40,43,51
' xtra
data 10, 52
'  wht 2
data 10,30
' output
data 10,30
'
'  ................... banks 2 ......................
'
' block
data 55,100
' - bank 
data 57,65,68,76,57,65,68,76,57,65,68,76
' + bank 
data 79,87,90,98,79,87,90,98,79,87,90,98
'
' ------------------------- colour -------------------------------------
'

colorp:
data 10,13,14,15,11,11,11,11,11,11,4,4,4,4,4,4,6,13,9
data 8,11,11,11,11,11,11,4,4,4,4,4,4

[dafhi]

i haven't coded a wht from scratch. my version with pow2_shl is an all-in-one, after examining GreenInk's .. yes i did note he used different wht at different stage. it's cool you made an illustration; i may stare at it a bit longer, i code from inspiration.

in GreenInk's diagram, the switches seem to reroute connections, whereas in the lissajous curve demo, switches flip values.

also, GreenInks demo initializes said flip values a only reads later. i'm often mis-analyze questions and guess mis-matched inspiration behind other peoples ideas, so bear with.

also, one reason i see WHT potential is it doesn't use multiply. there are also potential single thread parallelism through masking

for example in graphics you can mull Red and Green in one go
result = color * FF00FF


[edit]

for a rate i suggest 28%

[Luxan]

I'm glad the diagram is of interest, and possible use, to you; feel free to construct your own
diagrams with the Rectangle Drawing code.
Ah, I forgot to illustrate the sign flipping portion of the diagram.

As the WHT doesn't use multiply, or complex numbers, it's fairly easy to implement .

I definitely need to sructinize the original code and also unravel what Greenlnk was intending.