## Switch Net 4 neural network

General FreeBASIC programming questions.
dafhi
Posts: 1619
Joined: Jun 04, 2005 9:51

### Re: Switch Net 4 neural network

i assumed 'width' meant array size. i only glanced at the blog b/c i read slowly.

i highly recommend GreenInk's mutator concept for simplicity

the idea is to save a portion of the network which will be changed

in .mutate(),
pIdx() stores location
previous() stores value

 if you want, you could post here and i'll do my best to add a mutator. the concept is simple enough. i'll also add (depending on how well i grasp your code) synthetic data generator of some sort.
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

After referring to the Mathwork site, and my local installation
of GNU Octave, I determined that my version of the Fast Walsh Hadamard
Transform and that presented by the originator of this discussion,
are correct.

The following code verifies this ; I haven't included my routine
WHT() within this.

Examining the use of the xwht() type function, in the original code,
the expansion of parameters appears to happen within the switch network.

A cost function is easy to write, adjusting the weights to minimise
this; is more involved.

Code: Select all

``````
/'

wht_wn4.bas
sciwiseg@gmail.com

fast Walsh Hadamard Transform and possible wnet4.

0 1 2 3 4 5  6 7
x = [4 2 2 0 0 2 -2 0]
y = fwht(x)    "sequency"  format
y = [1 1 0 1 0 0 1 0]

x1 = ifwht(y)
x1 = [4 2 2 0 0 2 -2 0]

Using GNU Octave , with signal package loaded and this script :

x = [4 2 2 0 0 2 -2 0]
y = fwht(x)
y = fwht(x,8)

returns :

x =    4   2   2   0   0   2  -2   0

y =    1   1   0   1   0   0   1   0

y =    1   1   0   1   0   0   1   0

y =    1   0   1   0   1   1   0   0

'/
declare sub flipper(A() as single)
declare sub genrand1(g1() as single)
declare sub genrand2(g2() as single)
declare sub prt_1ds(g1() as single)
declare sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
'
declare sub xwht( vec() as single, hs_pow2_shl as long = 0 )
'
dim as long L
dim as integer n,m,i
dim as single x

L=3
n=2^L
m=2^(L-2)

dim vec(0 to n-1) as single

'
' ......................................................................
'
restore xdat
for i=0 to n-1
next i
'
print " Input "
for i=0 to n-1
print vec(i);" , ";
next i
print
'
xwht(vec() , 0 )
'
print " xwht(x)  output "
for i=0 to n-1
print vec(i)/n;" , ";
next i
print
'
print " Expected output "
restore ydat
for i=0 to n-1
print x;" , ";
next i
print
print
'
' ......................................................................
'
dim Wn(0 to 3,0 to m-1) as single
dim Wp(0 to 3,0 to m-1) as single
'
'
'
genrand2(Wp())
genrand2(Wn())
genrand1(vec())
flipper(vec())
'
xwht(vec() , 0 )
swnet4(vec(),Wp(),Wn(),n,m)
xwht(vec() , 0 )
'
'
print " xwht(), swnet4(), xwht()  output "
prt_1ds(vec())
'
'
end
'
'=======================================================================
'
'
'' This routine is from another author .
'' As illustrated
'' Input vec() , output vec()
sub xwht( vec() as single, hs_pow2_shl as long = 0 ) '' 1 or 2 for Partial Fast W.H.T.
static as long n
n = ubound(vec)+1
var hs = 1 shl hs_pow2_shl
'print " xwht, n , hs ";n;" , ";hs
while (hs < n)
var i = 0
while (i < n)
var j = i + hs
while (i < j)
var a = vec(i)
var b = vec(i + hs)
vec(i) = a + b
vec(i + hs) = a - b
i += 1
wend
i += hs
wend
hs += hs
wend
end sub
' _________________________________________________________________
'
'
sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
'
'   2 way switches and net composed of Wn and Wp adjustable weights
'
static as integer i,j,k,p
static as single z,s
static as single a(0 to 4)
static as single y(0 to n-1)
'
k=0
for p=0 to m-1
j=0
for i=k to k+3
z=x(i)
if z<0 then s=z*Wn(j,p) else s=z*Wp(j,p) end if
a(j)=s
j=j+1
next i
j=0
for i=k to k+3
y(i)=y(i) + a(j)
j=j+1
next i
k=k+4
next p
'
for i=0 to n-1
x(i)=y(i)
next i
'
'
end sub
'
' _________________________________________________________________
'
'
sub genrand1(g1() as single)
'
'   Generate signed random values for 1d array
'
static as integer i,n
static as single z
'
n=ubound(g1)
for i=0 to n-1
z=rnd
if z<0.5 then  z=-z else z=z end if
g1(i)=z
next i
end sub
'
' _________________________________________________________________
'
sub genrand2(g2() as single)
'
'        Generate signed random values for 2d array
'
static as integer i, j, n, m
static as single z
'
n=ubound(g2,1)
m=ubound(g2,2)
'
for j=0 to m
for i=0 to n
z=rnd
if z < 0.5 then z=-z  end if
g2(i,j)=z
next i
next j
'
'
end sub
'
' _________________________________________________________________
'
sub flipper(A() as single)
'
'          Random sign flips, use with input .
'
static as integer n,i
static as single s
'
n=ubound(A)
for i=0 to n
s=rnd
if s<0.5 then s=-1 else s=1 end if
A(i)=s*A(i)
next i
'
'
end sub
' _________________________________________________________________
'
'
sub prt_1ds(g1() as single)
'
'    print 1d, single precision, array .
'
static as integer n,i
'
'
n=ubound(g1)
for i=0 to n
print g1(i);" , ";
next i
print
'
end sub
'
' _________________________________________________________________
'
xdat:
data 4, 2, 2, 0, 0, 2, -2, 0
ydat:
data 1, 0, 1, 0, 1, 1,  0, 0

``````
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

This program illustrates a few well known procedures that are possible
before and after a transform, these aren't limited to a doubling of the
size of the vector.

The question is, did the originator use any of these.

Code: Select all

``````
'
'  ft_pc.bas
'
'  General function transform procedures for discrete, and possibly
' continuous, data .
'
'  sciwiseg@gmail.com
'
'
screen 12
'line(0,0)-(639,479),11,b
window screen(-0.1,-0.1)-(100.1,100.1)
'view screen (0,0)-(639,479)
line(-0.1,-0.1)-(100.1,100.1),14,b

dim as integer i,j,n
dim as single x,x1,y,y1,a,b
'
n=8
a=5
'
for i=0 to n-1
locate i+2,2
print i
next i
'
for i=0 to n-1
y=i*3.4 + 3
x=a
y1=y+3.4
x1=x+5
line(x,y)-(x1,y1),12,bf
line(x,y)-(x1,y1),11,b
next i
'
locate 23,5
print "x()"
'
' ...........................................................
'
n=16
a=20
'
for i=0 to n-1
locate i+2,13
print i
next i
'
for i=0 to n-1
y=i*3.4 + 3
x=a
y1=y+3.4
x1=x+5
if i<(n/2) then line(x,y)-(x1,y1),12,bf end if
line(x,y)-(x1,y1),11,b
next i
'
locate 23,15
print "x(), zero extended"
'
' ......................................................................
'
n=8
a=60
'
for i=0 to n-1
locate i+2,45
print i
next i
'
for i=0 to n-1
y=i*3.4 + 3
x=a
y1=y+3.4
x1=x+5
line(x,y)-(x1,y1),12,bf
line(x,y)-(x1,y1),11,b
next i
'
locate 23,47
print "  x()"
'
' ......................................................................
'
n=16
a=75
'
for i=0 to n-1
locate i+2,56
print i
next i
'
j=0
for i=0 to n-1
y=i*3.4 + 3
x=a
y1=y+3.4
x1=x+5
if j mod 2 = 0 then line(x,y)-(x1,y1),12,bf end if
line(x,y)-(x1,y1),11,b
j=j+1
next i
'
locate 23,58
print "x(), zero insert"
'
' ......................................................................
'
sleep  6000
line(-0.1,-0.1)-(100.1,100.1),0,bf
line(-0.1,-0.1)-(100.1,100.1),14,b
'
n=8
a=5
'
for i=0 to n-1
locate i+2,2
print i
next i
'
for i=0 to n-1
y=i*3.4 + 3
x=a
y1=y+3.4
x1=x+5
line(x,y)-(x1,y1),i+1,bf
line(x,y)-(x1,y1),11,b
next i
'
locate 23,5
print "x()"
'
' ...........................................................
'
n=16
a=20
'
for i=0 to n-1
locate i+2,13
print i
next i
'

for i=0 to n-1
y=i*3.4 + 3
x=a
y1=y+3.4
x1=x+5

line(x,y)-(x1,y1),int(i/2)+1,bf
line(x,y)-(x1,y1),11,b

next i
'
locate 23,15
print "x(), replicate"
'
' ......................................................................
'

sleep

end

``````
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

A slight update.
It's slow going with the coding, so many unrelated notions
on my mind and too many late nights.

I haven't thoroughly read the articles.
I have found that microfiber cloth is effective at cleaning
my eye ware.

Code: Select all

``````
'
'  swnet4f.bas
'                     Switchnet 4, preliminary
'
'       sciwiseg@gmail.com
'

declare function round(in as double, places as ubyte = 2) as string
declare function max( a as double, b as double ) as double
declare function costL2(vec() as single, tar() as single ) as double
declare sub xwht( vec() as single, hs_pow2_shl as long = 0 )
'
declare sub genrand1(g1() as single)
declare sub genrand2(g2() as single)
declare sub flipper(A() as single)
declare sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
declare sub prt_1ds(g1() as single)

declare sub UWCopy(Av() as single, Bv() as single, m as integer)
declare sub mutate(Wn() as single, Wp() as single, m as integer)

declare function bp_ramp(x as single) as single
declare sub apply_activation(g1() as single)
'
'
dim as integer L,n,m,i,j,k
dim as double cost, cost2
'
L=3
n=2^L
m=2^(L-2) ' using 4 elements per bank .
print
print " L= ";L;" n= ";n;" m= ";m
print
dim as single x(0 to n-1)
dim as single y(0 to n-1)
dim Wn(0 to 3,0 to m-1) as single
dim Wp(0 to 3,0 to m-1) as single
'
dim Un(0 to 3,0 to m-1) as single
dim Up(0 to 3,0 to m-1) as single
'
'
'
genrand2(Wp())
genrand2(Wn())
genrand1(x())
'
for i=0 to n-1
y(i)=x(i) ' [0,1]
next i
'
flipper(x())
xwht( x() ,  0 )
swnet4(x(),Wp(),Wn(),n,m)
apply_activation(x())
xwht( x() ,  0 )
'
prt_1ds(x())
cost = costL2(x() , y())
print
print " Initial cost = "; cost
'
' .............................. repeats ...............................
'
for k=1 to 256
'
for i=0 to n-1
x(i)=y(i)
next i
'                          copy, mutate; before wht & swnet4
UWCopy(Wp() , Up() , m )
UWCopy(Wn() , Un() , m )
mutate(Wn() , Wp(), m)
'
flipper(x())
xwht( x() ,  0 )
swnet4(x(),Wp(),Wn(),n,m)
apply_activation(x())
xwht( x() ,  0 )
cost2 = costL2(x() , y())
'
if cost2>cost then
UWCopy(Up() , Wp() , m )
UWCopy(Un() , Wn() , m )
'  print " cost2 > cost "
else
'  print k;" cost : "; cost2
cost = cost2
end if
'
next k
print " final   cost = ";cost
'
end
'
' ================================================================
'
sub genrand1(g1() as single)
'
'   Generate signed random values for 1d array
'
static as integer i,n
static as single z
'
n=ubound(g1)
for i=0 to n-1
z=rnd
if z<0.5 then  z=-z else z=z end if
g1(i)=z
next i
end sub
'
' _________________________________________________________________
'
sub genrand2(g2() as single)
'
'        Generate signed random values for 2d array
'
static as integer i, j, n, m
static as single z
'
n=ubound(g2,1)
m=ubound(g2,2)
'
for j=0 to m
for i=0 to n
z=rnd
if z < 0.5 then z=-z  end if
g2(i,j)=z
next i
next j
'
'
end sub
'
' _________________________________________________________________
'
sub flipper(A() as single)
'
'          Random sign flips, use with input .
'
static as integer n,i
static as single s
'
n=ubound(A)
for i=0 to n
s=rnd
if s<0.5 then s=-1 else s=1 end if
A(i)=s*A(i)
next i
'
'
end sub
' _________________________________________________________________
'
'
sub swnet4(x() as single, Wp() as single, Wn() as single, n as integer, m as integer)
'
'   2 way switches and net composed of Wn and Wp adjustable weights
'
static as integer i,j,k,p
static as single z,s
static as single a(0 to 4)
static as single y(0 to n-1)
'
k=0
for p=0 to m-1
j=0
for i=k to k+3
z=x(i)
if z<0 then s=z*Wn(j,p) else s=z*Wp(j,p) end if
a(j)=s
j=j+1
next i
j=0
for i=k to k+3
y(i)=y(i) + a(j)
j=j+1
next i
k=k+4
next p
'
for i=0 to n-1
x(i)=y(i)
next i
'
'
end sub
' _________________________________________________________________
'
'
sub prt_1ds(g1() as single)
'
'    print 1d, single precision, array .
'
static as integer n,i
'
'
n=ubound(g1)
for i=0 to n
print g1(i);" , ";
next i
print
'
end sub
' _________________________________________________________________
'
'
sub mutate(Wn() as single, Wp() as single, m as integer)
'
'     Randomly alter a few weights, retain if cost less else retain
'   previous few weights .
'
static as integer i,j,k,p,q
static as single nv,pv
static as double s
'
'  Assuming this selection criteria produces only a few changes.
'
Randomize ,1
'
q=int(m*rnd)
for p=0 to m-1
for j=0 to 3
nv = Wn(j,p)
s=rnd
if (s<0.5) then nv=nv*(1+csng(s)) else nv=nv end if
pv = Wp(j,p)
s=rnd
if (s<0.5) then pv=pv*(1+csng(s)) else pv=pv end if
Wn(j,i)=nv
Wp(j,i)=pv
next j
next p

'
'
end sub
' _________________________________________________________________
'
'
sub UWCopy(Av() as single, Bv() as single, m as integer)
'
'   Copy values from Av to Bv
'
static as integer j,p

'
'  Assuming this selection criteria produces only a few changes.
'
for p=0 to m-1
for j=0 to 3
Bv(j,p) = Av(j,p)
next j
next p
'
end sub
' _________________________________________________________________
'
'
function bp_ramp(x as single) as single

'  bipolar ramp function .
static as single y
if abs(x) >1 then y=x/abs(x) else y=x end if
return y

end function
'
sub apply_activation(g1() as single)
'
'  Apply activation function to x()
'
static as integer n,i
static as single y
'
'
n=ubound(g1)
for i=0 to n
y=g1(i)
y=bp_ramp(y)
g1(i) = y
next i
'
end sub

' _________________________________________________________________
'
'
'    Next 3 functions, From another author
function round(in as double, places as ubyte = 2) as string
dim as long _mul = 10 ^ places
return str(cdbl(int(in * _mul + .5) / _mul))
End Function
'
' ----------------------------------------------------------------------
'
function max( a as double, b as double ) as double
return iif( a > b, a, b)
end function
'
' ----------------------------------------------------------------------
'
'' Sum of squared difference
' Inputs   vec() , tar
' Outputs  cost
function costL2(vec() as single, tar() as single ) as double
static as integer i
dim as double cost
for i as long= 0 to ubound(vec)-1
var e = vec(i) - tar(i)
cost += e*e
next i
return cost
end function
'
' ----------------------------------------------------------------------
'
'
'' This routine is from another author .
'' As illustrated
'' Input vec() , output vec()
sub xwht( vec() as single, hs_pow2_shl as long = 0 ) '' 1 or 2 for Partial Fast W.H.T.
static as long n , i
n = ubound(vec)+1
var hs = 1 shl hs_pow2_shl
'print " xwht, n , hs ";n;" , ";hs
while (hs < n)
var i = 0
while (i < n)
var j = i + hs
while (i < j)
var a = vec(i)
var b = vec(i + hs)
vec(i) = a + b
vec(i + hs) = a - b
i += 1
wend
i += hs
wend
hs += hs
wend
'   scale
for i=0 to n
vec(i)=vec(i)/n
next i
end sub
' _________________________________________________________________
'
'

``````
dafhi
Posts: 1619
Joined: Jun 04, 2005 9:51

### Re: Switch Net 4 neural network

will have a look, thanks. i train hyperparams to find faster network convergence, had a realization that some hyperparam sets are obviously bad so i break loop early. not quite ready for release tho

here is a great video on 'mutation' Christoph Adami - TED talk

 here's my hyperparam thing

Code: Select all

``````/'  hyperparam search for switchNet4 demo - 2023 Sep 13.1 - by dafhi

proof-of-concept hyper-parameter trainer.

my hyper-parameters represent linear interpolation

i linearly interpolate 3 tihngs for this demonstration
1. mutator size
2. swap chance
3. flip chance

the hyper-parameters are 3 variables, for each of the above:
base + percentage * delta.

so that's 9 variables in total which get mutated until fast-ish
network performance is discovered.

mutation formula:
sub hyperparam.new_val
var big_nudge = lo+rnd*(hi-lo)
var nudge = curr * (1 + .025*(rnd-.5))
curr = iif( rnd < .11, big_nudge, nudge )
clamp( curr )

you can see network performance after some hyperparameter training,
comment out  #define train_hyperparams, below

'/

#define train_hyperparams

'#include "../ba_ecs.bas"
'#include once "util.bas"
/' -- util.bas - 2023 Aug 17 - by dafhi

'/
'#include "boilerplate.bas"
/' -- boilerplate.bas - 2023 May 12 - by dafhi

+ ------------------------ +
|  freebasic  |  c++       |
+ ----------- + ---------- +
|  true = -1  |  true = 1  |
|  0.99 =  1  |  0.99 = 0  | .. i hope that covers it
+------------- ----------- +

'/

#define sng         as single
#define dbl         as double

function min( a as double, b as double ) as double
return iif( a < b, a, b)
end function

function max( a as double, b as double ) as double
return iif( a > b, a, b)
end function

function clamp( in dbl, hi dbl = 1, lo dbl = 0) dbl
return min( max(in, lo), hi ) '' June 12
End Function

union uptrs
as any ptr          a
as ubyte ptr        b
as ushort ptr       sho
as ulong ptr        ul
as ulongint ptr     uli
As Single Ptr       s
as double ptr       d
End Union

' -------------- boilerplate.bas

' ------- util.bas continued ..
#macro sw( a, b, tmp )
tmp = a: a = b: b = tmp
#endmacro
'
' -------- util.bas

' -------- hyperparams proof-of-concept continued ..
'
#include "string.bi" '' format() [rounding]

'' switch net 4
'' https://editor.p5js.org/congchuatocmaydangyeu7/sketches/IIZ9L5fzS

'' translation w/o hyperparameter trainer:
'' https://freebasic.net/forum/viewtopic.php?p=299998#p299998

type SwNet4
'' vecLen must be 4,8,16,32.....
declare sub setup( vecLen as long, as long )
declare sub recall( () as single, byref as single ptr ) '' () arrays
declare sub _abcd( () as single, as long )
declare sub _flips_backup
declare sub _flips_restore
as long   depth
as single scale
sng params( any )
as ubyte  _flips( any )
as ubyte  _flip_undo( any )
as single _a,_b,_c,_d
as long   _paramIdx, _j_base
end type

sub SwNet4.setup( vecLen as long, _depth as long)
depth = _depth
scale = 1 / sqr( vecLen shr 2 )

redim params ( 8 * vecLen * depth - 1 )
var j = 0
for i as long = 0 to ubound(this.params) step 8
this.params(i+j) = this.scale
this.params(i+4+j) = this.scale
j=(j+1) and 3
next

redim _flips( vecLen - 1 )
for  i as long=0 to vecLen-1
this._flips(i)= rnd
next
redim _flip_undo ( ubound(_flips) )

end sub

sub wht( vec() as single, hs_pow2_shl as long = 0 ) '' 1 or 2 for Partial Fast W.H.T.
dim as long n = ubound(vec)+1
var hs = 1 shl hs_pow2_shl
while (hs < n)
var i = 0
while (i < n)
var j = i + hs
while (i < j)
var a = vec(i)
var b = vec(i + hs)
vec(i) = a + b
vec(i + hs) = a - b
i += 1
wend
i += hs
wend
hs += hs
wend
end sub

sub SwNet4._abcd( result() as single, j as long )
_paramIdx += 8
dim sng x=result( j+_j_base )
#if 1
if(x<0)then
_a+=x*params(_paramIdx)
_b+=x*params(_paramIdx+1)
_c+=x*params(_paramIdx+2)
_d+=x*params(_paramIdx+3)
else
_a+=x*params(_paramIdx+4)
_b+=x*params(_paramIdx+5)
_c+=x*params(_paramIdx+6)
_d+=x*params(_paramIdx+7)
endif
#else
if(x<0)then
else
endif
#endif
_j_base += 1
end sub

sub SwNet4.recall( result() as single, byref inVec as single ptr )

for i as long = 0 to ubound(result)
result(i) = inVec[i] * (scale/9) * iif( this._flips(i)and 1, 1, -1 )
next
wht( result() )
_paramIdx = -8
for i as long = 0 to this.depth - 1
for j as long = 0 to ubound(result) step 4
_j_base = 0
_a=0:_b=0:_c=0:_d=0
_abcd result(), j
_abcd result(), j
_abcd result(), j
_abcd result(), j
result(j)=_a
result(j+1)=_b
result(j+2)=_c
result(j+3)=_d
next
const pow2_shl = 2 '' August 1
wht( result(), pow2_shl )
next
end sub

sub SwNet4._flips_backup
for i as long = 0 to ubound(_flips)
_flip_undo(i) = _flips(i)
next
end sub

sub SwNet4._flips_restore
for i as long = 0 to ubound(_flips)
_flips(i) = _flip_undo(i)
next
end sub

function costL2(vec() as single, byref tar as single ptr) as double
dim dbl cost
for i as long= 0 to ubound(vec)-1
var e = vec(i) - tar[i]
cost += e*e
next
return cost
end function

type hyperparam
declare constructor
declare constructor( sng, sng )
declare operator cast sng
declare operator cast as string
declare operator let( sng )
sng     hi, lo, curr
declare sub new_val
declare sub val_from_file
end type

constructor hyperparam
end constructor

constructor hyperparam( _lo sng, _hi sng )
hi = _hi
lo = _lo
end constructor

operator hyperparam.cast sng
return curr
end operator

operator hyperparam.cast as string
return str(curr)
end operator

operator hyperparam.let( s sng)
curr = s
end operator

sub hyperparam.val_from_file
get #1,, curr
end sub

sub hyperparam.new_val
var big_nudge = lo+rnd*(hi-lo)
var nudge = curr * (1 + .025*(rnd-.5))
curr = iif( rnd < .14, big_nudge, nudge )
curr = clamp(curr,hi,lo)
end sub

type hypers_base
declare sub       dbg( sng, as string )
declare sub       quick_vals( sng, sng, sng )
declare sub       print
declare sub       from_file
as hyperparam     bas, delt, dcay
sng               delt0 '' for copying pre-decay delta
end type

sub hypers_base.dbg( smax sng, msg as string )
if bas + delt > smax orelse dcay > 1 then
? msg
? bas + delt
? dcay
: sleep
endif
end sub

sub hypers_base.quick_vals( b sng, del sng, dcy sng )
bas = b
delt = del
dcay = dcy
end sub

sub hypers_base.print
? bas'format(bas, ".##")
? delt'format(delt0, ".##")
? dcay'format(dcay, ".###")
end sub

sub hypers_base.from_file
bas.val_from_file
delt.val_from_file
dcay.val_from_file
end sub

type hyper_set
as hypers_base   muta_size
as hypers_base   swap_chance
as hypers_base   flip_chance
dbl             cost
end type

sub backup_delta( byref t as hyper_set )
t.muta_size.delt0 = t.muta_size.delt
t.swap_chance.delt0 = t.swap_chance.delt
t.flip_chance.delt0 = t.flip_chance.delt
end sub

sub restore_decayed_delta( byref t as hyper_set )
t.muta_size.delt = t.muta_size.delt0
t.swap_chance.delt = t.swap_chance.delt0
t.flip_chance.delt = t.swap_chance.delt0
end sub

sub print_hypers( byref t as hyper_set )
? " muta_size": t.muta_size.print
? " swap_chance": t.swap_chance.print
? " flip_chance": t.flip_chance.print
end sub

type Mutator
declare constructor( as long, as long, as single )
declare sub mutate( byref as SwNet4, byref as hyper_set )
declare sub undo( byref as SwNet4 )
as long   size, precision
as single limit
as single previous( any)
as long   pIdx( any)
end type

constructor mutator( _size as long, precis as long, limit  as single )
size = _size
redim this.previous( size-1 )
redim this.pIdx( size-1 )
this.precision = precis
this.limit = limit
end constructor

sub Mutator.mutate( byref net as SwNet4, byref hypers as hyper_set )

dim sng     sc = ubound(net.params) + .499 '' c++ .999
dim as long rpos, rpos2, c
dim sng     vm, m

'' previous() and pIdx() detail a small set for mutation

for i as long= 0 to ubound( this.pidx ) ' relatively small array

'' to continue GreenInk's sub-random idea, i try swapping - dafhi
if rnd < hypers.swap_chance.bas + rnd*hypers.swap_chance.delt andalso i < size-1 _
then

rpos = rnd * sc
this.pIdx(i) = rpos '' muta location
this.previous(i) = net.params(rpos) '' save pre-mutate
i += 1
rpos2 = rnd * sc
this.pIdx(i) = rpos2 '' muta location
this.previous(i) = net.params(rpos2) '' save pre-mutate

static as typeof(net.params) tmp '' custom swap
sw( net.params(rpos), net.params(rpos2), tmp )

else

rpos = rnd * sc   ' random elem
this.pIdx(i) = rpos '' muta location
this.previous(i) = net.params(rpos) '' save pre-mutate
m = 2 * this.limit * exp(rnd*-this.precision)
vm = net.params(rpos) + iif(rnd<.5,m,-m)
if (vm > this.limit)orelse(vm < -this.limit) then continue for
net.params(rpos) = vm
endif

next

hypers.swap_chance.delt *= hypers.swap_chance.dcay '' new

'' new: mutate flips (Aug 23
net._flips_backup
sc = ubound(net._flips)+.499 '' c++ .999
c = ubound(net._flips)+1
var u = c*( hypers.flip_chance.bas + hypers.flip_chance.delt) - 1
for i as long = 0 to u
dim as long dest_index = rnd*sc
net._flips( dest_index ) = rnd
next

hypers.flip_chance.delt *= hypers.flip_chance.dcay

end sub

sub Mutator.undo( byref net as SwNet4 )

for i as long = ubound(pIdx) to 0 step -1
net.params(pIdx(i))= previous(i)
next

net._flips_restore
end sub

'#include "swnet_hypers_demo.bas"

namespace demo

'' Test with Lissajous curves
dim as ulong c1 = rgb(0,0,0),c2 = rgb(255,255,0)
dim sng ex(8,255)
dim sng work(255)
dim dbl parentCost

dim as long w,h
dim as SwNet4 parentNet

const success_frames_scalar = 1.02

sub visualize
cls ' clearscreen

locate 2,1
? "Training Data"

for i as long = 0 to 7
for j as long= 0 to 255 step 2
var y=44 + 18 * ex(i,j + 1)
pset (25 + i * 40 + 18 * ex(i,j), y), c2
next
next

locate 10,1
? "Recall"

for i as long = 0 to 7
parentNet.recall( work(), @ex(i,0) )
for j as long = 0 to  255 step 2
pset(25 + i * 40 + 18 * work(j), 104 + 18 * work(j + 1)), c2
next
next
end sub

sub random_hypers( byref t as hyper_set )
const chance = 1.
if rnd < chance then t.muta_size.delt.new_val
if rnd < chance then t.muta_size.dcay.new_val
if rnd < chance then t.muta_size.bas.new_val
if rnd < chance then t.swap_chance.delt.new_val
if rnd < chance then t.swap_chance.dcay.new_val
if rnd < chance then t.swap_chance.bas.new_val
if rnd < chance then t.flip_chance.delt.new_val
if rnd < chance then t.flip_chance.dcay.new_val
if rnd < chance then t.flip_chance.bas.new_val
end sub

sub best_to_file( byref t as hyper_set )
restore_decayed_delta t
open "hypers.txt" for output as #1
put #1,, t.muta_size.bas.curr
put #1,, t.muta_size.delt.curr
put #1,, t.muta_size.dcay.curr
put #1,, t.swap_chance.bas.curr
put #1,, t.swap_chance.delt.curr
put #1,, t.swap_chance.dcay.curr
put #1,, t.flip_chance.bas.curr
put #1,, t.flip_chance.delt.curr
put #1,, t.flip_chance.dcay.curr
write #1, ""
write #1, "size base " + str(t.muta_size.bas.curr)
write #1, "delta "+ str(t.muta_size.delt.curr)
write #1, "delta decay "+ str(t.muta_size.dcay.curr)
write #1, "swap base " + str(t.swap_chance.bas.curr)
write #1, "swap delta "+ str(t.swap_chance.delt.curr)
write #1, "swap decay "+ str(t.swap_chance.dcay.curr)
write #1, "flip base " + str(t.flip_chance.bas.curr)
write #1, "flip delta "+ str(t.flip_chance.delt.curr)
write #1, "flip decay "+ str(t.flip_chance.dcay.curr)
close
end sub

dim as double     perf
dim sng           cost_best(2)
dim as hyper_set  hypers_best

sub save_new_stats( hardening as boolean, byref t as hyper_set )

for i as long = ubound(cost_best) to 1 step -1
cost_best(i) = cost_best(i-1)
next
cost_best(0) = parentCost

restore_decayed_delta t
hypers_best = t
end sub

sub training_info( hardening as boolean )

locate 22
? "best: "; format(cost_best(0), ".##")

locate 24
if hardening then
? "hardening result from file"
else
? "bad set if > 1 -->  "; format(perf, ".##"); " "
endif
end sub

dim as long       epoch
dim as single     outer_frame, _outer_frame
dim as hyper_set  hyp(4)

sub run_epoch( byref t as hyper_set, hardening as boolean = false )

parentNet.setup 256,2

'' after a bug hunt, i realized i need to save / restore 'delta'
'' because after a run, delta has decayed and if net was good,
'' i save to hypers_best

backup_delta t

demo.parentCost = 1/0

t.muta_size.dbg 255.499, "muta size"
t.swap_chance.dbg 1, "swap"
t.flip_chance.dbg 1, "swap"

_outer_frame = outer_frame
#ifdef train_hyperparams
if not hardening then _outer_frame = outer_frame * .7 + _
rnd * outer_frame * .5
#endif

for frame as long = 1 to _outer_frame

var precision = 35
var limit = 2*parentNet.scale

'' mutator with hyperparams.  2023 Aug 8 - by dafhi
dim as Mutator mut = type( t.muta_size.bas + rnd*t.muta_size.delt, precision, limit )
t.muta_size.delt.curr *= t.muta_size.dcay.curr '' reduce max over time

for i as long = 0 to 6 '' 100 originally.  reduced for some cpu sleep
mut.mutate( parentNet, t )
dim dbl cost = 0
for j as long = 0 to 7
parentNet.recall( work(), @ex(j,0) )
cost += costL2( work(), @ex(j,0) )
next
if (cost < parentCost) then
parentCost = cost
else
mut.undo( parentNet )
endif
next

#ifdef train_hyperparams
if hardening then
visualize
else
var break_early = .8
perf = (parentCost / cost_best(0)) ^ break_early * _
(frame / _outer_frame) ^ 1.2
if perf  > .99 then parentCost = 1/0: exit for
endif
training_info hardening
#else
visualize
#endif

locate 21
?"Cost: "; ; format(parentCost, ".##"); " "
locate 19
? "frame "; frame; " of"; _outer_frame; " "

sleep 1 '' large sleep val to keep laptop cool-ish

dim as string kstr = lcase(inkey)
select case kstr
case ""
case chr(27)
end
case "z","x","c","v", " "
parentCost *= (frame / _outer_frame) ^ .5
exit for
end select

next frame

t.cost = parentCost

#ifdef train_hyperparams
if parentCost < cost_best(0) then

save_new_stats hardening, t
if not hardening then
best_to_file t
endif

visualize

if not hardening then outer_frame = _outer_frame

endif

#endif

end sub ' ---------- run_epoch

sub print_cost( byref t as hyper_set, y as long )
locate 26+y
print t.cost
end sub

sub run_array
run_epoch hyp(0)
hyp(0) = hypers_best
random_hypers hyp(0)
end sub

sub setup

dim as hyper_set ptr p = @hyp(0)

'' hyperparam ranges
p->muta_size.bas = type(4,43)
p->muta_size.delt = type(65,160)
p->muta_size.dcay = type(.91,.99999)

p->swap_chance.bas = type( .001, .4)
p->swap_chance.delt = type(.001, .5)
p->swap_chance.dcay = type(.89, .99999)

p->flip_chance.bas = type( .001, .4)
p->flip_chance.delt = type(.001, .5)
p->flip_chance.dcay = type(.88, .99999)

w = 400
h = 400

screenres w,h,32

const tau = 8*atn(1)
for i as long = 0 to 127
'' Training data
dim sng t = (i * tau) / 127
ex(0,2 * i) = sin(t)
ex(0,2 * i + 1) = sin(2 * t)
ex(1,2 * i) = sin(2 * t)
ex(1,2 * i + 1) = sin(t)
ex(2,2 * i) = sin(2 * t)
ex(2,2 * i + 1) = sin(3 * t)
ex(3,2 * i) = sin(3 * t)
ex(3,2 * i + 1) = sin(2 * t)
ex(4,2 * i) = sin(3 * t)
ex(4,2 * i + 1) = sin(4 * t)
ex(5,2 * i) = sin(4 * t)
ex(5,2 * i + 1) = sin(3 * t)
ex(6,2 * i) = sin(2 * t)
ex(6,2 * i + 1) = sin(5 * t)
ex(7,2 * i) = sin(5 * t)
ex(7,2 * i + 1) = sin(2 * t)
next

open "hypers.txt" for input as #1

p->muta_size.from_file
p->swap_chance.from_file
p->flip_chance.from_file

close

var file_found = p->muta_size.bas > 0

cost_best(0) = 1/0
#ifdef train_hyperparams
outer_frame = 659
if file_found then
var hardening = true
for i as long = 1 to 2
run_epoch *p, hardening
next
visualize
outer_frame *= .93
else
random_hypers *p
hypers_best = *p
endif
#else
outer_frame = 5499
#endif

end sub

end namespace

randomize

demo.setup

#ifdef train_hyperparams
dim as long epoch_max = 9999
#else
dim as long epoch_max = 1
#endif

for demo.epoch = 1 to epoch_max
locate 18
? "epoch "; demo.epoch; " of "; epoch_max

demo.run_array

sleep 24

next

locate 30
?"done!"

sleep

``````
Last edited by dafhi on Sep 13, 2023 12:13, edited 6 times in total.
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

The video is interesting, reminds me a little of Conway's Game of Life.
dafhi
Posts: 1619
Joined: Jun 04, 2005 9:51

### Re: Switch Net 4 neural network

i notice WHT similarity to FFT. my grasp of FFT is weak but i understand how both highlight that 'typical' NN's are overkill.

working out a variant of switch net in my head where input layer has equal size layer #2, connections mutate-shuffled
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

FFTs have some versatility , even a size 8 represents a fair amount of calculation and connectedness .

Reading what's on the internet, deep learning involves larger NN's ;
my matrix approach to NN might qualify if that's the instance.
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

In an attempt to re familiarise myself with NNs I examined some
introductory examples from the internet.
There the notion of regularisation is discussed; this can take many
forms, including normalising the weights, selectively subtracting a random
amount, nullifying certain connections.
In this instance an activation function, and its derivative were used with
back propagation.

Your approach is a little different.
dafhi
Posts: 1619
Joined: Jun 04, 2005 9:51

### Re: Switch Net 4 neural network

GreenInk described years ago the idea
1. mutate
2. keep vs. revert

and i stuck with it.

my hyper-parameters interest spans a few years. discovering optimizations and realizing oversight from previous projects.
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

There was a question about the FFT.

The FFT is similar to the WHT, except there are
complex additions and multiplications by a complex
value, known as the twiddle factor, this might be
interpreted as being like a fixed weight.
Just as the WHT has a decimation in frequency arrangement,
so does the FFT.

If you run this code you'll see that even a small FFT, or WHT,
is already considerably larger than the first NN illustrated.
The first NN was effectively trained, after 10000 iterations,
to simulate an exclusive or gate and, perhaps, that's all.
The WHT [FFT] version is almost like a deep neural network in
comparison.

There are various properties of the FFT that might be useful.
For instance, by having extra values in the last inputs that
are zero ; we accomplish interpolation at the last pass.
This provides extra, related , values for the main switched
NN ; possibly guarding somewhat against over fitting .

Code: Select all

``````

' nn2_draw2a.bas

'  Draw Neural Network structure from selection variables.

declare sub nn_illustrate(numInputs as integer,numHiddenNodes as integer,numOutputs as integer,numTrainingSets as integer)

' NN
dim as integer numInputs = 3
dim as integer numHiddenNodes = 4 ' 2
dim as integer numOutputs = 1
dim as integer numTrainingSets = 8

dim as integer numHiddenLayers=1
'
' ______________________________________________________________________
'
nn_illustrate(numInputs ,numHiddenNodes ,numOutputs ,numTrainingSets)

print"done"
sleep
end
'
' ======================================================================
'
sub nn_illustrate(numInputs as integer,numHiddenNodes as integer,numOutputs as integer,numTrainingSets as integer)
'
'  Illustrate the structure of the Neural Network .
'
dim as integer i,j
'
screen 12
locate 2,30
print" Neural Network Structure "
locate 3,4
print "Inputs"
for i=0 to numInputs - 1
locate i+4,1
print i
line(30,i*15+50)-(60,(i+1)*15+50),14,b
next i
'
' HiddenNodes
'
locate 3,12
color 11,0
print "Hidden Nodes"
for i=0 to numHiddenNodes - 1
'  locate i+4,1
'  print i
line(90,i*15+50)-(120,(i+1)*15+50),11,b
next i
'
' Outputs
'
locate 3,41
print "Outputs"
for i=0 to numOutputs-1
line(320,i*15+50)-(350,(i+1)*15+50),10,b
next i
'

'
' HiddenNodes Biases
'
locate 8,17
color 13,0
print "Hidden Nodes Biases"
for i=0 to numHiddenNodes - 1
'  locate i+4,1
'  print i
line(125,i*15+50)-(155,(i+1)*15+50),13,b
next i
'
' Outputs Biases
'
locate 8,46
print "Outputs Biases"
for i=0 to numOutputs-1
line(355,i*15+50)-(385,(i+1)*15+50),13,b
next i
'
'
'  Training sets .
'
for j=0 to numTrainingSets-1
locate 13,j*5+5
print j
for i=0 to numInputs - 1
locate i+14,1
print i
line(30+j*40,i*15+210)-(60+j*40,(i+1)*15+210),14,b
next i

next j
'
'
dim as integer k
k= numInputs*15+15
for j=0 to numTrainingSets-1
' locate 13+numInputs,j*5+5
' print j
for i=0 to numOutputs - 1
locate numInputs+15,1
print i
line(30+j*40,k+i*15+210)-(60+j*40,k+(i+1)*15+210),10,b
next i
next j
'
locate 12,6
print " Training Set"
line(25,170)-( numTrainingSets*40+30,(numInputs+1+numOutputs)*15+5+210),7,b

'
locate 30,54
color 12,0
print " press any key to continue"
sleep
'
'
'   Structure of length 8 fft , or wht .
'
cls
dim as integer n

n=8
locate 2,30
color 15,0
print" FFT or WHT Structure, N=8 "
locate 3,4
print "Inputs"
for i=0 to n - 1
locate i+4,1
print i
line(30,i*15+50)-(60,(i+1)*15+50),14,b
next i
'
locate 3,10
color 10,0
print " Bit Reversed"
for i=0 to n - 1
' locate i+4,1
' print i
line(65,i*15+50)-(95,(i+1)*15+50),10,b
next i
'
locate 12,14
color 11,0
print " Pass 1"
for i=0 to n - 1
line(120,i*15+50)-(150,(i+1)*15+50),11,b
next i
'
locate 12,22
color 11,0
print " Pass 2"
for i=0 to n - 1
line(175,i*15+50)-(205,(i+1)*15+50),11,b
next i
'
locate 12,29
color 11,0
print " Pass 3"
for i=0 to n - 1
line(230,i*15+50)-(260,(i+1)*15+50),11,b
next i
'
locate 3,38
color 15,0
print"Switched NN "
i=0
line(290,i*15+50)-(380,(n)*15+50),15,b

'
locate 30,54
color 12,0
print " press any key to continue"
sleep

dim as ulong SCREEN_EXIT=&h80000000
Screen 0, , , SCREEN_EXIT' (with SCREEN_EXIT=&h80000000)
' The previous command returns to native console/terminal window.

'screen
'
end sub

``````
dafhi
Posts: 1619
Joined: Jun 04, 2005 9:51

### Re: Switch Net 4 neural network

xor gate .. yeah okay. one of my past projects i attempted 'all the gates'

input

00
01
10
11 .. sparsenet

..  ..

my hyper-parameter comments need updating. linear interp on its own could be replaced by one variable.
my interp is .. base + [decaying] delta
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

This code will illustrate some of the concepts I perceive
as being used within switchnet4.

The switch section selectively uses the positive or negative
Neural Network weights. These weights reside within banks of size
four.

The main banks contain the Neural Network weights for each layer,
separated into negative [ cyan ] and positive [ red ] banks.
The stored banks are copied from the main banks.
Then conditionally written back to the main banks, depending upon
the difference between the output values and the expected output values.

The section labeled "extra" is to be determined.

Now I have a few questions.
How much of the main banks is randomly altered, is it just a few banks
of size four; or all of the main banks.
Are these alterations dependent upon whether the positive or negative
banks in use at any particular epoch.

In the code for SwNet4(), firstly wht( result() ) is used, then
after some calculations, wht( result(), pow2_shl ) is used.
What is the effect of this, are we sampling from a larger data set.

Code: Select all

``````

'  sciwiseg@gmail.com

'   Rectangle drawing

screen 12
window screen (0,0)-(100,100)

type pt
x as single
y as single
end type

dim as integer n,i,cm,j
n=64

dim pts(0 to n-1) as pt
dim chrome(0 to n-1) as integer
'
'
restore datax
for i=0 to n-1
next i
restore datay
for i=0 to n-1
next i
restore colorp
for j=0 to (n/2)-1
next j
'
'  Display connected pts
'
dim as single x,y,x1,y1
'
'  Line [target,] [[STEP]|(x1, y1)]-[STEP] (x2, y2) [, [color][, [B|BF][, style]]]
' or
'  Line - (x2, y2) [, [color][, [B|BF][, style]]]
'

for i=0 to n-1 step 2 'n-1
x=pts(i).x
y=pts(i).y
cm=chrome(int(i/2))
x1=pts(i+1).x
y1=pts(i+1).y
if x1 > 0 and x > 0 then line(x,y)-(x1,y1),cm,b

next i
'
' ______________________________________________________________________
'
color 10,0
locate 2,2
print "in";
color 13,0
locate 2,8
print "wht"
color 14,0
locate 2,12
print " switch"
color 15,0
locate 2,21
print "main banks"
color 6,0
locate 2,32
print " extra"
color 13,0
locate 2, 40
print "wht"
color 9,0
locate 2,46
print "out"
color 8,0
locate 18,3
print "stored banks"
color 11,0
locate 14,50
print "- banks"
color 4,0
locate 15,50
print "+ banks"

' within SwNet4()
' wht( result() )
' wht( result(), pow2_shl )
color 13,0
locate 12,1
print "wht(result())"
locate 12,38
print "wht(result(), pow2_shl )"
sleep
end
'
' ======================================================================
'
' ----------------------------------- x --------------------------------
'
datax:
'input
data 2,5
' wht
data 7,15
' switch
data 17,20
' block
data 22,39
' - bank
data 24,27,24,27,29,32,29,32,34,37,34,37
' + bank
data 24,27,24,27,29,32,29,32,34,37,34,37
' xtra
data 41,44
' wht 2
data 46,54
' output
data 56,59
'
' -------------------------- banks 2 ----------------------------------
'
' block
data 22,39
' - bank
data 24,27,24,27,29,32,29,32,34,37,34,37
' + bank
data 24,27,24,27,29,32,29,32,34,37,34,37
'
' ------------------------------ y ------------------------------------
'
datay:
' input
data 10,30
' wht
data 10,30
' switch
data 10,52
' block
data 8,53
' - bank
data 10,18,21,30,10,18,21,30,10,18,21,30
' + bank
data 32,40,43,51,32,40,43,51,32,40,43,51
' xtra
data 10, 52
'  wht 2
data 10,30
' output
data 10,30
'
'  ................... banks 2 ......................
'
' block
data 55,100
' - bank
data 57,65,68,76,57,65,68,76,57,65,68,76
' + bank
data 79,87,90,98,79,87,90,98,79,87,90,98
'
' ------------------------- colour -------------------------------------
'

colorp:
data 10,13,14,15,11,11,11,11,11,11,4,4,4,4,4,4,6,13,9
data 8,11,11,11,11,11,11,4,4,4,4,4,4

``````
dafhi
Posts: 1619
Joined: Jun 04, 2005 9:51

### Re: Switch Net 4 neural network

i haven't coded a wht from scratch. my version with pow2_shl is an all-in-one, after examining GreenInk's .. yes i did note he used different wht at different stage. it's cool you made an illustration; i may stare at it a bit longer, i code from inspiration.

in GreenInk's diagram, the switches seem to reroute connections, whereas in the lissajous curve demo, switches flip values.

also, GreenInks demo initializes said flip values a only reads later. i'm often mis-analyze questions and guess mis-matched inspiration behind other peoples ideas, so bear with.

also, one reason i see WHT potential is it doesn't use multiply. there are also potential single thread parallelism through masking

for example in graphics you can mull Red and Green in one go
result = color * FF00FF

for a rate i suggest 28%
Luxan
Posts: 205
Joined: Feb 18, 2009 12:47
Location: New Zealand

### Re: Switch Net 4 neural network

I'm glad the diagram is of interest, and possible use, to you; feel free to construct your own
diagrams with the Rectangle Drawing code.
Ah, I forgot to illustrate the sign flipping portion of the diagram.

As the WHT doesn't use multiply, or complex numbers, it's fairly easy to implement .

I definitely need to sructinize the original code and also unravel what Greenlnk was intending.