Improved Results from New Simulation (May 10, 2006)

The neural network simulation program was modified to be a more accurate simulation. Rather than starting with a 100%-connected network and ending with a 90%-connected network, a more realistic simulation of keeping the synapse count close to 20% at all times was completed. The algorithm moved (instead of removed) synapses based on their weights, to adapt the network during the learning process.

This showed poor results in terms of learning speed, but surprisingly, when generalization capabilities was tested using an alternate dataset selected from the same population of the training data, the dynamic re-routing of synapses greatly improved generalization performance over both the 20%-connected network and the 100%-connected network.

Learning Performance

Starting at Cycle 200


Starting at Cycle 800


Generalization Performance
(using dataset selected at random from all data)


Starting at Cycle 200


Starting at Cycle 800


Original Results

tanh Sigmoid simulation over a common-mode range of 0.5 to 1.5 V

sech2 Sigmoid Derivative simulation over a common-mode range of 0.5 to 1.5 V

Gilbert multiplier simulation, with common-mode range limited to 2-3V, and input swing of 300mV

Performance of Neural Learning Methods



For the neural network simulations, plots were constructed for the entire data, as well as for cycles 200 to 1000, the mainly linear segment, and for cycles 800-1000, in which the methods converge (See graphs).

Graphs of the observed and predicted MSE (Mean Square Error) for each of the 3 methods against cycle number, clearly show that the 20% partially-connected learning method performs more poorly than both the fully-connected method and the dynamically re-routed network. This is supported by the regression analyses (see Stata log) with P-values for the intercept and slope comparisons between both the 20% connected network and the fully-connected, and the 20%-connected and the dynamically re-routed network being <0.0001, both for the 200-1000 and 800-1000 cycles windows. There was no significant differences between the fully-connected network and the dynamically re-routed network models for the 200-1000 cycles window, but the dynamically re-routed network intercept was significantly higher than the fully-connected intercept in the 800-1000 cycles window.

However, over all the data, the dynamically re-routed method had significantly fewer mean total synapses (hidden + output) than the fully-connected method, with a two-sample t-statistic of 55.47 (P<0.0001). The fully-connected network had a mean of 5500 (SD 0) synapses whereas Method 2 had 5283.03 (SD 98.12).

Copyright © Malcolm Stagg 2006. All Rights Reserved.
Website: http://www.virtualsciencefair.org/2006/stag6m2. Email: malcolmst@shaw.ca.