PIKING EURAL ETWORKS - CWI Amsterdam | Research in mathematics and

148 Pages·2003·4.59 MB·Dutch

SPIKING NEURAL NETWORKS

SPIKING NEURAL NETWORKS
PROEFSCHRIFT
terverkrijgingvan
degraadvanDoctoraandeUniversiteitLeiden,
opgezagvandeRectorMagni(cid:2)cusDr. D.D.Breimer,
hoogleraarindefaculteitderWiskundeen
Natuurwetenschappen endiederGeneeskunde,
volgensbesluitvanhetCollegevoorPromoties
teverdedigenopwoensdag5Maart2003
teklokke14.15uur
door
SanderMarcelBohte
geborenteHoorn(NH)
in1974
Promotiecommisie
Promotores: Prof. Dr. J.N.Kok
UniversiteitLeiden
Prof. Dr. Ir. J.A.LaPoutre·
CWI/TechnischeUniversiteitEindhoven
Referent: Dr. S.Thorpe,D.Phil(Oxon)
CNRSCentredeRechercheCerveauetCognition
Toulouse,France
Overigeleden: Prof. Dr. W.R.vanZwet
Prof. Dr. G.Rozenberg
Prof. Dr. H.A.G.Wijshoff
The work in this thesis has been carried out under the auspices of the re-
searchschoolIPA(InstituteforProgrammingresearchandAlgorithmics).
Work carried out at the Centre for Mathematics and Computer Sci-
ence(CWI),Amsterdam.
SpikingNeuralNetworks
SanderMarcelBohte.
ThesisUniversiteitLeiden. -Withref.
ISBN90-6734-167-3
c 2003,SanderBohte,allrightsreserved.
(cid:176)
LayoutkindlyprovidedbyDr. DickdeRidder,Delft.
(cid:147)LOYALTY TO PETRIFIED
OPINION NEVER YET BROKE A CHAIN OR FREED A HUMAN SOUL.(cid:148)
(cid:150) MARK TWAIN

C
ONTENTS
1. Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1
1.1 Arti(cid:2)cialNeuralNetworks . . . . . . . . . . . . . . . . . . . 1
1.2 Computingwithasynchronousspike-times . . . . . . . . . . 4
2. Unsupervised Clustering with Spiking Neurons by Sparse Tem-
poralCodingandMulti-LayerRBFNetworks : : : : : : : : : : 11
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Networksofdelayedspikingneurons . . . . . . . . . . . . . 13
2.3 Encodingcontinuousinputvariablesinspike-times . . . . . 18
2.4 ClusteringwithReceptiveFields . . . . . . . . . . . . . . . . 20
2.4.1 Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.2 Scalesensitivity . . . . . . . . . . . . . . . . . . . . . . 21
2.4.3 Clusteringofrealisticdata . . . . . . . . . . . . . . . . 23
2.5 Hierarchicalclusteringinamulti-layernetwork . . . . . . . 26
2.6 Complexclusters . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.7 DiscussionandConclusions . . . . . . . . . . . . . . . . . . . 30
3. Error-Backpropagation in Temporally Encoded Networks of
SpikingNeurons : : : : : : : : : : : : : : : : : : : : : : : : : : : 33
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 Error-backpropagation . . . . . . . . . . . . . . . . . . . . . . 35
3.3 TheXOR-problem . . . . . . . . . . . . . . . . . . . . . . . . 39
3.3.1 Errorgradientandlearningrate . . . . . . . . . . . . 41
3.4 OtherBenchmarkProblems . . . . . . . . . . . . . . . . . . . 42
3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4. A Framework for Position-invariant Detection of Feature-
conjunctions : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 51
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 LocalComputationwithDistributedEncodings . . . . . . . 55
viii CONTENTS
4.2.1 Architecture. . . . . . . . . . . . . . . . . . . . . . . . 55
4.2.2 Neuraldata-structure. . . . . . . . . . . . . . . . . . . 58
4.2.3 LocalFeatureDetection. . . . . . . . . . . . . . . . . . 58
4.2.4 LocalFeatureBinding. . . . . . . . . . . . . . . . . . . 58
4.2.5 Conjunctiondetection. . . . . . . . . . . . . . . . . . . 59
4.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5. FormalSpeci(cid:2)cationofInvariantFeature-conjunctionDetection 73
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2 FormalDescription . . . . . . . . . . . . . . . . . . . . . . . . 74
5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6. The effects of pair-wise and higher order correlations on the (cid:2)r-
ingrateofapost-synapticneuron : : : : : : : : : : : : : : : : : 79
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6.2 MathematicalSolutionofthethree-neuronproblem . . . . . 83
6.3 CalculatingtheDistributionwithNIdenticalNeurons . . . 84
6.4 Anarti(cid:2)cialneuralnetwork . . . . . . . . . . . . . . . . . . . 91
6.4.1 Theneuronmodel . . . . . . . . . . . . . . . . . . . . 91
6.4.2 NetworkSimulations . . . . . . . . . . . . . . . . . . . 93
6.4.3 Firing-rateofapost-synapticneuron . . . . . . . . . . 97
6.4.4 Estimation of the N-cluster distribution by entropy
maximization . . . . . . . . . . . . . . . . . . . . . . . 99
6.4.5 Theeffectsofvaryingbin-widthonthemaximalen-
tropydistribution. . . . . . . . . . . . . . . . . . . . . 99
6.4.6 Theeffects ofnetworkscaling . . . . . . . . . . . . . . 103
6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
7. TheBiologyofSpikingNeurons : : : : : : : : : : : : : : : : : : 109
7.1 RealNeuronsSpike . . . . . . . . . . . . . . . . . . . . . . . 109
7.2 PrecisionandReliabilityofRealSpikes . . . . . . . . . . . . 111
Publications : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 117
Bibliography : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 119
Samenvatting : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 129
CurriculemVitae : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 134
1
I
NTRODUCTION
1.1 Arti(cid:2)cialNeuralNetworks
Arti(cid:2)cial neural networks attempt to understand the essential computa-
tions that take place in the dense networks of interconnected neurons
making up the central nervous systems in living creatures (see also (cid:147)On
Networks of Arti(cid:2)cial Neurons(cid:148)). Originally, McCulloch and Pitts (1943)
proposed a model based on simpli(cid:2)ed (cid:147)binary(cid:148) neurons, where a single
neuron implements a simple thresholding function: a neuron’s state is ei-
ther (cid:147)active(cid:148) or (cid:147)not active(cid:148), and this is determined by calculating the
weightedsumofthestatesofneuronsitisconnectedto. Forthispurpose,
connectionsbetweenneuronsaredirected(fromneuronitoneuronj),and
haveaweight(w ). Iftheweightedsumofthestatesoftheneuronsicon-
ij
nectedtoaneuronj exceedssomethreshold,thestateofneuronj issetto
active,otherwiseitisnot.
Remarkably,networksofsuchsimple,connectedcomputationalelements,
can implement a range a mathematical functions relating input states to
output states, and, with algorithms for setting the weights between neu-
rons,thesearti(cid:2)cialneuralnetworkscan(cid:147)learn(cid:148)manysuchfunctions.
However,thelimitationsoftheseearlyarti(cid:2)cialneuralnetworksweream-
plyrecognized,i.e. seeMinskyandPapert(1969). Toalleviatetheseissues,
theoriginalbinarythresholdingcomputationintheneuronhasoftenbeen
replaced by the sigmoid: the sum of the weighted input into a neuron is
mappedontoarealoutputvalueviaasigmoidaltransformation-function,
thus creating a graded response of a neuron to changes in its input. Ab-
stracted in this transformation-function is the idea that real neurons com-
municate via (cid:2)ring rates: the rate at which a neuron generates action po-
tentials(spikes). Whenreceivinganincreasingnumberofspikes,aneuron
isnaturallymorelikelytoemitanincreasingnumberofspikesitself.
2 INTRODUCTION
OnNetworksofArti(cid:2)cialNeurons
The human brain consists of an intricate
webofbillionsofinterconnectedcellscalled
(cid:147)neurons(cid:148). The study of neural networks
in computer science aims to understand
how such a large collection of connected el-
ements can produce useful computations,
suchasvisionandspeechrecognition.
A (cid:147)real(cid:148) neuron receives pulses from many
otherneurons. Thesepulsesareprocessedin
a manner that may result in the generation
ofpulsesinthereceivingneuron, whichare
then transmitted to other neurons ((cid:2)g. A).
The neuron thus (cid:147)computes(cid:148) by transform-
inginputpulsesintooutputpulses.
Arti(cid:2)cial Neural Networks try to capture
the essence of this computation: as de-
picted in (cid:2)gure B, the rate at which a neu-
ron (cid:2)res pulses is abstracted to a scalar
(cid:147)activity-value(cid:148), or output, assigned to the
neuron. Directional connections determine
which neurons are input to other neurons.
Each connection has a weight, and the out-
put of a particular neuron is a function of
the sum of the weighted outputs of the neurons it receives input from. The
applied function applied is called the transfer-function, F(§). Binary (cid:147)thresh-
olding(cid:148) neurons have as output a (cid:147)1(cid:148) or a (cid:147)0(cid:148), depending on whether or not
the summed input exceeds some threshold. Sigmoidal neurons apply a sig-
moidal transfer-function, and have a real-valued output (inset (cid:2)g. B, solid
resp. dottedline). Neuralnetworksaresetsofconnectedarti(cid:2)cialneurons. Its
computational power is derived from clever choices for the values of the con-
nection weights. Learning rules for neural networks prescribe how to adapt
theweightstoimproveperformancegivensometask. Anexampleofaneural
network is the Multi-Layer Perceptron (MLP, (cid:2)g. C). Learning rules like error-
backpropagation (Rumelhart et al., 1986) allow it to learn and perform many
tasksassociatedwithintelligentbehavior,likelearning,memory,patternrecog-
nition,andclassi(cid:2)cation(Ripley,1996;Bishop,1995).
With the introduction of sigmoidal arti(cid:2)cial neurons, and learning rules
for training networks consisting of multiple layers of neurons (Werbos,
1974;Rumelhartet al.,1986),someofthede(cid:2)cienciesoftheearlierneural
networks were overcome: the most prominent example was the ability to
learntocomputetheXORfunctioninanarti(cid:2)cialneuralnetwork.