Local dynamic modeling with self-organizing feature maps

136 Pages·1996·5.1 MB·English

by  Wang, Ludong

LOCALDYNAMICMODELING
WITHSELF-ORGANIZINGFEATUREMAPS
By
LUDONGWANG
ADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOL
OFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHE
REQUIREMENTSFORTHEDEGREEOF
DOCTOROFPHILOSOPHY
UNIVERSITYOFFLORIDA
1996
ACKNOWLEDGEMENTS
IwouldliketothankDr.JoseC.Principe,mymajoradvisor,forhisguidance,
support,motivation,andpatience,andforhelpwiththecompletionofthisdissertation.I
amextremely grateful tothe members ofmy supervisory committee. Dr. Donald G.
Childers,Dr.FredrickJ.Taylor,Dr.JohnM.M.Anderson,Dr.KermitN.Sigmon,Dr.Paul
E.Ehrlich,fortheircontributionstomyresearchandtothepreparationandreviewofthis
manuscript.ImustalsothankmanyCNELfellowstudentswhohaveenrichedmylife
duringthisprogram.SpecialthanksgotoJohnFisherandSamelCelebi.
Thanksareduetomyfamilyfortheirencouragement.Finally,Iwouldliketothank
mylovingwife.Hong,forhersupport,andbeautifuldaughter,Amy,formakingmylife
special.
ii
TABLEOFCONTENTS
page
ACKNOWLEDGEMENTS
ii
ABSTRACT vi
CHAPTERS
1 INTRODUCTION 1
1.1Motivations 1
1.2ResearchOutlineandObjectives 2
1.3OverviewoftheDissertation 5
2 NONLINEARDYNAMICMODELING 6
2.1DynamicSystemsandSignals 7
2.2SignalAnalysis:LinearandNonlinearModels 8
2.2.1LinearModelsandLimitations 8
2.2.2AnalysisofLinearandNonlinearSystems 10
2.3NonlinearDynamicModeling 11
2.3.1GlobalModels 13
2.3.2LocalModels 15
2.4DynamicalInvariantsinModeling 17
2.4.1DynamicsReconstructionandCorrelationDimension 18
2.4.2TheLyapunovExponents 19
2.4.3ModelValidation 21
3 SELF-ORGANIZINGFEATUREMAP 23
3.1Introduction 23
3.2FormationofSOFM 24
iii
3.2.1Structures 24
3.2.2TheAlgorithmofKohonenModel 26
3.2.3Convergence 29
3.3LocalizedNeuralRepresentationofSignals 31
3.3.1PropertiesofSOFM 31
3.3.2SimulationsofSOFMwithTemporalSignalProcess 39
3.4ApplicationPotential 44
4 NON-LINEARTIMESERIESMODELINGWITHSOFM 47
4.1Introduction 47
4.2State-DependentARModels 48
4.2.1NonlinearAutoregressiveModels 48
4.2.2State-DependentPredictionofNonlinearProcesses 50
4.3LocalLinearApproximationofGlobalDynamicsandLimitation ... 51
4.4SOFM-BasedLocalLinearModelingNetworks 55
4.4.1Methodology 55
4.4.2Architecture 56
4.4.3NetworkConstruction 57
5 METHODOLOGYOFLEARNINGEQUATIONS 60
5.1PropertiesofSOFM-BasedLocalModeling 60
5.2ImprovedEstimationofLocalLinearModels 62
5.3Dynamic-OrientedRepresentations 63
5.3.1TheConstraintsofSOFMLearningProcess 64
5.3.2DynamicLearningRule 65
5.4TheWeightedLeast-SquaresSolution 69
6 EXPERIMENTALRESULTS 73
6.1ModelingofNumericallyGeneratedTimeSeries 75
6.1.1Mackey-GlassTimeSeries 77
6.1.2LorenzTimeSeries 88
6.1.3ApproximationofSeamlessPatchingofLocalModels 95
6.2ConsistencyoftheConstructedModels 98
6.2.1Consistencyvs.DifferentInitialSOFMStates 98
6.2.2TemporalConsistency 102
6.3ModelingReal-WorldSignals 104
6.3.1LaserTimeSeries 105
iv
6.3.2EEGSignal 108
6.3.3SunspotTimeSeries 108
7 CONCLUSIONSANDFUTURERESEARCH 114
7.1Conclusions 114
7.2FutureResearch 117
REFERENCES 119
BIOGRAPHICALSKETCH 127
V
AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridain
PartialFulfillmentoftheRequirementsfortheDegreeofDoctorofPhilosophy
LOCALDYNAMICMODELING
WITHSELF-ORGANIZINGFEATUREMAP
By
LudongWang
December1996
Chairman:Dr.JoseC.Principe
MajorDepartment:ElectricalandComputerEngineering
Chaotic signals are associated with autonomous response ofcertain nonlinear
dynamicalsystems.Whiletheyaredeterministicwithfewdegreesoffreedom,chaotic
signalsarenotpredictableinthelongterm.Theyclearlyhaveconsiderablymorestructure
thancanbeinferredfromandexploitedbytraditional stochasticmodelingtechniques.
Consequentlyitisimportanttodevelopnewsignalprocessingtechniquesthatarematched
tothespecialcharacteristicsofthisclassofsignals.
Inthisresearch,afinitesetoflinearlocalmodelsisestablishedasafeasibleand
effectiveimplementationofdynamicmodeling.Asaself-organizingfeaturemap(SOFM)
isusedasthemodelinginfrastructure, the method iscalled SOFM-based local linear
modelingnetwork.
Thepreviousworkonlocalmodelsonly focusesoninterpolatinglocaldatafor
short-termprediction.Anexampleisstate-dependentfunctionalmappingwhichdoesnot
considerdeducingtheequationsofmotion.Theaimoftheproposedmodelingmethodis
toderiveafinitesetoflocallinearmodelstoapproximatetheglobaldynamics.These
modelsareconstructedwithspatialconstraintswhilematchingthedynamicsofasignalin
vi
thetemporalsense.Thisspatial-temporalarchitecturerequiresfewerassumptionsaboutthe
underlyingdynamics.
The SOFM constructed with Kohonen's learning law is a localized neural
representationofsignals.WeappendedanewlinearlayertotheSOFMwhichistrainedto
approximatethetangentspacetothedynamicsrepresentedbythelocalneuralfield.The
useofaneighborhoodfunctionimposesastrongstatisticalconstraintovertheconverged
neuralfield,suchthattheirregularspacingoflocaldatacanbesmoothedout.Thisis
significanttothelocalmodelingmethodasagivensignalisusuallyoffinitelength.Based
onthisstructure,theconstructedlocallinearmodelsarenottotallyindependentofeach
other, whichhelps toreduce the discontinuity between them. On the otherhand, the
statisticaldensitymatchingpropertymakesSOFMagoodvectorquantizationprocedure.
Thisisnotanoptimalstructurefortheoverallpurposeofdynamicmodeling.Thereforea
modifiedKohonenlearningruleisproposedasanewtrainingprocedureoftheemployed
SOFM.Asthepredictionerrorisinvolvedtoreflectthelocaldynamicfluctuation,this
procedureisthuscalleddynamiclearning.Withthisprocedure,theconvergedneuralfield
becomesadynamic-orientedrepresentationofthesignal.
TheproposedSOFM-basedmodelingschemehasbeenappliedtobothsynthetic
andreal-worldsignals.Experimentsdemonstratethatthismethodiscapableoffaithfully
capturingtheunderlyingdynamicsofchaoticsignals.
Theoverallstructureoftheproposedmethodisasignificantextensionofthelocal
linearmodelingtechniques. It also representsa new researchdirectiontoexplore the
potentialofthevigorousself-organizingfeaturemap.
vii
CHAPTER
1
INTRODUCTION
1.1Motivations
Temporalpatternrecognition,systemcontrol,signalprediction,andchaoticdata
analysisshareacommonproblem:deducingequationsofmotionfromobservationsof
time-dependentdata.Eachofthemseekstomodelthephysicalworldwithacertaingoalin
mind. These models encapsulate the data complexity in a compact, algorithmic
specificationproducingavastdatareduction.Thereforemodelingcanbedefinedasa
processtoextracttheunderlyingdynamicmodelfromagivensignal.
Inconventionalsignalprocessingawiderangeofapproacheshavebeendeveloped
formodelingsignalsthataredeterministicandpredictable.Thesignalsaregenerallytaken
astherealizationofsomestochasticprocess,andtheunderlyingsystemismodeledas
linear[92].Insuchamodelingscheme,therandominputproducestheuncertaintyinthe
systemoutput.
Chaoticsignalsareassociatedwiththeautonomousresponseofcertainnonlinear
dynamical systems. Without any random input, the output ofsuch systems exhibits
complex behavior and possesses noise-like spectra. These systems, named chaotic
dynamicalsystems[27],areubiquitous,e.g.,laserbeams,biologicalsystems,astronomical
motions,weatherpatterns,fluidflows,andelectriccircuits,etc.[20][32][67][93].Asthe
uncertaintyinitsoutputoriginatesfromthesystemdynamicsinsteadofanexternaldriving
force,theyclearlypossessconsiderably more structurethancan be inferredfromand
exploitedbytraditionalstochasticmodelingtechniques.
Tomodelthechaotictimeseries,manytechniqueshavebeendeveloped.Theycan
generallybeclassifiedasglobalandlocalmodels.Globalmodelsareconstructedonceand
1
2
fitallthestatespace,whilelocal modelsrepresent local regions. Structureinchaotic
attractors tends to be very intricate and nonuniform. Depending on the functional
representation, global models have difficulty mimicking such systems adequately.
Localizedrepresentationscanchartoutthenuancesofchaoticmorphologyandprovidean
atlasfortheentirestatespace.Amongthelocalmethods,linearmodelsareprovedtobe
computationallycost-effective.
Theself-organizingfeaturemap(SOFM)constructedwithKohonen'slearninglaw
[52]isnormallyutilizedforfeatureextraction. Itischaracterizedbytheformationofa
topographicmapoftheinputpatterns,inwhichthespatiallocationoftheneuronsinthe
outputneuralfieldcorrespondtointrinsicfeaturesoftheinputpatterns.Asacomputational
map,itconstitutesabasicbuildingblockforinformationprocessing.Thefeaturemapso
derivedprovidesabetterrepresentationofthelocalinformation.
Thus,exploringtheapplicationoflocallinearmodelsinglobaldynamicmodeling,
andutilizingSOFMasthemodelinginfrastructureforthispurpose,composethemotiva-
tionofthisresearch.
1.2ResearchOutlineandObjectives
So farthe research on the method and application oflocal linear models is
restrictedtotheforecastingofchaoticsignals.Inthisscenario,alinearpredictivemodelis
estimatedasthelocaldynamicmapbasedonthenearestneighborsofthecurrentstate,
andthepredictionisthusobtainedbysimplefunctionevaluation.Duetothefactthatcha-
otictimeseriesisnotpredictableinthelong-termsense,smallshort-termpredictionerror
isthegoalofthisscenario.Undertheassumptionthatthestatedynamicsissmoothand
sufficientdatasamplesareavailable,thismethodcangenerallyprovideagoodpredicting
performance.
Capturingthedynamics,i.e.,thelongtermbehaviorofthesystem,isdefinitelya
significantextensionofthelocallinearforecastingscenario.Thegoalistofitasetoflocal
linearmodelstothegivensignalreconstructedinthestatespacesuchthatautonomous
outputofthe constructed structure exhibits similardynamic behavior. However, such
attemptshavebeenhinderedbysomeconcerns[23].Dependingonthepartitioningand
scales,piecewiselinearequationsofmotionmayexhibitperiodicbehaviorwhentheorig-
inaldynamicsischaoticandviceversa.Piecewiselineardynamicsareconsideredasvio-
latingthephysicallymotivatedhypothesisofsmoothdynamicalsystems.Mostphysical
processesdonotexhibitarbitrarilyfastchangesintheirfirstderivatives.Moreoverthe
establishedsetoflocallinearmodelsgenerallymaynotbeadiscontinuousfunctional
map,whichcanleadtoundesiredbehavioritisiteratedasanautonomoussystem.
Inthisresearch,theaboveconcernsarereconsideredinthefollowingaspects.As
longasthelocallinearmodelscanexhibitchaoticbehaviorwhentheoriginaldynamicsis
periodicasshowedintheinvestigationbyCrutchfield,etal.[23],itisreasonabletoexpect
thatthestructureoflocallinearmodelsiscapableofpossessingchaoticdynamics.Inthe
contextofchaoticdynamics,thelackofsmoothcontinuationmaydegradetheshort-term
predictionperformance.However,itisnotsufficienttoconcludethatafaithfulapproxima-
tionofglobaldynamicsisimpossibleintermsofafinitesetoflocallinearmodels.Finally
thediscontinuityproblemmaybereducedtoasufficientextent,ifnoteliminated,such
thattheestablishedlocallinearmodelspiecedtogetherasawholemaynotbehinderedto
approximatetheoriginaldynamics.
Torealizethescenariooflocallinearmodeling,itisproposedheretoutilizethe
SOFMasthemodelinginfrastructure.WhileSOFMisalocalizedneuralrepresentationof
theinput,itisconstructedintheglobalsenseviathecompetitivelearningprocess.The
locallinearmodelsaredirectlyfittedfromtheconstructedSOFMmap.Twomajorissues
willbeinvestigated:a)CantheneuronsofeachlocalsegmentintheSOFMneuralfield
collectivelybearmorereliableinformationaboutthelocaldynamics?b)Canall local
dynamicsextractedfromtheindividuallocalsegmentscollectivelyrepresenttheglobal
dynamicsofthegivensignal?