Neural-Based Orthogonal Data Fitting: The EXIN Neural Networks (Adaptive and Learning Systems for Signal Processing, Communications and Control Series)

277 Pages·2010·3.91 MB·English

by  Giansalvo Cirrincione

NEURAL-BASED
ORTHOGONAL
DATA FITTING

NEURAL-BASED
ORTHOGONAL
DATA FITTING
The EXIN Neural Networks
Giansalvo Cirrincione
Universite´ de Picardie–Jules Verne
Amiens, France
Maurizio Cirrincione
Universite´ de Technologie de Belfort–Montbe´liard
Belfort, France
AJOHNWILEY&SONS,INC.,PUBLICATION
Copyright2010byJohnWiley&Sons,Inc.Allrightsreserved.
PublishedbyJohnWiley&Sons,Inc.,Hoboken,NewJersey.
PublishedsimultaneouslyinCanada.
Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedinanyform
orbyanymeans,electronic,mechanical,photocopying,recording,scanning,orotherwise,exceptas
permittedunderSection107or108ofthe1976UnitedStatesCopyrightAct,withouteithertheprior
writtenpermissionofthePublisher,orauthorizationthroughpaymentoftheappropriateper-copyfee
totheCopyrightClearanceCenter,Inc.,222RosewoodDrive,Danvers,MA01923,[phone],
fax[phone],oronthewebatwww.copyright.com.RequeststothePublisherforpermission
shouldbeaddressedtothePermissionsDepartment,JohnWiley&Sons,Inc.,111RiverStreet,
Hoboken,NJ07030,[phone],fax[phone],oronlineat
http://www.wiley.com/go/permission.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbestefforts
inpreparingthisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyor
completenessofthecontentsofthisbookandspecificallydisclaimanyimpliedwarrantiesof
merchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedbysales
representativesorwrittensalesmaterials.Theadviceandstrategiescontainedhereinmaynotbe
suitableforyoursituation.Youshouldconsultwithaprofessionalwhereappropriate.Neitherthe
publishernorauthorshallbeliableforanylossofprofitoranyothercommercialdamages,including
butnotlimitedtospecial,incidental,consequential,orotherdamages.
Forgeneralinformationonourotherproductsandservicesorfortechnicalsupport,pleasecontactour
CustomerCareDepartmentwithintheUnitedStatesat(800)762-2974,outsidetheUnitedStatesat
[phone]orfax(317)572-4002.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprint
maynotbeavailableinelectronicformats.FormoreinformationaboutWileyproducts,visitourweb
siteatwww.wiley.com.
LibraryofCongressCataloging-in-PublicationData:
Cirrincione,Giansalvo
Neural-basedorthogonaldatafitting:theEXINneuralnetworks/GiansalvoCirrincione,
MaurizioCirrincione
Includesbibliographicreferencesandindex.
ISBN978-0-471-32270-2
PrintedinSingapore
10987654321
ToourparentsConcettaMariaandPaolo
oursisterNuccetta
andlittlePaolo

CONTENTS
Foreword ix
Preface xi
1 TotalLeastSquaresProblems 1
1.1 Introduction / 1
1.2 Some TLS Applications / 2
1.3 Preliminaries / 3
1.4 Ordinary Least Squares Problems / 4
1.5 Basic TLS Problem / 5
1.6 Multidimensional TLS Problem / 9
1.7 Nongeneric Unidimensional TLS Problem / 11
1.8 Mixed OLS–TLS Problem / 14
1.9 Algebraic Comparisons Between TLS and OLS / 14
1.10 Statistical Properties and Validity / 15
1.11 Basic Data Least Squares Problem / 18
1.12 Partial TLS Algorithm / 19
1.13 Iterative Computation Methods / 19
1.14 Rayleigh Quotient Minimization Nonneural and Neural
Methods / 22
2 MCAEXINNeuron 25
2.1 Rayleigh Quotient / 25
2.2 Minor Components Analysis / 28
2.3 MCA EXIN Linear Neuron / 32
2.4 Rayleigh Quotient Gradient Flows / 34
2.5 MCA EXIN ODE Stability Analysis / 36
2.6 Dynamics of the MCA Neurons / 50
2.7 Fluctuations (Dynamic Stability) and Learning Rate / 66
2.8 Numerical Considerations / 73
2.9 TLS Hyperplane Fitting / 77
2.10 Simulations for the MCA EXIN Neuron / 78
2.11 Conclusions / 86
vii
viii CONTENTS
3 VariantsoftheMCAEXINNeuron 89
3.1 High-Order MCA Neurons / 89
3.2 Robust MCA EXIN Nonlinear Neuron (NMCA EXIN) / 90
3.3 Extensions of the Neural MCA / 96
4 IntroductiontotheTLSEXINNeuron 117
4.1 From MCA EXIN to TLS EXIN / 117
4.2 Deterministic Proof and Batch Mode / 119
4.3 Acceleration Techniques / 120
4.4 Comparison with TLS GAO / 125
4.5 TLS Application: Adaptive IIR Filtering / 126
4.6 Numerical Considerations / 132
4.7 TLS Cost Landscape: Geometric Approach / 135
4.8 First Considerations on the TLS Stability Analysis / 139
5 GeneralizationofLinearRegressionProblems 141
5.1 Introduction / 141
5.2 Generalized Total Least Squares (GETLS EXIN)
Approach / 142
5.3 GeTLS Stability Analysis / 149
5.4 Neural Nongeneric Unidimensional TLS / 178
5.5 Scheduling / 184
5.6 Accelerated MCA EXIN Neuron (MCA EXIN+) / 188
5.7 Further Considerations / 194
5.8 Simulations for the GeTLS EXIN Neuron / 198
6 GeMCAEXINTheory 205
6.1 GeMCA Approach / 205
6.2 Analysis of Matrix K / 210
6.3 Analysis of the Derivative of the Eigensystem of GeTLS
EXIN / 213
6.4 Rank One Analysis Around the TLS Solution / 218
6.5 GeMCA spectra / 219
6.6 Qualitative Analysis of the Critical Points of the GeMCA EXIN
Error Function / 224
6.7 Conclusions / 225
References 227
Index 239
FOREWORD
The history of data analysis can be divided into two main periods: before the
1970–1980 decades and after. Before the 1970–1980 decades it was mainly
the domain of applied mathematics and statistics. During this period, researchers
developedlineartechniques(nowdefinedas“classical”)undervariouspredefined
algorithms with some successful approaches to relatively standard problems and
data structures.
During the 1970–1980 decades, a new technique, often addressed to non-
mathematicians,appeared;itisknownunderthegenericnameofneuralnetworks.
The idea arose from the understanding that our (biological) neural networks are
able to perform in a very short time very complex tasks on difficult problems,
without a logically derived algorithm and without an “if–then–else” reasoning
scheme.Thesenetworksareknowntobemainlynonlinearandalsotobecapable
of learning from the data.
Although they were oversimplifications of biological models, the first mathe-
maticalmodels of neuralnetworkshaveprovedcapableof dealingwith complex
problems. The community of researchers became more and more interested in
these models, progressively establishing a completely new approach to data pro-
cessing and analysis. First considered cautiously by mathematicians, this new
technique gave rise to increasing interest from the mathematical and statistical
communities.
In the past decades, many international conferences have been devoted to
neuralnetworksperse,progressivelyestablishing thefoundationsof anewtech-
nique. Mathematicians who formerly considered this domain rather questionable
have now adopted it, and we can say that today it qualifies as one of many
classical statistical tools.
Inthisbooktheauthorsaddresstheproblemoflinearmultidimensionalregres-
sion from a completely new point of view. First, they proceed to an in-depth
theoreticalanalysisoftheproblemofregressionandshowthatmanyresearchers,
using ordinary least squares, have not properly considered the presence of noise
in their data, thus obtaining erroneous results. Based on the fact that noise may
be present in columns as well as in lines of a data matrix, the authors address
the problems of total least squares and minor components analysis.
ix