Learning Theory: An Approximation Theory Viewpoint

237 Pages·2007·1.98 MB·English

by  Felipe Cucker

This page intentionally left blank
CAMBRIDGEMONOGRAPHSON
APPLIEDANDCOMPUTATIONAL
MATHEMATICS
SeriesEditors
M.J.ABLOWITZ,S.H.DAVIS,E.J.HINCH,A.ISERLES,
J.OCKENDEN,P.J.OLVER
LearningTheory:
AnApproximationTheoryViewpoint
The Cambridge Monographs onApplied and Computational Mathematics reflect the
crucial role of mathematical and computational techniques in contemporary science.
Theseriespublishesexpositionsonallaspectsofapplicableandnumericalmathematics,
withanemphasisonnewdevelopmentsinthisfast-movingareaofresearch.
State-of-the-artmethodsandalgorithmsaswellasmodernmathematicaldescriptions
ofphysicalandmechanicalideasarepresentedinamannersuitedtograduateresearch
studentsandprofessionalsalike.Soundpedagogicalpresentationisaprerequisite.Itis
intendedthatbooksintheserieswillservetoinformanewgenerationofresearchers.
Within the series will be published titles in the Library of Computational
Mathematics, published under the auspices of the Foundations of Computational
Mathematics organisation. Learning Theory: An Approximation Theory View Point
isthefirsttitlewithinthisnewsubseries.
TheLibraryofComputationalMathematicsiseditedbythefollowingeditorialboard:
Felipe Cucker (Managing Editor) Ron Devore, Nick Higham, Arieh Iserles, David
Mumford,AllanPinkus,JimRenegar,MikeShub.
Alsointhisseries:
ApracticalGuidetoPseudospectralMethods,BengtFornberg
DynamicalSystemsandNumericalAnalysis,A.M.StuartandA.R.Humphries
LevelSetMethods,J.A.Sethian
TheNumericalSolutionofIntegralEquationsoftheSecondKind,
KendallE.Atkinson
OrthogonalRationalFunctions,AdhemarBultheel,PabloGonzález-Vera,
ErikHendriksen,andOlavNjåstad
TheoryofComposites,GraemeW.Milton
GeometryandTopologyforMeshGeneration,HerbertEdelsbrunner
Schwarz-ChristoffelMapping,TobinA.DriscollandLloydN.Trefethen
High-OrderMethodsforIncompressibleFluid,M.O.Deville,
E.H.MundandP.Fisher
PracticalExtrapolationMethods,AvramSidi
GeneralizedRiemannProblemsinComputationalFluidDynamics,
M.Ben-ArtziandJ.Falcovtz
RadialBasisFunctions,MartinBuhmann
Learning Theory: An Approximation
Theory Viewpoint
FELIPE CUCKER
CityUniversityofHongKong
DING-XUAN ZHOU
CityUniversityofHongKong
CAMBRIDGEUNIVERSITYPRESS
Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo
Cambridge University Press
TheEdinburghBuilding,CambridgeCB28RU,UK
Published in the United States of America by Cambridge University Press, New York
www.cambridge.org
Information on this title: www.cambridge.org/9780521865593
© Cambridge University Press 2007
Thispublicationisincopyright.Subjecttostatutoryexceptionandtotheprovisionof
relevantcollectivelicensingagreements,noreproductionofanypartmaytakeplace
withoutthewrittenpermissionofCambridgeUniversityPress.
Firstpublishedinprintformat 2007
ISBN-13 978-0-511-27551-7 eBook (NetLibrary)
ISBN-10 0-511-27551-X eBook (NetLibrary)
ISBN-13 978-0-521-86559-3 hardback
ISBN-10 0-521-86559-X hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyofurls
forexternalorthird-partyinternetwebsitesreferredtointhispublication,anddoesnot
guaranteethatanycontentonsuchwebsitesis,orwillremain,accurateorappropriate.
Contents
Foreword ix
Preface xi
1 Theframeworkoflearning 1
1.1 Introduction 1
1.2 Aformalsetting 5
1.3 Hypothesisspacesandtargetfunctions 9
1.4 Sample,approximation,andgeneralizationerrors 11
1.5 Thebias–varianceproblem 13
1.6 Theremainderofthisbook 14
1.7 Referencesandadditionalremarks 15
2 Basichypothesisspaces 17
2.1 Firstexamplesofhypothesisspace 17
2.2 RemindersI 18
2.3 HypothesisspacesassociatedwithSobolevspaces 21
2.4 ReproducingKernelHilbertSpaces 22
2.5 SomeMercerkernels 24
2.6 HypothesisspacesassociatedwithanRKHS 31
2.7 RemindersII 33
2.8 Onthecomputationofempiricaltargetfunctions 34
2.9 Referencesandadditionalremarks 35
3 Estimatingthesampleerror 37
3.1 Exponentialinequalitiesinprobability 37
3.2 Uniformestimatesonthedefect 43
3.3 Estimatingthesampleerror 44
3.4 Convexhypothesisspaces 46
3.5 Referencesandadditionalremarks 49
v
vi Contents
4 Polynomialdecayoftheapproximationerror 54
4.1 RemindersIII 55
4.2 Operatorsdefinedbyakernel 56
4.3 Mercer’stheorem 59
4.4 RKHSsrevisited 61
4.5 CharacterizingtheapproximationerrorinRKHSs 63
4.6 Anexample 68
4.7 Referencesandadditionalremarks 69
5 Estimatingcoveringnumbers 72
5.1 RemindersIV 73
5.2 CoveringnumbersforSobolevsmoothkernels 76
5.3 Coveringnumbersforanalytickernels 83
5.4 Lowerboundsforcoveringnumbers 101
5.5 Onthesmoothnessofboxsplinekernels 106
5.6 Referencesandadditionalremarks 108
6 Logarithmicdecayoftheapproximationerror 109
6.1 PolynomialdecayoftheapproximationerrorforC∞
kernels 110
6.2 Measuringtheregularityofthekernel 112
6.3 EstimatingtheapproximationerrorinRKHSs 117
6.4 ProofofTheorem6.1 125
6.5 Referencesandadditionalremarks 125
7 Onthebias–varianceproblem 127
7.1 Ausefullemma 128
7.2 ProofofTheorem7.1 129
7.3 Aconcreteexampleofbias–variance 132
7.4 Referencesandadditionalremarks 133
8 Leastsquaresregularization 134
8.1 Boundsfortheregularizederror 135
8.2 Ontheexistenceoftargetfunctions 139
8.3 Afirstestimatefortheexcessgeneralizationerror 140
8.4 ProofofTheorem8.1 148
8.5 RemindersV 151
8.6 Compactnessandregularization 151
8.7 Referencesandadditionalremarks 155
9 Supportvectormachinesforclassification 157
9.1 Binaryclassifiers 159
Contents vii
9.2 Regularizedclassifiers 161
9.3 Optimalhyperplanes:theseparablecase 166
9.4 Supportvectormachines 169
9.5 Optimalhyperplanes:thenonseparablecase 171
9.6 Erroranalysisforseparablemeasures 173
9.7 Weaklyseparablemeasures 182
9.8 Referencesandadditionalremarks 185
10 Generalregularizedclassifiers 187
10.1 Boundingthemisclassificationerrorintermsofthe
generalizationerror 189
10.2 Projectionanderrordecomposition 194
10.3 BoundsfortheregularizederrorD(γ,φ)offγ 196
10.4 Boundsforthesampleerrorterminvolvingfγ 198
φ
10.5 Boundsforthesampleerrorterminvolvingfz,γ 201
10.6 Strongererrorbounds 204
10.7 Improvinglearningratesbyimposingnoiseconditions 210
10.8 Referencesandadditionalremarks 211
References 214
Index 222
Foreword
ThisbookbyFelipeCuckerandDing-XuanZhouprovidessolidmathematical
foundationsandnewinsightsintothesubjectcalledlearningtheory.
Some years ago, Felipe and I were trying to find something about brain
scienceandartificialintelligencestartingfromliteratureonneuralnets.Itwas
in this setting that we encountered the beautiful ideas and fast algorithms of
learning theory. Eventually we were motivated to write on the mathematical
foundationsofthisnewareaofscience.
I have found this arena to with its new challenges and growing number of
application,beexciting.Forexample,theunificationofdynamicalsystemsand
learningtheoryisamajorproblem.Anotherproblemistodevelopacomparative
study of the useful algorithms currently available and to give unity to these
algorithms. How can one talk about the “best algorithm” or find the most
appropriate algorithm for a particular task when there are so many desirable
features, with their associated trade-offs? How can one see the working of
aspectsofthehumanbrainandmachinevisioninthesameframework?
Iknowbothauthorswell.IvisitedFelipeinBarcelonamorethan13years
ago for several months, and when I took a position in Hong Kong in 1995, I
askedhimtojoinme.ThereLenoreBlum,MikeShub,Felipe,andIfinished
a book on real computation and complexity. I returned to the USAin 2001,
butFelipecontinueshisjobattheCityUniversityofHongKong.Despitethe
distancewehavecontinuedtowritepaperstogether.IcametoknowDing-Xuan
as a colleague in the math department at City University. We have written a
number of papers together on various aspects of learning theory. It gives me
greatpleasuretocontinuetoworkwithbothmathematicians.Iamproudofour
jointaccomplishments.
Ileavetotheauthorsthetaskofdescribingthecontentsoftheirbook.Iwill
givesomepersonalperspectiveonandmotivationforwhattheyaredoing.
ix