Table Of ContentAn Introductitoon
ComputationLaela rninTgh eory
CopyrigMhatteedr ial
An Introductitoon
Computational LeaTrhneionrgy
MichaeKle aJr.n s
UmeshV .V azirani
TheM IT Press
CambridgMea,s sachusetts
LondonE,n gland
CopyrigMhatteedr ial
@1994M assachuseItntsst itoufTt eec hnology
Allr ighrtess ervNeod p.a rotf t hibso okm ayb er eproduced ina ny
forbmy a nye lectroornm iecc hanimceaaln s (incplhuodtioncgo pying,
recordoirni gn,f ormation asntdro ertargiee wviatlhu)ot p ermission in
writifnrgo mt hep ublisher.
Thibso okw asty pesetb yt hea uthoarnsd w as printaenddb ounidn
theU niteSdt atoefsA merica.
LibraorfyC ongreCsast aloging-in-PDuabtlai cation
KearnsM, ichaJe.l
An introducttoic oonm putatiolneaalrn itnhge or/y M ichaelJ .
KearnUsm,e shV .V azira.ni.
p. cm.
Includbeisb Hographriecfaelr enacnedis n dex.
ISDN0 -262-11193-4
1.M achinlee arnin2g.A. r tificiinatle llig3e.nAc leg.o rithms.
4.N eura.nle tworkIs..V aziraUlm1eis, hV irkuma.rI.I T.i tle.
Q325.5.K14949 4
006.3-dc20 94-16588
CIP
109876
Copyrighted Material
Contents
Preface xi
1 The ProbablAyp proximateCloyr recLte arninMgo del 1
1.1A Rectangle Learning Game 1
1.2A General Model 6
121 Definitoifot nh eP ACM odel 7
. .
1.2.R2e presentSaitziaeon ndI nstaDnicmee nsion 12
1.3L earniBnogo leCaonn junctions 16
1.4I ntractaobfiL leiatryn i3n-Tge rmD NF Formulae 18
1.5U singC N3F Formultaeo A v oiIdn tractability 22
-
1.6E xercises 26
17. BibliogrNaoptheisc 28
2 Occam' 8R azor 31
2.1O ccamL earnianngdS uccinctness 33
CopyrigMhatteedr ial
Contents
vi
2.2I mprovtihnegS amplSei zfeo Lre arngi Cnonjunctions3 7
2.3L earngi CnonjunctiwointsFh e wR elevaVnatr iables 38
2.4L earngi DnecisLiiosnt s 42
2.5E xercises 44
2.6B ibliogrNaoptheisc 46
3 TheV apnik-ChervonDeinmkeinss ion 49
3.1W henC anI nfiniCtlases esB eL earnweidt ah F iniStaem ple4?9
3.2T heV apnik-Chervonenkis Dimension 50
3.3E xamploefts h eV C Dimension 51
3.4A PolynomBioalu nod nl 11c(8)1 54
3.5A PolynomBioauln odn t heS ampleS izfeo PrA CL earni5n7g
3.5.T1h eI mportaonfcf e- Nets 57
3.5.A2 Smallf -Neftr oRamn domS ampling 59
3.6S amplSei zLeo weBro unds 62
3.7A n ApplicattoiN oenu rNale tworks 64
3.8E xercises 67
3.9B ibliogrNaopthiesc 70
4 Weaka ndS trong Learning 13
4.1A RelaxDeedfi nitoifoL ne arning? 73
4.2
Boostgit nheC onfidence 76
CopyrigMhatteedr ial
Contents vii
4.3Bo ostign theA ccuarcy 78
4.3.1 79
A ModesAtcc uracByo ostiPnrgo cedure
4.3.2 81
ErroArn alysfiotsrh eM odesPtr ocedure
4.3.A 3Re cusriveA ccuraBcoyo tsingA lgorithm 85
4.3.4 88
Boundintgh eD eptohf t heR ecurns io
4.3.5
AnalyosfiF si elrtinEgffi ciency 89
4.3.6
FinishUipn g 96
101
4.4E xercises
4.5B iliobgraphic Notes 102
5 Learningt hiePn r esencoefN oise 103
104
5.1T heC lassificNaotiisoMeno del
5.2A n AlgoritfhoLmre arniCnogn junctfiroonSmst a tistic1s 06
5.3 108
TheS tatisQtuiecryaL le arniMnogd el
5.4
SimulatSitnagt isQtuiecrailie nts h eP resenocfNe o ise 111
5.4.1
A NicDee cmoposiitono fP x. 112
5.4.2
Solving foarn E stimaotfPe x. 114
5.4.3 115
GuessianngdV erifytihneNg o isRea te
5.4.D4e scrpitoin of timhuela tioSnA logrithm 117
5.5 119
Exerciess
5.6
BibliogrNaoptheisc 121
CopyrigMhatteedr ial
Contents
viii
6 InherUennptr edictability 123
6.1 RepresentDaetpieonnd eanntdI ndependent Har1d2n3e ss
6.2 TheD iscrCeutbee R ootP roblem 124
6.2.1T heD ifficulotfyD isrcetCeu beR oots 126
6.2.2D iscrCeutbee R ootass a Learning Problem 128
6.3 SmalBlo oleCainrc uitsA reI nherenUtnplryed ictable 131
6.4 ReducitnhgeD eptohf In herentlyU npredcitableC ircui1t3s3
6.4.1E xpadningt heI nput 135
6.5 A GenerMaelt hoadn dI tAsp plicattioNo enu arlN etwor1ks3 9
6.6 Exercises 140
6.7 Biblogiraphic Notes 141
7 ReducibiliinPt Ay C Learning 143
7.1R educing DNFt oM onotoDnNeF 144
7.2A GeneralM ethdo foRre ducibility 147
194
7.3R educiBnogo leFaonr multaoFe i niAtueto mata
7.4E xercises 153
7.5B ibliogrNaoptheisc 154
8 LearningF initAeu tomatbay Experimentation 155
8.1A ctiavned P assivLee raning 155
8.2 ExacLte arniUnsgi nQgu ersi e 158
CopyrigMhatteedr ial
Contents ix
8.3E xacLte arnionfFg in ite Automata 160
8.3.1Ac cessS tringsa ndD istniguishingS trings 160
8.3.2A n Efficientyl ComputbaleS tatPea rtition 162
8.3.3T heT entatiev Hypotheits is 164
8.3.4U snig Cao unterexample 166
8.3.5T heA lgoriftohLrme arning Finite Automata 169
8.3.6R unninTgim e Analyiss 171
8.4L earniwnigt hoau Rte set 174
8.4.1U sinag H oming SequteoLn ecaer n 176
8.4.2B uildnig Ha omingS equenUcsei nOgv ersiGzeend-
eralizCeldas sificaTrteieosn 178
8.4.3T heN o-ResAelgto rithm 181
8.4.4M akinSgu rLe(1 B uilGdesn eralCizleasds ification
Trees 182
8.5E xericses 185
8.6B ibliogrNaoptheisc 186
9 AppendiSxo:m eT oolfso Prr obabilAinsatliycs is 189
9.1T heU nioBno und 189
9.2M arkovI'nse quality 189
9.3C hernoffB ounds 190
CopyrigMhatteedr ial
x Contents
Bibliography 193
Index 205
CopyrigMhatteedr ial
Preface
Int heF allt ermo f1 990w,e j ointtalugyh ta graduatsee minianrc om
putatiolneala rntihnego riynt hec omputsecri endceep artmoefnt th e
UniversoiftCy a lifoartnB ierake leTyhem atertihaalit sp resenhteerde
.
has itosr igiinnts h at coursbeo,t hi nc ontneta ndex positRiaotnh.e r
thana ttemtpotg ivea ne xhaustoivveer vioeftw h irasp idleyx panding
andc hangianrgeo afr esearwcehh a,v ter iteodca reufllys elefcutn damen
tatlo pitchsa dte monstriamtpeo rtparnitn citphlatem sa yb ea plpicable
ina widesre ttitnhga nt heo neex am inde hereI.n t het echnisceacl
tionwse,h avet reidt oe mphasiiznet uitwihoenn evpeors siwbhliels,et ill
providpirnegc iasreg uments.
Theb ooiks i nteenddf orre searcahnedsr tsu deinnta sr tifiicnitaell li
gencen,e uraneltw orks,t heorectoimcpault secri enancdes tatisatnidc s,
anyoneel sien tereisntm eadt hematmiocdaell osfl e arinng Iti sa ppro
.
priatef oru seas thec entrtaelxi tn a specialsiemziendca ourr s,eo ras
a supplemteenxtitan la broader courtshea pte hrapsa lssot uditehse
viewpoitnatkseb ny a rtificinitaellgle ince andn eurnaelt worWkhsi.l e
Chapt1el ra yasc ommofno undatfioaorln tl h esu bsequmeantte rtiahle,
latchearp terasr ee ssentisaelllfy- conatnadim naeydb er easde lectively
andi nan y ordeErx.e rciasreeprs ov ided att hee ndo fe acchh apter.
Someb riceofm menotsnt hee xpectbeackdgr onud oft her eadaerre
approprhiearteFe.a miliawritihts yo mbeas itco oolfst hfeo rmaanla lysis
ofa lgoritish mecnses sar,yas isf amiliawriittohyn ly them oste lemen
tarnyo tioonfsc omplextihetoryy, s ucash NP-completeFnoerts hse.
CopyrigMhatteedr ial
