Stochastic Multiplayer Games: Theory and Algorithms

175 Pages·2010·1.982 MB·English

by  Ummels, Michael

S
t
o
c
Stochastic games provide a versatile model for reactive systems that are h
a
affected by random events. This dissertation advances the algorithmic theory st
i
c
ofstochastic games to incorporate multiple players, whose objectives are not
M
necessarily conflicting. The basis ofthis workis a comprehensive complexity- u
lt
theoretic analysis of the standard game-theoretic solution concepts in the i
p
l
context ofstochastic games over a finite state space. One main result is that a
y
e
the constrained existence ofa Nash equilibrium becomes undecidable in this r
G
setting. This impossibility result is accompanied by several positive results, a
m
includingefficientalgorithmsfornaturalspecialcases. e
s
:
T
h Stochastic Multiplayer Games
MichaelUmmelsreceivedhisdiplomadegreeincomputersciencefromRWTHAachen e
o
University. He started his doctoral studies at the same university in 2006, supervised r
y
byProf. Dr. Erich Grädel and Prof. Dr. Dr.h.c. Wolfgang Thomas. As ofFebruary2010, a Theory and Algorithms
n
d
theauthorisapostdoctoralresearcheratENSCachan.
A
l
g
o
r
it Michael Ummels
h
m
s
M
i
c
h
a
e
l
U
m
m
e
l
s
✐ ✐
✐ ✐
StochasticMultiplayerGames
TheoryandAlgorithms
✐ ✐
✐ ✐
✐ ✐
✐ ✐
TypesetbytheauthorinFedraSerifBusingLATEX.
CoverdesignbySamRoss-Gower.
ISBN 9789085550402
NUR 918
D82(Diss.RWTHAachenUniversity,2010)
©MichaelUmmels,2010.
PublishedbyPallasPublications,AmsterdamUniversityPress,Amsterdam.
cbnd
ThisworkislicensedundertheCreativeCommonsAttribution-NonCommercial-
NoDerivs 3.0 Unported License. To view a copy of this license, visit http://
creativecommons.org/licenses/by-nc-nd/3.0/,orsendalettertoCreativeCom-
mons,1712ndStreet,Suite300,SanFrancisco,California,94105,USA.
✐ ✐
✐ ✐
✐ ✐
✐ ✐
Stochastic Multiplayer Games:
Theory and Algorithms
Von der Fakultät für Mathematik, Informatik und
NaturwissenschaftenderRWTHAachenUniversity
zurErlangungdesakademischenGradeseinesDoktors
der Naturwissenschaften genehmigte Dissertation
vorgelegtvon
Diplom-Informatiker
MichaelUmmels
ausKöln
Berichter: UniversitätsprofessorDr.ErichGrädel
UniversitätsprofessorDr.WolfgangThomas
AssistantProfessorDr.MarcinJurdziński
TagdermündlichenPrüfung:27.Januar2010
DieseDissertationistaufdenInternetseitenderHochschulbibliothekonlineverfügbar.
✐ ✐
✐ ✐
✐ ✐
✐ ✐
✐ ✐
✐ ✐
✐ ✐
✐ ✐
Preface
Thelastdecadeshaveseenanimmenseamountofresearchonthealgo-
rithmiccontentofgametheory. Ontheonehand,anewsubjectcalled
algorithmic game theoryhasemergedthatisconcernedwiththestudyofthe
algorithmictheoryoffinitegameswithmultipleplayers.Ontheotherhand,
infiniteand,inparticular,stochastictwo-playerzero-sumgameshavebecome
animportanttoolfortheverificationofopensystems,whichinteractwith
theirenvironment.
Theaimofthisworkistobringtogetheralgorithmicgametheorywith
thegamesthatareusedinverificationbyextendingthealgorithmicthe-
oryofstochastictwo-playerzero-sumgamestoincorporatemultipleplay-
ers, whose objectives are not necessarily conflicting. In particular, this
workcontainsacomprehensivestudyofthecomplexityofthemostpromi-
nentsolutionconceptsthatareapplicableinthissetting,namelyNashand
subgame-perfectequilibria.
ThisbookistheresultofmydoctoralstudiesatRWTHAachenUniver-
sity.IamindebtedtomyprimarysupervisorErichGrädelforgivingmethe
opportunitytopursuethesestudies,forintroducingmetothescientific
communityandforgivingmeadvicejustwhenIneededit. Iamequally
grateful to my secondary supervisor Wolfgang Thomas for his constant
supportandencouragement.
MarcinJurdzińskididnothesitatetoactasanexternalreviewerforthis
thesis.Ithankhimnotonlyforhiscarefulreadingandnumerousremarks,
butalsoforgivinganinspiringtalkonbranchingvectoradditionsystems,
whichindirectlyledtotheresolutionofaproblemthatwasleftopeninthe
originalversionofthisthesis.
AsubstantialpartofthisbookisbasedonjointworkwithDominikWojt-
czak. Iamindebtedtohimforournumerousilluminatingdiscussions,
forhisinsightsandideas,and—lastbutnotleast—forhostingmeinEdin-
burgh,AmsterdamandOxford.
5
✐ ✐
✐ ✐
✐ ✐
✐ ✐
Amongthevariousotherpeoplewhocontributedtothiswork,Iwould
liketothankinparticularŁukaszKaiserformanyenlighteningdiscussions
andfordiscoveringProposition3.18.SpecialthanksalsogotoFlorianHorn
formanyinterestingdiscussions,toJánosFleschforpointingoutProposi-
tion3.13,andtoPeterBroMiltersenfordrawingmyattentiontoCorollary4.4.
Moreover,IamgratefultoHugoGimbertandEilonSolanforansweringmy
questionsandtoRohitChadha,TobiasGanzow,JörgOlschewskiandEdeline
Wongfortheircommentsonpreliminarydraftsofthiswork.
Finally,IwouldliketothankSamRoss-Gowerfordesigningthecoverof
thisbook,andDonaldKnuthandLeslieLamportforcreating(LA)TEX.
Paris,November2010
6
✐ ✐
✐ ✐
✐ ✐
✐ ✐
Contents
1 Introduction • 15
1.1 Gamesandequilibria • 15
1.2 Thestochasticdiningphilosophersproblem • 21
1.3 Contributions • 25
1.4 Relatedwork • 27
1.5 Outline • 28
2 StochasticGames • 31
2.1 Arenasandobjectives • 31
2.2 Strategiesandstrategyprofiles • 37
2.3 Subarenasandendcomponents • 41
2.4 Values,determinacyandoptimalstrategies • 42
2.5 Algorithmicproblems • 47
2.6 Existenceofresiduallyoptimalstrategies • 51
3 Equilibria • 55
3.1 Definitionsandbasicproperties • 55
3.2 ExistenceofNashequilibria • 59
3.3 Existenceofsubgame-perfectequilibria • 64
3.4 Computingequilibria • 69
3.5 Decisionproblems • 73
4 ComplexityofEquilibria • 77
4.1 Positionalequilibria • 77
4.2 Stationaryequilibria • 82
4.3 Pureandrandomisedequilibria • 88
4.4 Finite-stateequilibria • 96
4.5 Summaryofresults • 98
7
✐ ✐
✐ ✐
✐ ✐
✐ ✐
Contents
5 DecidableFragments • 99
5.1 Thestrictlyqualitativefragment • 99
5.2 Thepositive-onefragment • 113
5.3 Thequalitativefragmentfordeterministicgames • 122
5.4 Summaryofresults • 133
6 Conclusion • 135
6.1 Summaryandopenproblems • 135
6.2 Perspectives • 138
A Preliminaries • 141
A.1 Probabilitytheory • 141
A.2 Computationalcomplexity • 144
B MarkovChainsandMarkovDecisionProcesses • 149
B.1 Markovchains • 149
B.2 Markovdecisionprocesses • 152
Bibliography • 157
Notation • 169
Index • 171
8
✐ ✐
✐ ✐
✐ ✐
✐ ✐
List of Figures
1.1 Matchingpenniesasagameinextensiveform • 18
1.2 Diningphilosophers • 22
1.3 Processesforthestochasticdiningphilosophersproblem • 23
1.4 Thestochasticdiningphilosophersgamewithtwo
philosophers • 24
2.1 Ahierarchyofprefix-independentobjectives • 36
2.2 Anexampleofatwo-playerSSMG • 38
2.3 AnMDPwithnooptimalstrategy • 44
2.4 AnotherMDPwithnooptimalstrategy • 44
3.1 Atwo-playerreachabilitygamewithanirrationalNash
equilibrium • 58
3.2 Atwo-playergamewithapairofoptimalstrategiesthatcannot
beextendedtoaNashequilibrium • 62
3.3 AnSSMGwithnostationaryNashequilibrium • 63
3.4 Atwo-playerSSMGwithnopositionalNashequilibrium • 64
3.5 ABüchiSMGwithnosubgame-perfectequilibrium • 68
3.6 AnSSMGthathasastationarySPEwhereplayer0winsalmost
surelybutnopureNEwhereplayer0winswithpositive
probability • 75
3.7 ThedifferentdecisionproblemsrelatedtoNashand
subgame-perfectequilibria • 76
4.1 ReducingSATtoPosNE,StatNE,PosSPEandStatSPE • 79
4.2 ReducingSqrtSumtoStatNEandStatSPE • 85
4.3 Simulatingatwo-countermachine • 90
4.4 Reducingfromthehaltingproblem • 97
9
✐ ✐
✐ ✐