Table Of ContentSPRINGER BRIEFS IN COMPUTER SCIENCE
Daniil Ryabko
Asymptotic
Nonparametric
Statistical Analysis
of Stationary Time
Series
123
SpringerBriefs in Computer Science
Serieseditors
StanZdonik,BrownUniversity,Providence,RI,USA
ShashiShekhar,UniversityofMinnesota,Minneapolis,MN,USA
XindongWu,UniversityofVermont,Burlington,VT,USA
LakhmiC.Jain,UniversityofSouthAustralia,Adelaide,SA,Australia
DavidPadua,UniversityofIllinoisUrbana-Champaign,Urbana,IL,USA
XueminShermanShen,UniversityofWaterloo,Waterloo,ON,Canada
BorkoFurht,FloridaAtlanticUniversity,BocaRaton,FL,USA
V. S. Subrahmanian, Department of Computer Science, University of Maryland,
CollegePark,MD,USA
MartialHebert,CarnegieMellonUniversity,Pittsburgh,PA,USA
KatsushiIkeuchi,Meguro-ku,UniversityofTokyo,Tokyo,Japan
BrunoSiciliano,DipartimentodiIngegneriaElettricaedelleTecnologie
dell’Informazione,UniversitàdiNapoliFedericoII,Napoli,Italy
SushilJajodia,GeorgeMasonUniversity,Fairfax,VA,USA
Newton Lee, Institute for Education, Research and Scholarships, Los Angeles,
CA,USA
SpringerBriefs present concise summaries of cutting-edge research and practical
applicationsacrossawidespectrumoffields.Featuringcompactvolumesof50to
125pages,theseriescoversarangeofcontentfromprofessionaltoacademic.
Typicaltopicsmightinclude:
(cid:129) Atimelyreportofstate-of-theartanalyticaltechniques
(cid:129) A bridge between new research results, as published in journal articles, and a
contextualliteraturereview
(cid:129) Asnapshotofahotoremergingtopic
(cid:129) Anin-depthcasestudyorclinicalexample
(cid:129) Apresentationofcoreconceptsthatstudentsmustunderstandinordertomake
independentcontributions
Briefs allow authors to present their ideas and readers to absorb them with
minimal time investment. Briefs will be published as part of Springer’s eBook
collection, with millions of users worldwide. In addition, Briefs will be available
forindividualprintandelectronicpurchase.Briefsarecharacterizedbyfast,global
electronic dissemination, standard publishing contracts, easy-to-use manuscript
preparation and formatting guidelines, and expedited production schedules. We
aim for publication 8–12 weeks after acceptance. Both solicited and unsolicited
manuscriptsareconsideredforpublicationinthisseries.
Moreinformationaboutthisseriesathttp://www.springer.com/series/10028
Daniil Ryabko
Asymptotic Nonparametric
Statistical Analysis
of Stationary Time Series
123
DaniilRyabko
FishlifeResearch
BelizeCity,Belize
ISSN2191-5768 ISSN2191-5776 (electronic)
SpringerBriefsinComputerScience
ISBN978-3-030-12563-9 ISBN978-3-030-12564-6 (eBook)
https://doi.org/10.1007/978-3-030-12564-6
LibraryofCongressControlNumber:2019933098
©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG.
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
This book is about making statistical inference from stationary discrete-time
processes. The assumption of stationarity alone is often considered too weak to
make any meaningful inference. Here, this view is challenged by showing that,
while some rather basic problems indeed can be proven not to admit any solution
in this setting, surprisingly many are solvable without any further assumptions.
These include such complex problems as clustering and change-point analysis.
Some general results characterizing those problems that admit a solution are also
presented.
The material in this volume is presented in a way that presumes familiarity
with basic concepts of probability and statistics, up to and including probability
distributionsoverspacesofinfinitesequences.Alltherequiredbackgroundmaterial
canbefoundintheexcellentmonograph[16],whichalsocontainsamuchdeeper
expositionofsomeofthekeyconceptsusedhere,suchasthedistributionaldistance.
Familiarity with ergodic theory is not required for understanding the material
exposed in the present volume. Indeed, with two exceptions, the proofs do not
rely on any facts deeper than the convergence of frequencies. One exception is
Chap.5, which deals with hypothesis testing and provides a characterization of
hypothesesforwhichconsistenttestsexist;therequiredbackgroundmaterialforthis
chaptercanbefoundinChap.2.TheotherexceptionisSect.3.4,whichestablishes
impossibility of discrimination between process distributions; this section is self-
contained.Thereaderwhoisfamiliarwithergodictheoryandfeelstheexposition
inthisvolumeissomewhatunorthodoxcanfindallthenecessarylinkstothemore
familiarframeworkin[58];thelatterbookisalsorecommendedtoanyoneseeking
adeeperunderstandingofsuchresultsastheslowconvergenceoffrequenciesand
entropyestimates,theclassicergodictheorem,andmuchmore.
Thisbook isorganized asfollows.Chapter1isintroductory:besidesproviding
some motivation for studying the problems addressed, it also introduces in an
informalmannerthemainconceptsusedandthemainresultspresented.Chapter2
introduces the notation and definitions used in the subsequent chapters, as well as
somenecessarybackgroundmaterial.Chapter3considersthemostbasicproblems
ofstatisticalinference,onwhichtherestofthevolumebuilds:estimatingadistance
v
vi Preface
between processes (the distributional distance) and the problem of homogeneity
testing or process discrimination, which, crucially for the subsequent problems
addressed, is shown to be impossible to solve in the general setting of this book.
Chapter4isdevotedtoclusteringandchange-pointproblems,whichcanbesolved,
or, in some cases, can be shown to admit no solution, based on the result of the
preceding chapter. Chapter 5 addresses the problems of hypothesis testing in the
general form: studying which pairs of hypotheses admit a consistent test. Finally,
Chap.6 discusses various generalizations of the presented results,as well as some
directionsforfutureresearch.
Acknowledgements ThankstoLéonBottouforgivingmetheideatowriteabookonthissubject
andforencouragingmetodoit.ThankstoBorisRyabkoandAzadehKhaleghi,incollaboration
withwhomsomeoftheresultspresentedherewereobtained.
SantaCruzdelaSierra,Bolivia DaniilRyabko
Contents
1 Introduction ................................................................... 1
1.1 Stationarity,Ergodicity,AMS........................................... 2
1.2 What Is Possible and What Is Not Possible to Infer
fromStationaryProcesses ............................................... 4
1.3 OverviewoftheInferenceProblemsCovered.......................... 7
2 Preliminaries .................................................................. 9
2.1 Stationarity,Ergodicity .................................................. 10
2.2 DistributionalDistance .................................................. 11
3 BasicInference................................................................ 13
3.1 Estimating the Distance Between Processes
andReconstructingaProcess............................................ 13
3.2 Calculatingdˆ............................................................. 15
3.3 TheThree-SampleProblem ............................................. 17
3.4 ImpossibilityofDiscrimination ......................................... 18
3.4.1 SetupandDefinitions............................................ 19
3.4.2 TheMainResult ................................................. 19
4 ClusteringandChange-PointProblems.................................... 29
4.1 Time-SeriesClustering .................................................. 30
4.1.1 ProblemFormulation............................................ 31
4.1.2 AClusteringAlgorithmandItsConsistency................... 32
4.1.3 Extensions: Unknown k, Online Clustering and
ClusteringwithRespecttoIndependence ...................... 34
4.2 Change-PointProblems.................................................. 37
4.2.1 SingleChangePoint............................................. 39
4.2.2 MultipleChangePoints,KnownNumberofChangePoints .. 41
4.2.3 UnknownNumberofChangePoints ........................... 42
5 HypothesisTesting............................................................ 45
5.1 Introduction .............................................................. 45
5.1.1 MotivationandExamples ....................................... 46
vii
viii Contents
5.2 TypesofConsistency .................................................... 47
5.2.1 UniformConsistency............................................ 47
5.2.2 AsymmetricConsistency........................................ 48
5.2.3 AsymptoticConsistency......................................... 49
5.2.4 OtherNotionsofConsistency................................... 49
5.3 OneExampleThatExplainsHypothesesTesting ...................... 49
5.3.1 Bernoullii.i.d.Processes ........................................ 50
5.3.2 MarkovChains................................................... 51
5.3.3 StationaryErgodicProcesses.................................... 52
5.4 TopologicalCharacterizations........................................... 53
5.4.1 UniformTesting ................................................. 53
5.4.2 AsymmetricTesting ............................................. 54
5.5 Proofs..................................................................... 55
5.6 Examples ................................................................. 61
5.6.1 SimpleHypotheses,IdentityorGoodness-of-FitTesting...... 61
5.6.2 MarkovandHiddenMarkovProcesses:
BoundingtheOrder.............................................. 62
5.6.3 SmoothParametricFamilies .................................... 63
5.6.4 HomogeneityTestingorProcessDiscrimination .............. 63
5.6.5 Independence .................................................... 64
5.7 OpenProblems........................................................... 65
5.7.1 RelatingtheNotionsofConsistency............................ 65
5.7.2 CharacterizingHypothesesforWhich
ConsistentTestsExist ........................................... 66
5.7.3 IndependenceTesting............................................ 67
6 Generalizations................................................................ 69
6.1 OtherDistances .......................................................... 69
6.1.1 sumDistances.................................................... 69
6.1.2 TelescopeDistance .............................................. 70
6.1.3 supDistances..................................................... 71
6.1.4 Non-metricDistances............................................ 72
6.1.5 AMSDistributions............................................... 72
6.2 Piece-WiseStationaryProcesses........................................ 73
6.3 BeyondTimeSeries...................................................... 73
6.3.1 ProcessesOverMultipleDimensions........................... 73
6.3.2 InfiniteRandomGraphs......................................... 74
References.......................................................................... 75
Chapter 1
Introduction
This book is about making statistical inference from discrete-time processes
under what is perhaps the weakest of statistical assumptions: stationarity. Before
embarking on this journey, it is worth asking the question of why it is interesting
to study statistical problems under this assumption alone, or under similar related
assumptions. To answer this question, one should first consider what it means to
haveagoodsetofassumptions,oragoodmodel,forastatisticalproblemathand.
Choosing the right assumptions presents the following trade-off. On the one
hand,makingstrongassumptionsmakestheinferencetaskeasierandallowsoneto
obtainstrongerperformanceguaranteesforthealgorithmsdeveloped.Forexample,
by assuming that the data are independent and identically distributed (i.i.d.), one
gets at one’s disposal an extremely versatile statistical toolkit that is a result of
centuriesofresearchonthismodel.Withthis,itispossibletoobtainsharpbounds
onerrorprobabilitiesoftheresultingmethods.Evenstrongerresultscanbeobtained
ifonefurthermakesparametricassumptions.Ontheotherhand,allsuchresultsare
useless if the assumptions made do not hold for the data at hand. Of course, one
can try to apply a statistical test to the data in order to verify the validity of one
or another model. This, however, only pushes back the problem, because to use a
testoneneedstomakeanothersetofassumptions,calledthealternative.Indeed,it
is not possible to test, based on data, that the assumption H holds versus it does
0
not hold. For example, it is not possible to test that the data are Gaussian i.i.d.
versus the distribution of the data is anything else except Gaussian i.i.d. This is
because the alternative “anything else” is too general and includes, for example,
such distributions as the one that is concentrated precisely on the data available.
It is, however, possible to design a test for the hypothesis “the data are Gaussian
i.i.d.”versus“thedataarei.i.d.butnotGaussian”or“thedataarei.i.d.”versus“the
distribution of the data is stationary.” In other words, it may be possible to test a
set of assumptions H versus an alternative set of assumptions H . The latter is
0 1
typically much more general; in fact, one is interested in making it as general as
possible.Nonetheless,thealternativehypothesisisstillasetofassumptions.
©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2019 1
D.Ryabko,AsymptoticNonparametricStatisticalAnalysisofStationary
TimeSeries,SpringerBriefsinComputerScience,
https://doi.org/10.1007/978-3-030-12564-6_1
