Bounds for the normal approximation of the maximum likelihood estimator

205 Pages·2016·14.35 MB·English

by  Andreas Anastasiou

Bounds for the normal approximation
of the maximum likelihood estimator
Andreas Anastasiou
Supervisor: Professor Gesine Reinert
Department of Statistics
University of Oxford
This dissertation is submitted for the degree of
Doctor of Philosophy
Jesus College Trinity Term, 2016
Tomyparents,KyriacosandMaria,andmysiblings,AnnaandTassos.
Acknowledgements
Thefirstandmostimportantpersontothankismysupervisor,ProfessorGesineReinert.
Iamdeeplygratefultoherfortheguidance,encouragementandsupportsheofferedme
throughoutmytimeasherstudent.
IwouldliketothankProfessorChristopheLeyforafruitfulcollaborationwhichleadto
apaper,ofwhichtheresultscanbefoundinChapter5ofthethesis.
I would also like to thank Dr Robert Gaunt for various insightful comments and
suggestions.
Last,butnotofleastimportance,Iwouldliketothankmyparentswhoencouragedme
tocontinuewithpostgraduatestudies.
Theresearch isconductedwiththe supportofa Teaching AssistantshipBursaryfrom
theDepartmentofStatistics,UniversityofOxford,andtheEngineeringandPhysical
SciencesResearchCouncil(EPSRC)grantEP/K503113/1.
Abstract
Theasymptoticnormalityofthemaximumlikelihoodestimator(MLE)underregularity
conditionsisalongestablishedandfamousresult. Thisisaqualitativeresultandthe
assessmentofsuchanormalapproximationisourmaininterest. Forthistaskwepartly
use Stein’s method, which is a probabilistic technique that can be used to explicitly
measurethedistributionaldistancebetweentwodistributions. Sinceitsfirstappearance
in1972,themethodhasbeendevelopedforvariousdistributions;hereweusetheresults
relatedtoStein’smethodfornormalapproximation.
Inthisthesis,wederiveexplicitupperboundsonthedistributionaldistancebetweenthe
distributionoftheMLEandthenormaldistribution. First,thefocusisonindependent
andidenticallydistributedrandomvariablesfrombothdiscreteandcontinuoussingle-
parameterdistributionswithparticularattentiontoexponentialfamilies. Fordiscrete
distributions,thecasewheretheMLEcanbeontheboundaryoftheparameterspace
is treated through a perturbation approach, which allows us to obtain bounds on the
distributionaldistanceofinterest. Theboundsareoforder √1 ,wherenisthenumber
n
ofobservations. Simulation-basedresultsaregiventoillustratethepowerofthebound.
Furthermore,oftentheMLEcannotbeobtainedanalyticallyandoptimisationmethods
(suchastheNewton-Raphsonalgorithm)areused. Eveninsuchcases,order √1 bounds
n
aregivenforthedistributionaldistancerelatedtotheMLE.
Thecaseofmulti-parameterdistributionsfollowssmoothlyafterthedetaileddiscussion
v
relatedtoascalarparameter. Apartfromextendingourapproachtoamulti-parameter
setting, we alsocover thecaseof independentbut notnecessarilyidenticallydistributed
(i.n.i.d.) randomvectorswithspecificfocusonthewidelyapplicablelinearregression
models.
Goingback tothesingle-parameter settingadifferent approachtoget anupperbound
on the distributional distance between the distribution of the MLE and the normal
distribution,based ontheDelta method,is alsodeveloped. The MLE fora Generalised
Gamma distribution gives an illustration of the results obtained through this Delta
methodapproach.
Finally, we relax the independence assumption and results for the case of locally
dependent random variables are obtained. An example of correlated sums of normally
distributedrandomvariablesillustratesthebounds. Again,resultsthatdonotrequirean
analyticexpressionoftheMLEtobeknownaregiven. Weendthisthesiswithideas
currentlyinprogressandfurtheropenresearchquestions.
Table of contents
Listoffigures ix
Listoftables x
1 Introduction 1
1.1 Motivationforthethesis . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Stein’smethodasausefultool . . . . . . . . . . . . . . . . . . . . . 3
1.3 Outlineofthethesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Background 8
2.1 CentralLimittheorem . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Maximumlikelihoodestimation . . . . . . . . . . . . . . . . . . . . 10
2.3 Stein’smethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3 BoundsonthedistancetonormalfortheMLE 24
3.1 TheboundedWassersteinandtheKolmogorovdistance . . . . . . . . 25
3.2 BoundsintermsoftheboundedWassersteindistance . . . . . . . . . 27
3.3 Single-parameterexponentialfamilies . . . . . . . . . . . . . . . . . 36
3.3.1 Theexponentialdistribution . . . . . . . . . . . . . . . . . . 40
3.3.2 Empiricalresults . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4 Discretedistributions: Theboundaryissue . . . . . . . . . . . . . . . 51
3.4.1 Theperturbationapproach . . . . . . . . . . . . . . . . . . . 52
Tableofcontents vii
3.4.2 Example: ThePoissondistribution . . . . . . . . . . . . . . . 60
3.5 Boundsonthemeansquarederror . . . . . . . . . . . . . . . . . . . 68
4 Multi-parameterdistributions 79
4.1 Non-identicallydistributedrandomvectors . . . . . . . . . . . . . . 81
4.1.1 Ageneralbound . . . . . . . . . . . . . . . . . . . . . . . . 82
4.1.2 Linearregression . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2 Identicallydistributedrandomvectors . . . . . . . . . . . . . . . . . 102
4.2.1 Thenormaldistribution . . . . . . . . . . . . . . . . . . . . . 104
4.2.2 BoundswhentheMLEisnotknownexplicitly . . . . . . . . 114
5 AnapproachusingtheDeltamethod 133
5.1 NewboundsonthedistancetonormalfortheMLE . . . . . . . . . . 134
5.2 Comparisoninexponentialfamilies . . . . . . . . . . . . . . . . . . 143
5.2.1 Boundsforsingle-parameterexponentialfamilies . . . . . . . 143
5.2.2 Boundsfortheexponentialdistribution . . . . . . . . . . . . 149
6 Locallydependentrandomvariables 151
6.1 Ageneralapproach . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.2 Locallydependentnormalrandomvariables . . . . . . . . . . . . . . 160
6.3 Analternativebound . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.4 ResultsfornotanalyticallyknownMLE . . . . . . . . . . . . . . . . 174
6.4.1 Boundedsupport . . . . . . . . . . . . . . . . . . . . . . . . 175
6.4.2 Boundedparameterspace . . . . . . . . . . . . . . . . . . . 179
7 Conclusion 183
7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
7.2 Openproblemsforfutureexploration . . . . . . . . . . . . . . . . . 186
7.2.1 Furtherapplications . . . . . . . . . . . . . . . . . . . . . . 186
7.2.2 BeyondtheclassicalframeworkoftheMLE . . . . . . . . . . 189
Tableofcontents viii
References 192
List of figures
3.1 The density function of (cid:112)ni(θ )(cid:0)θˆ (X)−θ (cid:1) for various sample
0 n 0
sizes. In this case X ,X ,···,X are i.i.d. random variables which
1 2 n
followtheExp(1)distribution. . . . . . . . . . . . . . . . . . . . . . 49
3.2 The density function of (cid:112)ni(θ )(cid:0)θˆ (X)−θ (cid:1) for various sample
0 n 0
sizes. In this case X ,X ,···,X are i.i.d. random variables which
1 2 n
followtheExp(0.5)distribution. . . . . . . . . . . . . . . . . . . . . 50
3.3 TheQ-Qplot,whenn=1000,andthebarplotoftheerrorfordifferent
samplesizesfortheexponentialdistributionexample,Exp(1). . . . . 50
3.4 TheQ-Qplot,whenn=1000,andthebarplotoftheerrorfordifferent
samplesizesfortheexponentialdistributionexample,Exp(0.5). . . . 51
√
3.5 Thedensityfunctionof n(θˆ (X)−θ )forvarioussamplesizes. In
n 0
this case X ,X ,···,X are i.i.d. random variables which follow the
1 2 n
Poisson(2)distribution. . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.6 TheQ-Qplot,whenn=1000,andthebarplotoftheerrorfordifferent
samplesizesforthePoissondistributionwithmeanequalto2. . . . . 68
6.1 Structureofa2m-dependentsequence. . . . . . . . . . . . . . . . . . 153
List of tables
3.1 SimulationresultsfromtheExp(1)distribution . . . . . . . . . . . . 47
3.2 Simulation results from the Exp(0.5) distribution treated as a non-
canonicalexponentialfamily . . . . . . . . . . . . . . . . . . . . . . 48
3.3 SimulationresultsfromthePoisson(2)distribution . . . . . . . . . . 66
3.4 SimulationresultsfromtheBeta(1.5,1)distribution . . . . . . . . . . 78