Table Of ContentModern Applied
Statistics with S
Fourth edition
by
W. N. Venables and B. D. Ripley
Springer (mid 2002)
Final 15 March 2002
Preface
S is a language and environmentfor data analysis originally developed at Bell
Laboratories (of AT&T and now Lucent Technologies). It became the statisti-
cian’scalculatorforthe1990s,allowingeasyaccesstothecomputingpowerand
graphicalcapabilities of modern workstations and personalcomputers. Various
implementations have been available, currently S-PLUS, a commercial system
fromtheInsightfulCorporation1inSeattle,andR,2anOpenSourcesystemwrit-
tenbyateamofvolunteers. BothcanberunonWindowsandarangeofUNIX/
Linuxoperatingsystems: RalsorunsonMacintoshes.
This is the fourthedition of a book which first appearedin 1994, and the S
environmenthasgrownrapidlysince.Thisbookconcentratesonusingthecurrent
systemstodostatistics;thereisacompanionvolume(VenablesandRipley,2000)
which discusses programming in the S language in much greater depth. Some
of themorespecializedfunctionalityofthe Senvironmentiscoveredin on-line
complements,additionalsectionsandchapterswhichareavailableontheWorld
Wide Web. The datasets and S functionsthat we use are supplied with most S
environmentsandarealsoavailableon-line.
Thisisnotatextinstatisticaltheory,butdoescovermodernstatisticalmethod-
ology. Each chapter summarizes the methods discussed, in order to set out the
notationandtheprecisemethodimplementedinS.(Itwillhelpifthereaderhas
abasicknowledgeofthetopicofthechapter,butseveralchaptershavebeensuc-
cessfully used for specialized courses in statistical methods.) Our aim is rather
toshowhowweanalysedatasetsusingS.Indoingsoweaimtoshowbothhow
S can be used and how the availability of a powerful and graphicalsystem has
alteredthewayweapproachdataanalysisandallowspenetratinganalysestobe
performed routinely. Once calculation became easy, the statistician’s energies
couldbedevotedtounderstandinghisorherdataset.
ThecoreSlanguageisnotverylarge,butitisquitedifferentfrommostother
statisticssystems. Wedescribethelanguageinsomedetailinthefirstthreechap-
ters,buttheseareprobablybestskimmedatfirstreading.Oncethephilosophyof
thelanguageisgrasped,itsconsistencyandlogicaldesignwillbeappreciated.
ThechaptersonapplyingStostatisticalproblemsarelargelyself-contained,
althoughChapter6describesthelanguageusedforlinearmodelsthatisusedin
severallaterchapters. Weexpectthatmostreaderswillwanttopickandchoose
amongthelaterchapters.
Thisbookisintendedbothforwould-beusersofSasanintroductoryguide
1http://www.insightful.com.
2http://www.r-project.org.
v
vi Preface
andforclassuse. Thelevelofcourseforwhichitissuitablediffersfromcountry
to country,butwouldgenerallyrangefromtheupperyearsofan undergraduate
course(especiallytheearlychapters)toMasters’level. (Forexample,almostall
the material is covered in the M.Sc. in Applied Statistics at Oxford.) On-line
exercises(andselectedanswers)areprovided,buttheseshouldnotdetractfrom
thebestexerciseofall,usingStostudydatasetswithwhichthereaderisfamiliar.
Our library providesmany datasets, some of which are not used in the text but
are there to providesourcematerial for exercises. Nolan and Speed (2000)and
RamseyandSchafer(1997,2002)arealsogoodsourcesofexercisematerial.
Theauthorsmaybecontactedbyelectronicmailat
[email protected]
andwouldappreciatebeinginformedoferrorsandimprovementstothecontents
of this book. Errataand updatesare availablefromourWorld Wide Web pages
(seepage461forsites).
Acknowledgements:
ThisbookwouldnotbepossiblewithouttheSenvironmentwhichhasbeenprin-
cipally developed by John Chambers, with substantial input from Doug Bates,
RickBecker,BillCleveland,TrevorHastie,DarylPregibonandAllanWilks. The
codeforsurvivalanalysisistheworkofTerryTherneau.TheS-PLUSandRim-
plementationsaretheworkofmuchlargerteamsacknowledgedintheirmanuals.
We are gratefulto the manypeople who have read and commentedon draft
materialandwhohavehelpedustestthesoftware,aswellastothosewhoseprob-
lemshavecontributedtoourunderstandingandindirectlytoexamplesandexer-
cises. We cannotname them all, butin particularwe would like to thank Doug
Bates, Adrian Bowman, Bill Dunlap, Kurt Hornik, Stephen Kaluzny, Jose´ Pin-
heiro,BrettPresnell,RuthRipley,CharlesRoosen,DavidSmith,PattySolomon
and Terry Therneau. We thank Insightful Inc. for early access to versions of
S-PLUS.
BillVenables
BrianRipley
January2002
Contents
Preface v
TypographicalConventions xi
1 Introduction 1
1.1 AQuickOverviewofS . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 UsingS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 AnIntroductorySession . . . . . . . . . . . . . . . . . . . . . . 6
1.4 WhatNext? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 DataManipulation 13
2.1 Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 DataManipulation . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 TablesandCross-Classification . . . . . . . . . . . . . . . . . . . 37
3 TheSLanguage 41
3.1 LanguageLayout . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 MoreonSObjects . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.3 ArithmeticalExpressions . . . . . . . . . . . . . . . . . . . . . . 47
3.4 CharacterVectorOperations . . . . . . . . . . . . . . . . . . . . 51
3.5 FormattingandPrinting . . . . . . . . . . . . . . . . . . . . . . . 54
3.6 CallingConventionsforFunctions . . . . . . . . . . . . . . . . . 55
3.7 ModelFormulae. . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.8 ControlStructures . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.9 ArrayandMatrixOperations . . . . . . . . . . . . . . . . . . . . 60
3.10 IntroductiontoClassesandMethods . . . . . . . . . . . . . . . . 66
4 Graphics 69
4.1 GraphicsDevices . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.2 BasicPlottingFunctions . . . . . . . . . . . . . . . . . . . . . . 72
vii
viii Contents
4.3 EnhancingPlots . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.4 FineControlofGraphics . . . . . . . . . . . . . . . . . . . . . . 82
4.5 TrellisGraphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5 UnivariateStatistics 107
5.1 ProbabilityDistributions . . . . . . . . . . . . . . . . . . . . . . 107
5.2 GeneratingRandomData . . . . . . . . . . . . . . . . . . . . . . 110
5.3 DataSummaries. . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4 ClassicalUnivariateStatistics . . . . . . . . . . . . . . . . . . . . 115
5.5 RobustSummaries . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.6 DensityEstimation . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.7 BootstrapandPermutationMethods . . . . . . . . . . . . . . . . 133
6 LinearStatisticalModels 139
6.1 AnAnalysisofCovarianceExample . . . . . . . . . . . . . . . . 139
6.2 ModelFormulaeandModelMatrices . . . . . . . . . . . . . . . 144
6.3 RegressionDiagnostics . . . . . . . . . . . . . . . . . . . . . . . 151
6.4 SafePrediction . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.5 RobustandResistantRegression . . . . . . . . . . . . . . . . . . 156
6.6 BootstrappingLinearModels . . . . . . . . . . . . . . . . . . . . 163
6.7 FactorialDesignsandDesignedExperiments . . . . . . . . . . . 165
6.8 AnUnbalancedFour-WayLayout . . . . . . . . . . . . . . . . . 169
6.9 PredictingComputerPerformance . . . . . . . . . . . . . . . . . 177
6.10 MultipleComparisons . . . . . . . . . . . . . . . . . . . . . . . 178
7 GeneralizedLinearModels 183
7.1 FunctionsforGeneralizedLinearModelling . . . . . . . . . . . . 187
7.2 BinomialData . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
7.3 PoissonandMultinomialModels . . . . . . . . . . . . . . . . . . 199
7.4 ANegativeBinomialFamily . . . . . . . . . . . . . . . . . . . . 206
7.5 Over-DispersioninBinomialandPoissonGLMs . . . . . . . . . 208
8 Non-LinearandSmoothRegression 211
8.1 AnIntroductoryExample . . . . . . . . . . . . . . . . . . . . . . 211
8.2 FittingNon-LinearRegressionModels . . . . . . . . . . . . . . . 212
8.3 Non-LinearFittedModelObjectsandMethodFunctions . . . . . 217
8.4 ConfidenceIntervalsforParameters . . . . . . . . . . . . . . . . 220
8.5 Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
Contents ix
8.6 ConstrainedNon-LinearRegression . . . . . . . . . . . . . . . . 227
8.7 One-DimensionalCurve-Fitting . . . . . . . . . . . . . . . . . . 228
8.8 AdditiveModels . . . . . . . . . . . . . . . . . . . . . . . . . . 232
8.9 Projection-PursuitRegression . . . . . . . . . . . . . . . . . . . 238
8.10 NeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8.11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
9 Tree-BasedMethods 251
9.1 PartitioningMethods . . . . . . . . . . . . . . . . . . . . . . . . 253
9.2 Implementationinrpart . . . . . . . . . . . . . . . . . . . . . . 258
9.3 Implementationintree . . . . . . . . . . . . . . . . . . . . . . 266
10 RandomandMixedEffects 271
10.1 LinearModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
10.2 ClassicNestedDesigns . . . . . . . . . . . . . . . . . . . . . . . 279
10.3 Non-LinearMixedEffectsModels . . . . . . . . . . . . . . . . . 286
10.4 GeneralizedLinearMixedModels . . . . . . . . . . . . . . . . . 292
10.5 GEEModels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
11 ExploratoryMultivariateAnalysis 301
11.1 VisualizationMethods . . . . . . . . . . . . . . . . . . . . . . . 302
11.2 ClusterAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
11.3 FactorAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
11.4 DiscreteMultivariateAnalysis . . . . . . . . . . . . . . . . . . . 325
12 Classification 331
12.1 DiscriminantAnalysis . . . . . . . . . . . . . . . . . . . . . . . 331
12.2 ClassificationTheory . . . . . . . . . . . . . . . . . . . . . . . . 338
12.3 Non-ParametricRules. . . . . . . . . . . . . . . . . . . . . . . . 341
12.4 NeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . 342
12.5 SupportVectorMachines . . . . . . . . . . . . . . . . . . . . . . 344
12.6 ForensicGlassExample . . . . . . . . . . . . . . . . . . . . . . 346
12.7 CalibrationPlots . . . . . . . . . . . . . . . . . . . . . . . . . . 349
13 SurvivalAnalysis 353
13.1 EstimatorsofSurvivorCurves . . . . . . . . . . . . . . . . . . . 355
13.2 ParametricModels . . . . . . . . . . . . . . . . . . . . . . . . . 359
13.3 CoxProportionalHazardsModel . . . . . . . . . . . . . . . . . . 365
x Contents
13.4 FurtherExamples . . . . . . . . . . . . . . . . . . . . . . . . . . 371
14 TimeSeriesAnalysis 387
14.1 Second-OrderSummaries . . . . . . . . . . . . . . . . . . . . . . 389
14.2 ARIMAModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
14.3 Seasonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
14.4 NottinghamTemperatureData . . . . . . . . . . . . . . . . . . . 406
14.5 RegressionwithAutocorrelatedErrors . . . . . . . . . . . . . . . 411
14.6 ModelsforFinancialSeries . . . . . . . . . . . . . . . . . . . . . 414
15 SpatialStatistics 419
15.1 SpatialInterpolationandSmoothing . . . . . . . . . . . . . . . . 419
15.2 Kriging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
15.3 PointProcessAnalysis . . . . . . . . . . . . . . . . . . . . . . . 430
16 Optimization 435
16.1 UnivariateFunctions . . . . . . . . . . . . . . . . . . . . . . . . 435
16.2 Special-PurposeOptimizationFunctions . . . . . . . . . . . . . . 436
16.3 GeneralOptimization . . . . . . . . . . . . . . . . . . . . . . . . 436
Appendices
A Implementation-SpecificDetails 447
A.1 UsingS-PLUSunderUnix/Linux . . . . . . . . . . . . . . . . . 447
A.2 UsingS-PLUSunderWindows. . . . . . . . . . . . . . . . . . . 450
A.3 UsingRunderUnix/Linux . . . . . . . . . . . . . . . . . . . . 453
A.4 UsingRunderWindows . . . . . . . . . . . . . . . . . . . . . . 454
A.5 ForEmacsUsers . . . . . . . . . . . . . . . . . . . . . . . . . . 455
B TheS-PLUSGUI 457
C Datasets,SoftwareandLibraries 461
C.1 OurSoftware . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
C.2 UsingLibraries . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
References 465
Index 481
Typographical Conventions
ThroughoutthisbookSlanguageconstructsandcommandstotheoperatingsys-
temaresetinamonospacedtypewriterfont like this. Thecharacter ~ may
appearas~onyourkeyboard,screenorprinter.
Weoftenusetheprompts $ fortheoperatingsystem(itisthestandardprompt
for the UNIX Bourne shell) and > for S. However, we do not use prompts for
continuation lines, which are indicated by indentation. One reason for this is
that the length of line available to use in a book column is less than that of a
standardterminalwindow,sowehavehadtobreaklinesthatwerenotbrokenat
theterminal.
Paragraphsorcommentsthatapplyto onlyoneS environmentare signalled
byamarginalmark:
• ThisisspecifictoS-PLUS(version6orlater). S+
• ThisisspecifictoS-PLUSunderWindows. S+Win
• ThisisspecifictoR. R
Some of the S output has been edited. Where complete lines are omitted,
theseareusuallyindicatedby
....
inlistings;howevermostblanklineshavebeensilentlyremoved. MuchoftheS
outputwasgeneratedwiththeoptionssettings
options(width = 65, digits = 5)
in effect, whereas the defaults are around 80 and 7. Not all functionsconsult
these settings, so on occasion we have had to manually reduce the precision to
moresensiblevalues.
xi
Chapter 1
Introduction
Statisticsisfundamentallyconcernedwiththeunderstandingofstructureindata.
One of the effects of the information-technologyera has been to make it much
easiertocollectextensivedatasetswithminimalhumanintervention.Fortunately,
thesametechnologicaladvancesallowtheusersofstatisticsaccesstomuchmore
powerful ‘calculators’ to manipulate and display data. This book is about the
modern developments in applied statistics that have been made possible by the
widespreadavailabilityofworkstationswithhigh-resolutiongraphicsandample
computationalpower. Workstationsneedsoftware,andtheS1 systemdeveloped
atBellLaboratories(LucentTechnologies,formerlyAT&T)providesaveryflex-
ibleandpowerfulenvironmentinwhichtoimplementnewstatisticalideas. Lu-
cent’scurrentimplementationof Sis exclusivelylicensedto the InsightfulCor-
poration2,whichdistributesanenhancedsystemcalledS-PLUS.
AnOpenSourcesystemcalledR3 hasemergedthatprovidesanindependent
implementationoftheSlanguage. Itissimilarenoughthatalmostalltheexam-
plesinthisbookcanberununderR.
AnSenvironmentisanintegratedsuiteofsoftwarefacilitiesfordataanalysis
andgraphicaldisplay.Amongotherthingsitoffers
• anextensiveandcoherentcollectionoftoolsforstatisticsanddataanalysis,
• a languageforexpressingstatistical modelsandtoolsforusinglinearand
non-linearstatisticalmodels,
• graphicalfacilities fordata analysisanddisplayeither ata workstationor
ashardcopy,
• an effective object-oriented programminglanguage that can easily be ex-
tendedbytheusercommunity.
Thetermenvironmentisintendedtocharacterizeitasaplannedandcoherent
systembuiltaroundalanguageandacollectionoflow-levelfacilities,ratherthan
the‘package’modelofanincrementalaccretionofveryspecific, high-leveland
1ThenameSaroselongagoasacompromisename(Becker,1994),inthespiritoftheprogram-
minglanguageC(alsofromBellLaboratories).
2http://www.insightful.com
3http://www.r-project.org
1
