ANOVA: A Paradigm for Low Power and Misleading Measures of Effect Size?

28 Pages·2008·0.64 MB·English

American Educational Research Association
ANOVA: A Paradigm for Low Power and Misleading Measures of Effect Size?
Author(s): Rand R. Wilcox
Source: Review of Educational Research, Vol. 65, No. 1 (Spring, 1995), pp. 51-77
Published by: American Educational Research Association
Stable URL: http://www.jstor.org/stable/1170478
Accessed: 01/01/2010 12:28
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.
Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/action/showPublisher?publisherCode=aera.
Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact [email protected].
American Educational Research Association is collaborating with JSTOR to digitize, preserve and extend
access to Review of Educational Research.
http://www.jstor.org
Review of Educational Research
Spring 1995, Vol. 65, No. 1, pp. 51-77
ANOVA: A Paradigm for Low Power and
Misleading Measures of Effect Size?
Rand R. Wilcox
University of Southern California
Over 30 years ago, Tukey made it evident that slight departures from
normality can substantially lower power when means are compared, and
that a popular measure of effect size can be highly misleading. At the time
there were no methods for dealing with the problem raised in Tukey'sp aper,
and some of the more obvious and seemingly intuitive solutions have since
been found to be highly unsatisfactory. Today there are practical methods
for not only dealing with the problem raised by Tukey, but also achieving
more accurate confidence intervals and control over the probability of a
Type I error. More generally, there are many robust and exploratory ways
of comparing groups that can reveal important differences that are missed
by conventional methods based on means, and even modern methods based
solely on robust measures of location. This article reviews these new tech-
niques.
Suppose two methods of teaching high school algebraa re being investigated.
One group learnsa lgebrau sing MethodA , anda n independentg roupu ses Method
B. Once data are collected, how shouldt hese two groupsb e compared?O f course,
the most common approachi n educationa nd psychology is to compare means,
though some would use the Mann-Whitney-Wilcoxonte st instead. Today there
are many alternatives olutionst hatc an reveali mportanat nd interestingd ifferences
that are missed by these more conventionalt echniques. Some of these newer
methods are based on replacingt he mean with some other measure of location.
The term measure of location is formallyd efined below, but for now it suffices
to think of it as a measure intended to representt he typical individual under
study. The mean and median are the two best-known examples of measure of
location, but other important measures of location are available which are
reviewed in this article. Much of this article deals with comparing measures of
location, but it is stressedt hat when attentioni s restrictedt o comparingm easures
of location, interesting differences between groups can be missed. One of the
main goals in this introductorys ection is to describem ethodst hat can supplement
techniques based solely on measures of location and why they can be useful.
These methods are generally called robust and exploratoryt echniques, and are
described in more detail in the two books by Hoaglin, Mosteller, and Tukey
(1983, 1985), which are writtena t a relativelyn ontechnicall evel. Here the goal
is to supplementt hese two books by describings ome recent graphicalt ools and
related techniques for comparing groups. A related goal is to point out that
popular measures of effect size can be highly misleading for reasons that are
explained below.
51
Wilcox
Before continuing, one point should be stressed. While this article is critical
of standardm ethods for comparingm eans and related measures of effect size,
it is not being implied that all publishedr esearchi s inaccurate.O n the positive
side, when a researcherr eportsa significantr esult when testing some hypothesis,
a truly significant result has probablyb een found. The concern is that many
nonsignificant results might have been highly significant if some alternative
method had been used, and there is the concern that significance levels and
reportedp robabilityc overage of confidence intervalsm ight be highly inaccurate.
A more generala ndp erhapsm orei mportanits sue is thatf lexibility and alternative
ways of viewing data can be very important.
To begin to appreciatew hy global comparisonso f distributionsc an be useful,
as opposed to comparing measures of location, first consider the problem of
comparing two independenta nd normally distributedr andom variables. More
realistic situationsa re consideredb elow. To be concrete, suppose Y corresponds
to an experimentalm ethodf or teachinga lgebrah aving mean xL,a, nd X represents
a control or standard method with mean I2. Suppose IJ = 92= 9, but Y has
variance &o2= .2, while X has variance oT = 1. Assume that high Y values
indicate that the experimentalm ethod is more effective than the control. Of
course, any method for comparing means should not reject Ho: [l- = ,p2, and for
the situationa t hand,n one of the methodsc onsideredi n this article for comparing
alternativem easures of location should reject, either. Yet there is a potentially
importantd ifference:S tudentsw ho do poorly undert he standardm ethod benefit
from the experimentalm ethod, but the experimentalm ethod is detrimentalf or
students who do well using the standardt echnique.
One way of dealing with this problemi s to comparet he quantiles of the two
groups.F ort he situationa t hand,o ne mighta lso comparev ariances,b ut comparing
quantiles can be more revealing for reasons that will become evident. Let xp be
the pth quantilec orrespondingt o the randomv ariableX . Thus, x5 is the median.
Considerg raphingx pv ersus 8(xp)= yp - xp,t he differenceb etween the quantiles.
If 8(xp)> 0, students who are at the pth quantileo f the standardm ethod would
typically perform better under the experimentalm ethod; and if 6(xp) < 0, the
experimentalm ethod is detrimental.F igure 1 shows a graph of 6 versus xp for
the two normald istributionsi n the illustrationf or the deciles, p = .1, .2, .3, .4,
.5, .6, .7, .8 and .9.
The quantity6 (xp)i s knowna s a shift functiona ndh as been studiedb y Doksum
(1974, 1977) as well as Doksum and Sievers (1976). One concern is that groups
might differ in more complicatedw ays than is the case in the illustrationj ust
given, and Doksum (1977) reanalyzess ome data on the effect of radiationo n
mice to demonstratet his point. Here, data analyzed by Salk (1973) is used to
provide yet anotheri llustration.A portiono f Salk's study dealt with weight gain
in newborns weighing at least 3,500 grams at birth.T hese newborns were sepa-
rated into two groups.T he first groupw as continuouslye xposed to the recorded
sound of a humanh eartbeat,w hile the otherg roupa cted as a control. The equality
of the means of the two groups was tested with Student's t test and rejected at
the .05 level of significance. Figure 2 shows an estimate of the shift function
where the quantiles are estimated using the method derived by Harrell and
Davis (1982). The Minitabm acrod shift.mtbi n Wilcox (in press-c) performst he
necessary computations.T he point here is that the graph descends, levels off,
52
.
.
*
xp
0 8 9. 10 11 12
FIGURE1 . Shiftf unctionf or a hypotheticanl ew teachingm ethod
120+
*
90+ *
*
60+
* ** *
*
30+
+---------+ ------------------+-------+----------------
-200 -150 -100 -50 0 50
FIGURE 2. Estimated shift function for data on weight gain in newborns
Note. Data are from Salk (1973).
Wilcox
and then descends again. This indicates that the largest effect occurs for babies
who have a relatively large loss of weight withoutt he treatment.C learly 8(xp)
can be importantb ecause it provides a more refinedm ethod of determiningw ho
might benefit from a particulart reatment.O f particulari mportancei s that 8(xp)
can reflect differencesn ot indicatedb y tpi - ,12o r by the difference in any other
measure of location. A method for obtainings imultaneousc onfidence intervals
for xp - yp when working with deciles is outlined in Wilcox (in press-c); the
necessary computationsa re done with his Minitabm acro mq.mtb,a nd details of
how this method performsc an be found in Wilcox (in press-b).
Several other methodsh ave been proposedw hich also capturet he spirit of the
shift function. The best known is the Kolmogorov-Smirov test, but it has rela-
tively low power, and its power can increases lowly as the sample sizes get large
(Randles& Wolfe, 1979, p. 384). Doksuma ndS ievers( 1976) deriveda confidence
bandf or all quantilesu sing a Kolmogorov-Smirov statistic,a nd a relatedm ethod
was derived by Switzer (1976). Earlierr elatedw ork is the quantile-quantilep lot
of Gnanadesikana nd Wilk (1968), and a methodb ased on ranks was proposed
by O'Brien (1988). O'Brien provides additionali llustrationso f why these tech-
niques can be important.Y et anotherg raphicala pproachi s to draw a boxplot of
two groups, one on top of the other and on the same scale. An illustration is
given in Wilcox (in press-c, chapter 8). For results on the properties of the
boxplot, see Brant (1990). For graphicalm ethodsa imed at comparing multiple
groups, see Tukey (1993). Finally, global measureso f how groups differ can be
obtainedb y estimatingt he distributionsc orrespondingto each group using meth-
ods such as those in Silverman (1986).
Some Practical Reasons for Using Robust Measures of Location and
Effect Size
Now consider the problem of comparingg roups based on some measure of
location. There are two related concerns about power when sample means are
used: outliers and heavy-tailed distributions.T o illustratet he first, consider the
data in Wilcox (1992d) which deals with a study on self-awareness.T he sample
means are 448 and 599. Using Welch's test of Ho:[ LI> L2,t he significance level
is .24. However, comparingt he groups in termso f 20% trimmedm eans or one-
step M-estimators,w hich are robustm easureso f locationd escribedb elow, yields
a significant difference at the .05 level. Moreover, various papers described
below indicate that these tests generally provide better control over Type I
errorp robabilities,e specially when distributionsa re skewed. The reason for the
discrepantr esults is that the first grouph as two extremeo utliers (unusuallyl arge
values relative to the bulk of the available data)t hat are revealed by a boxplot,
and the second grouph as an extremeo utlier,a s well. Outliersi nflate the standard
erroro f the sample mean, which, in turn,l owers power. Many robuste stimators
of location have standarde rrorst hat are relativelyi nsensitive to outliers, and in
the illustrationt his is why it was possible to reject. However, even when there
are no outliers, differentm easureso f locationc an give differentr esultsf or reasons
explained below.
Ever since Tukey (1960) published his work on the contaminated normal
distribution, it has been clear that slight departuresf rom normality can have
devastating effects on power when means are compared,a nd that popular mea-
54
ANOVA
sures of effect size can be misleading, as well. The contaminatedn ormal is an
example of a distributionw ith a heavy or thick tail relativet o the normald istribu-
tion. Tukey's paperh ad no immediatei mpacto n applied researchb ecause it was
unclear at the time how a researcherm ight address the problem he discussed.
During the ten years following Tukey's paper,a few mathematicals tatisticians
began laying a foundationf or getting betterr esults. Of particulari nterest is the
theory of robustness developed by Huber (1964) and Hampel (1968). By the
year 1974, therew ere some practicals olutionsf or the one- and two-sample cases,
but general techniques for more complicatedd esigns were not available even a
few years ago. Today,a ny of the commone xperimentald esigns can be improved.
Figure 3 shows a standardn ormala nd a contaminatedn ormald istribution.A
contaminatedn ormali s formedb y samplingf rom a standardn ormald istribution
(having mean 0 and variance 1) with probability1 - e; alternatively,s ampling
is from a normald istributionh aving mean 0 and standardd eviation K. (Readers
interestedi n furthert echnical details can refert o Hoaglin et al., 1983). In Figure
I, E = .1 and K = 10. Therei s an obvious similarityb etweent he two distributions.
The tails of the contaminatedn ormala ctually lie above the tails of the normal,
but this is difficult to discernw ithoute xtendingt he rangeo f x values and drawing
the figure on a much larger scale. The main point is that while the standard
normal has variance 1, the contaminatedn ormalh as variance 1 - E + EK2=
10.9. Put anotherw ay, if samplingi s from the contaminatedn ormal, even when
the variance is known, the length of the confidence interval for the population
mean ,L, will be over 3 times longer than what it is when sampling from the
normal distributioni nstead, and this has obvious implications for power. The
problem is that or, the standardd eviation, is highly sensitive to the tails of a
distribution,s o seemingly small departurefsr omn ormalityc an inflate the standard
error of the sample mean, u/;n, where n is the sample size. One might try to
salvage the situation by arguing that the contaminatedn ormal distribution is
actually a large departuref rom normality.T he extreme quantiles differ substan-
tially, which results in large differencesb etween the standardd eviations. But in
terms of common metrics for comparingd istributions( the Kolmogorov, Levy,
and Prohorov, the latter two being discussed by Huber, 1981), there is little
difference between the normal and contaminatedn ormal. Gleason (1993) com-
pares them in terms of what he calls elongation and concludes that there is
little difference.
f(x)
.....o.. normal curve
*
.......* .. '?...... . X
-3 -2 -1 0 1 2 3
FIGURE3 . Normala nd contaminatedn ormald istributions
55
Wilcox
To add some perspective,l ook at Figure4 , which shows two normald istribu-
tions with means 0 and 0.8, both having variance 1. Consider the common
measure of effect size
A = 1I - Ix21
or
where p,ji s the mean of the jth group, and the groups have a common standard
deviation a, = r2 = o. Cohen (1977, p. 26) defines a medium measure of effect
size as one largee nought o be visible to the nakede ye, and for normald istributions
he concludes that A values equal to .2, .5, and .8, correspondt o small, medium,
and large effect sizes. Thus, Figure4 has A = .8, which is large. Now look at
Figure 5, which shows two contaminatedn ormals, again with means 0 and 0.8.
Then A = .24, which is supposedly small, yet by appearancest he effect size
is large.
It has already been illustratedt hat outliers can substantiallyr educe power
when one is working with sample means, and in a similar manner,h eavy-tailed
distributionsi nflate the standarde rror of the sample mean. As a result, power
might be low relative to methods based on other measures of location. For
example, suppose samplingi s from two normal distributionsw ith R.I- Ip2 = 1
and variances equal to 1. Then with o = .05, and n\ = n2 = 25, Student's t test
has power .957, while Welch's( 1951) test has power approximatelye qual to .93.
If, instead, sampling is from contaminatedn ormal distributions, again with
f(x)
-3 -2 -1 0 1 2 3
FIGURE 4. Graphical display of A .8 for two normal distributions
f(x)
.....**::.:. ***" ''*-*..: :::*.. , x
-3 -2 -1 0 1 2 3
FFIIGGUURREE55 .. GGrraapphhiiccaalldd iissppllaayyoo ff AA == ..2244 ffoorr ttwwoo ccoonnttaammiinnaatteeddnn oorrmmaallss
56
ANOVA
-i1 - 12 = 1, then these two proceduresh ave power approximatelye qual to .28,
based on simulationsw ith 10,000 replications, and this agrees with the usual
approximationf or power derived under the assumption of normality.H owever,
powers for three other proceduresc omparedb y Wilcox (1992d) range between
.78 and .80, with little loss in power when distributionsa re normal.O ne of these
is Yuen's (1974) methodf or trimmedm eans, based on 20% trimming,w hile the
other two are recently developed methods for the median and a measure of
location called a one-stepM -estimator(. More details are given lateri n the article.
Computationald etails, plus Minitab macros for applying these techniques, can
be found in Wilcox, in press-c). Thus, to the extent that one wants to reject when
groups differ, these newer proceduresa re of interest. There are a variety of other
issues that one might considerb efore choosing a procedure;t hese are discussed
below. Note that when dealing with power undern onnormality,G lass, Peckham,
and Sanders (1972) focus exclusively on power as a function of A. Hence, the
results just reporteda re consistent with their paper, but the point here is that
other proceduresc an have substantiallym ore power.
To describe yet anotherp roblemr elatedt o outliers, let X, and X2b e the sample
means correspondingt o two groups, and suppose X2 > Xi, and that Ho: pl = 1x2
is rejected with ao = .05 using either Welch's method or Student's t test. Now
suppose the largest observation in the second group is made even larger, so
X2 - X increases as well. It would seem that there is stronger evidence for
concluding that 12 > RILy, et eventually both tests for means will not reject
because of the correspondingin creasei n the standarde rror.I llustrationsa re given
in Staudte and Sheather( 1990) as well as Wilcox (in press-c).
It should be stressedt hato utliersa re not the only reason for consideringr obust
methods. Small shifts in a distributionc an have a large impact on the mean
which might renderi t a potentiallym isleading measure of the typical individual.
Consider,f or example,a chi-squared istributionw ith 4 degrees of freedom which
has mean 4. If the distributioni s contaminatedb y multiplying an observationb y
10 with probability. 1, it does not change very much based on typical metrics
for comparing distributions.H owever, the mean is affected considerably as
depicted in Figure 6, which also shows the location of the median, 0 = 3.75,
and the 20% trimmed mean, 11t= 4.2, which is discussed in more detail in
f(z)
. = 3.75
.*" '*-.pt ' = 4.2
=.- . p 7.6
? x
0 2 4 6 8 10 12 16
FIGURE 6. Measures of location for a contaminated chi-square distribution
57
Wilcox
subsequents ections. The mean is increased to 7.6, and its value is so far in the
rightt ail, one mighta rguet hati t is not a very good reflectiono f the typicalp erson.
During the early years of modem mathematicals tatistics, there were reasons
to suspectt hats tricta dherencet o means, when describinga nd comparingg roups,
might be a problem( Pearson,1 931; Student, 1927). These resultsw ere overshad-
owed by theoreticalr esults, summarized by Tan (1982), indicating that when
two or more groupsh ave identical distributions,c onventionalh ypothesis testing
proceduresp rovide good control over the probability of a Type I error when
distributionsa re nonnormalI. n terms of power, however, therea re several serious
problems.T ukey's (1960) results imply that departuresf rom normalityt hat are
difficult to detect can inflate the variance by a large amount, and this implies
that power will be lowered. Moreover,T ukey argued that heavy-tailedd istribu-
tions and outliersa re common in applied research.I f this is true, low power due
to nonnormalityc ould be common. Surveys of data occurringi n applied work
indicate that severe departuresf rom normality can arise in practice (Micceri,
1989; Wilcox, 1990a).
The remaindero f this article provides a brief and relatively nontechnical
introductiont o robustm easureso f location and scale, and how they can be used
to compare groups. Some of these issues are summarizedb y Wilcox (1992d,
1993d). This article takes a much broader view of the problems that arise and
the solutions that have been proposed. Elementaryd escriptionso f many of the
recently developed robustm ethods examined here are given in the textbook by
Wilcox (in press-c). With Wilcox's book comes a floppy containing over 150
Minitabm acrosf or applyingt echniquesn ot found in standards tatisticalp ackages.
Consequently,n umericald escriptionso f these methods are kept to a minimum.
Here, attentioni s focused on general technical issues and practicald etails that
are relevantt o experiencedr esearchersw ho want a relativelyn ontechnicalu nder-
standingo f how and why modem robust methods are relevantt o their research.
Building a Mathematical Foundation for More Robust Methods
Appliedr esearchersw ho do not religiously follow developmentsi n mathemati-
cal statistics might still have the impression that robust methods are based on
ad hoc procedures.T he purpose of this section is to indicate that this is not at
all the case. While technical details are kept to a minimum,s ome mathematical
comments are needed in ordert o convey how the foundationso f modem robust
proceduresw ere developed.
There are two fundamentala nd related problems that must be addressed.T he
first is finding analogs of the population mean and variance, ,L and a2, that are
relativelyi nsensitivet o slight changes in a distribution.W hen dealing with effect
size, for example,i t is desirablet o have a measuret hat is consistentw ith A when
distributionsa ren ormal,b ut which does not give a distortedv iew of the magnitude
of the effect size unders light departuresf rom normality.T he second problemi s
findinge stimatorso f locationa nd scale with standarde rrorst hata re also relatively
insensitivet o slight changes in a distribution.I n particulars, mall departuresfr om
normalitys hould not drasticallyc olor our perceptions about how groups differ,
nor should they substantiallyl ower power. Today there is a well established
mathematicalf oundationf or addressingt hese problems which is summarizedi n
the books by Huber( 1981); Hampel, Ronchetti, Rousseeuw, and Stahel (1986);
58
ANOVA
and Staudtea nd Sheather( 1990). This section briefly reviews some aspects of
these resultst hat are particularlyr elevant to appliedr esearchers.O ne of the most
strikingr esults is that intuitionsa bout how to deal with outliersa nd heavy-tailed
distributions,b ased in part on notions related to normal distributions,t urn out
to be completely unsatisfactory.F or example, discardingo utliers and applying
Student'st test to the data that remain results in a procedureb ased on estimated
standarde rrorst hat are not even asymptotically correct. That is, the estimated
standarde rrorsd o not converge to the correct value as the sample sizes increase.
Anotherc ommon misconceptioni s that outliers can be detectedu sing rules based
on s, the sample standardd eviation. Because s can become highly inflated due
to outliers, methods for detecting outliers that are based on s are subject to
"masking."T hat is, the very presence of outliers hinderst he ability of methods
based on s to find them. A more satisfactory approachi s to use a measure of
dispersion that is insensitive to outliers. (See Brant, 1990; Davies & Gather,
1993; Hadi, 1994; Barett & Lewis, 1984; Goldberg & Iglewicz, 1992;
Rousseeuw & Leroy, 1987; and Rousseeuw & van Zomeren, 1990 for methods
dealing with the detection of outliers.)
Threed ifferentw ays of assessing an estimatorp lay a prominentr ole in modem
statisticalm ethods: (a) the breakdownp oint, (b) the influence function, and (c)
the continuityo f a descriptivem easure.T hese concepts are describedb y Staudte
and Sheather( 1990, p. 44) as quantitativer obustness,i nfinitesimalr obustness,
and qualitative robustness. But before delving into greater detail, we should
perhaps be more precise about what is meant by a measure of location. It is
intendedt o representt he typical subject,b ut what basic propertiess houldi t have?
There are four properties that are usually imposed (e.g., Staudte & Sheather,
1990, p. 101), and a fifth condition (described below) that is sometimes added.
SupposeX has distributionF , and let 0(X) be some descriptivem easureo f the
distribution.T hen 0(X) is a measure of location if for any constantsa and b,
0(X + b) = 0(X) + b, (Condition 1)
0(-X) = -0(X), (Condition2 )
X > 0 implies 0(X) > 0, and (Condition3 )
0(aX) = a0(X). (Condition4 )
Condition 1 is called location equivariancea nd simply requirest hat if b is added
to every observation,t hen a measureo f location should be increasedb y the same
amount.C onditions 1-3 imply that any measure of location should have a value
within the range of values of X. Condition 4 is called scale equivariancea nd
guaranteest hat estimators of measures of location, such as the sample mean,
give results independento f the unit of measurement.I n the context of testing
hypotheses, it should not be possible to multiply all the observationsb y the
constanta and come to different conclusions about whether two groups differ.
There are many measures of location in addition to the mean and median. To
makej udgmentsa bout how measures of location compare,a dditionald esiderata
must be imposed.
The discussion of the contaminatedn ormal and chi-squared istributionss ug-
gests one approacht o characterizingr obust and resistantm easureso f location,
and these distributionsa re examples of what is called a contaminationn eighbor-
59