Table Of ContentALTERNATIVE APPROACHES TO THE
ASSESSMENT OF ACHIEVEMENT
Evaluation in Education and
Human Services
Editors:
George F. Madaus, Boston College, Chestnut
Hill, Massachusetts, U.S.A.
Daniel L. Stufflebeam, Western Michigan
University, Kalamazoo, Michigan, U.S.A.
Previously published books in the series:
Kellaghan, T., Madaus, G., and Airasian, P.:
The Effects of Standardized Testing
Madaus, G. (editor):
The Courts, Validity, and Minimum
Competency Testing
Brinkerhoff, R., Brethower, D., Hluchyj, T., and
Nowakowski, J.:
Program Evaluation, Sourcebook/ Casebook
Brinkerhoff, R., Brethower, D., Hluchyj, T., and
Nowakowski, J.:
Program Evaluation, Sourcebook
Brinkerhoff, R., Brethower, D., Hluchyj, T., and
Nowakowski, J.:
Program Evaluation, Design Manual
Madaus, G., Scriven, M., Stufflebeam, D.,;
Evaluation Models: Viewpoints on Educational
and Human Services Evaluation
Hambleton, R., Swaminathan, H.
Item Response Theory
Stufflebeam, D., Shinkfield, A.;
Systematic Evaluation
Nowakowski, J.,
Handbook of Educational Variables:
A Guide to Evaluation
Stufflebeam, D.;
Conducting Educational Needs Assessments
Abrahamson, Stephen;
Evaluation of Continuing Education
in the Health Professions
Cooley, William, Bickel, William;
Decision-oriented Educational Research
Gable, Robert K.;
Instrument Development in the
Affective Domain
Sirotnik, Kenneth A., Oakes, Jeannie;
Critical Perspectives on the Organization
and Improvement of Schooling
Wick, John W.;
School-based Evaluation: A Guide for
Board Members, Superintendents, Principals,
Department Heads and Teachers
Worthen, Blaine R., White, Karl R.;
Evaluating Educational and Social Programs
ALTERNATIVE APPROACHES TO THE
ASSESSMENT OF ACHIEVEMENT
Edited by
David L. McArthur
UCLA Graduate School of Education
Center for the Study of Evaluation
~
"
Kluwer Academic Publishers
Boston/ Dordrecht/ Lancaster
DiSlribUIOU for NOrib Americl:
Klu ..... er Aeademic Puhli~hers
101 Philip Drive
Assinippi Park
Norwell. MA 02061 USA
DislribulOrs for Ibt UK 3nd Ireland:
Kluwcr Academic Publishers
MTP Press Limited
Falcon House. Queen Square
lancaster LAI IRN UNITED KINGDOM
Oi~tribulo", for all olber countri",
Kluwer Academic Publishers Group
Distribution Cellire
POSt Office BOJ! 322
3JOO A H Dord recht
The Netherlands
Library of C.mgrtss Cililoging-in-Publin.lion Oltl
Ahernative approache~ to the assessment of
achie~·emem.
(Evaluation in education and human services)
Includes bibliographies alld inde~.
I. &locationall"sts and measurements.
2. Educational tcsts and measuremems- Mathcmatical
models. J. Academic achie""menl- Tesling
Mathematical models. I. McAnhur. David.
II. Series. [O;\'lM: I Educational Measuremenl
methods. 2. Evaluation Studies. LB 3051 A466]
LB3051.A567 1986 )71.2'6 86-27483
tSIIN-!J: 1178·\1-1-011).7\161-7 <-tSBN-!J: \178-\1-1-009-3257-9
DOt: to.l007I97tt-9-I-009-J257-9
Copyright " 19S7 by Kluwer Academic I'ublishers
Softco,"er reprint of the hardco,"er I st edit ion \987
All riShlS reserved. No part uf this publication may be reproduced. SlUred in a retrieval system. or
transmiltcd ill an)' form or b)' an)' means. mechanical.llhotocopying. reo;ording. or otherwise. without
the prioT wTiuen permission of the publisher. KIII",eT Academic Publishers. Philip Drive. Assinippi
Park. Nor ....- cll. Massachu\ClIS 02061. USA.
CONTENTS
Contributors Vll
Preface IX
Introduction Xlll
Bruce H. Choppin and David L. McArthur
Chapter 1: Educational Assessment: A Brief History
David L. McArthur
Chapter 2: Toward More Sensible Achievement Measurement: 21
A Retrospective
Kenneth A. Sirotnik
Chapter 3: Analysis of Patterns: The S-P Technique 79
David L. McArthur
Chapter 4: The Rasch Model for Item Analysis 99
Bruce H. Choppin
Chapter 5: The Three-Parameter Logistic Model 129
Ronald K. Hambleton
Chapter 6: Measuring Achievement with Latent Structure Models 159
Rand R. Wilcox
Chapter 7: Generalizability Theory and Achievement Testing 187
Noreen M. Webb
Chapter 8: Analysis of Reading Comprehension Data 233
David L. McArthur
Chapter 9: A Comparison of Models for Measuring Achievement 249
J. Ward Keesling
Index 267
LIST OF CONTRIBUTORS
Bruce H. Choppin, PhD
deceased; Dr. Choppin was Director of the Methodology Research
program at the Center for Study of Evaluation
Graduate School of Education
University of California Los Angeles
Ronald K. Hambleton, PhD
Division of Educational Policy, Research and
Administration
University of Massachusetts at Amherst
Amherst, MA 01003
J. Ward Keesling, PhD
Computer Resources Special Project
Advanced Technology, Inc.
Camarillo, CA 93010
David L. McArthur, PhD
Center for Student Testing, Evaluation and Standards
University of California Los Angeles
Graduate School of Education
Los Angeles, CA 90024
Kenneth A. Sirotnik, PhD
Policy, Governance and Administration
College of Education
University of Washington
Seattle, WA 98195
Noreen M. Webb, PhD
Research Methods in Education
Graduate School of Education
University of California Los Angeles
Los Angeles, CA 90024
Rand R. Wilcox, PhD
Quantitative Area
Department of Psychology
University of Southern California
Los Angeles, CA 90089
PREFACE
Ingrained for many years in the science of educational assessment
were a large number of "truths" about how to make sense out of testing
results, artful wisdoms that appear to have held away largely by force
of habit alone. Practitioners and researchers only occasionally
agreed about how tests should be designed, and were even further apart
when they came to interpreting test responses by any means other than
categorically "right" or "wrong." Even the best innovations were
painfully slow to be incorporated into practice.
The traditional approach to testing was developed to accomplish
only two tasks: to provide ranking of students, or to select
relatively small proportions of students for special treatment. In
these tasks it was fairly effective, but it is increasingly seen as
inadequate for the broader spectrum of issues that educational
measurement is now called upon to address.
Today the range of questions being asked of educational test data
is itself growing by leaps and bounds. Fortunately, to meet this
challenge we have available a wide panoply of resource tools for
assessment which deserve serious attention. Many of them have
exceptionally sOphisticated mathematical foundations, and succeed well
where older and less versatile techniques fail dismally. Yet no
single new tool can conceivably cover the entire arena.
Our intent in this book is to convey to the reader a
well-balanced appreciation for the diversity of alternative
methodologies for educational assessment. To do this we have
attempted to set a stage which portrays both a cross-section of
methods and an intersection of the principles which underlie them.
Included are evaluations of both the strengths and weaknesses of every
approach, as well as completely worked-out examples which demonstrate
their present utility, and informed speculations about future
developments and applications to which they could be put.
x
Most mathematical models, at their heart, are quite simple; the
statistical tricks are in place solely to make them work efficiently.
From the reader who may be a trifle rusty in statistics, we ask only
for patience if at times the various formulae for number--crunching
make tough reading. Readers experienced in designing and interpreting
educational tests may have an advantage on this score, though the book
is also designed to be useful at introductory levels. Moreover, the
essential ingredients of this book are not the formulae themselves but
the competing rationales which precede them, and the variety of
conclusions which follow.
This book is the culmination of a project first set in motion by
the late Bruce Choppin. Its overall design is his, and it was his
imagination which first brought the contributors together to consider
detailing alternative models for the assessment of educational
achievement using a common framework. His desire to give full and
fair scrutiny to contrasting viewpoints and his ability to draw
connections across rival hypotheses with remarkable clarity, together
with his wide command of theory and practice, were a strong influence
on all who had the chance to work with him. One intent in this book
has been to capture Bruce's original sense of marvel at how good
questions can be raised--and answered well--in the field of
educational assessment.
Bruce suffered an untimely death while on the first leg of a
consulting trip around the world. He continues to be missed by his
many friends and colleagues.
My heartfelt thanks go to each of the authors for their
participation, and their willingness to develop each of the chapters
in this book into documents which integrate their views along a common
strand of critical issues. We hope that the reader will find this
means of presenting the several alternative methods to be one which
elucidates the essential philosophical bases beneath the measurement
process. We have also strived to provide a consistent technical
vocabulary necessary to convey statistical concepts, and frequent
guideposts to applying the several models in actual practice.
XI
Support for work presented in this document was originally
provided by the National Institute of Education, Grant Number
NIE-G-80-112, P1. The opinions expressed herein do not necessarily
represent the position or policy of the Institute, now known as the
Office of Educational Research and Improvement, and no official
endorsement should be inferred. Additional support was generously
provided by the Center for Student Testing, Evaluation and Standards
at UCLA. Special appreciation is extended to Katharine Fry, Ruth
Pays en and Dan Reich for diligent typing of each contribution.
David L. McArthur
INTRODUCTION
Fragmentation is occurring within the psychometric field.
Dissatisfaction with the limitations inherent in traditional forms of
mental test analysis typified by the norm-referenced multiple-choice
test of achievement, has led in recent years to a variety of new
psychometric theories and procedures. Novel applications have
stimulated new psychometric models and methods, each shaped to deal
with the specific problems of the particular situation. The last two
decades have seen the development of new types of tests, new scoring
methods, new procedures for item analysis, and entirely new
conceptions of the mental measurement process.
A marked characteristic of the professional literature on these
novel approaches to educational measurement is its parochialism. Many
of the most prolific psychometricians display little interest in
models other than their own, and there have been few attempts to
integrate separate results. The proponents of different models
have different objectives which on first impression appear to be
mutually exclusive. Implicitly or explicitly they make widely
differing assumptions; frequently they use the same words and phrases
to mean different things (e.g., reliability, accuracy, guessing, error
and true-score). Separate methodologies based on different models
have diverged to a point where it is no longer possible to identify a
mainstream approach to educational measurement, and where informed and
balanced advice on the full range of alternative approaches is almost
impossible to obtain.
The primary goal of this book is to build a convergent
perspective on psychometrics by describing, fairly and
comprehensively, the rationales that underlie several of the leading
