Table Of ContentDOCUMENT RESUME
FL 026 620
ED 451 701
AUTHOR Norris, John M.; Brown, James Dean; Hudson, Thom; Yoshioka,
Jim
Designing Second Language Performance Assessments. Technical'
TITLE
Report.
Hawaii Univ., Manoa. Second Language Teaching and Curriculum
INSTITUTION
Center.
SPONS AGENCY Department of Education, Washington, DC.
REPORT NO
TR-18
ISBN-0-8248-2109-2
ISBN
1998-00-00
PUB DATE
NOTE
237p.
CONTRACT P229A60007
University of Hawaii Press, Order Department, 2840 Kolowalu
AVAILABLE FROM
Street, 96822 ($12). Web site:
http://www.uhpress.hawaii.edu.
Reports
PUB TYPE Research (143)
EDRS PRICE
MF01/PC10 Plus Postage.
Evaluation Methods; *Evaluation Research; *Language Tests;
DESCRIPTORS
Measures (Individuals); *Second Language Instruction; Second
Language Learning; Test Construction; *Test Reliability;
*Test Validity
ABSTRACT
This technical report focuses on the decision-making
potential provided by second language performance assessments. First,
performance assessment is situated within the broader discussion of
alternatives in language assessment and in educational assessment in general.
Next, issues in performance assessment design, implementation, reliability,
and validity are discussed. Finally, a prototype framework is presented for
second language performance assessment based on the integration of
theoretical underpinnings and research findings from the literature on
task-based language teaching, language testing, and educational measurement.
Test and item specifications are outlined, and numerous examples of
prototypical language tasks are presented. A research agenda is proposed,
focusing on the operationalization of second language performance
assessments. (Contains 112 references.)
(KFT)
Reproductions supplied by EDRS are the best that can be made
from the original document.
,1t
U S DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
EDUCATIONAL RESOURCES INFORMATION
CENTER (ERIC)
This document has been reproduced as
received from the person or organization
originating It
Minor changes have been made to
improve reproduction quality.
Paints of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.
1
DESIGNING
SECOND
LANGUAGE
PERFORMANCE
ASSESSMENTS
JOHN M. NORRIS
JAMES DEAN BROWN
THOM HUDSON
JIM YOSHIOKA
r)
4
D
DESIGNING
SECOND
LANGUAGE
PERFORMANCE
ASSESSMENTS
3
TECHNICAL REPORT #18
DESIGNING
SECOND
LANGUAGE
PERFORMANCE
ASSESSMENTS
JOHN M. NORRIS
JAMES DEAN BROWN
THOM HUDSON
JIM YOSHIOKA
SECOND LANGUAGE TEACHING & CURRICULUM CENTER
Si
at Manoa
of Hawai`i
University
O 1998 Second Language Teaching & Curriculum Center
University of Hawai`i
All rights reserved
Manufactured in the United States of America
The contents of this Technical Report were developed under
a grant from the
Department of Education (CFDA 84.229, P229A60007). However, the
contents do
not necessarily represent the policy of the Department of Education, and one should
not assume endorsement by the Federal Government.
ISBN 0-8248-2109-2
The paper used in this publication meets the minimum requirements of the
American National Standard for Information SciencesPermanence of Paper for
Printed Library Materials.
ANSI Z39.48-1984
Book design by Deborah Masterson
Distributed by
University of Hawai'i Press
Order Department
2840 Kolowalu Street
Honolulu, HI 96822
5
ABOUT THE NATIONAL FOREIGN LANGUAGE
RESOURCE CENTER
THE SECOND LANGUAGE TEACHING AND CURRICULUM CENTER of the University
of Hawai'i is a unit of the College of Languages, Linguistics, and Literature. Under a
grant from the US Department of Education, the Center has since 1990 served as a
National Foreign Language Resource Center (NFLRC). The general direction of
the Resource Center is set by a national advisory board. The Center conducts
research, develops materials, and trains language professionals with the goal of
improving foreign language instruction in the United States. The Center publishes
research reports and teaching materials; it also sponsors a summer intensive teacher
training institute. For additional information about Center programs, write:
Dr. Richard Schmidt, Director
National Foreign Language Resource Center
1859 East-West Road #106
University of Hawai`i
Honolulu, HI 96822
or visit our Web site: http: / /www.11l.hawaii.edu /nflrc
v
NF L RC ADVISORY BOARD
Kathleen Bardovi-Harlig
Program for TESOL and Applied Linguistics
Indiana University
John Clark
Defense Language Institute (Emeritus)
Monterey, California
James Pusack
Department of German
University of Iowa
vi
CONTENTS
CHAPTER ONE: ALTERNATIVES IN SECOND LANGUAGE ASSESSMENT
1
"Alternative Assessments"
2
Alternatives Assessment
3
in
7
CHAPTER TWO: PERFORMANCE ASSESSMENTS
What Are Performance Assessments?
7
9
What Should a Performance Assessment Look Like?
10
How Are Performance Assessments Developed?
Why Bother With Performance Assessment?
. 14
What Problems Occur in Performance Assessment?
16
.
21
What Steps Can Be Taken to Avoid Performance Assessment Problems?
Where Do Performance Assessments Fit Into a Language Curriculum?
27
.30
What Examples Exist of Actual Performance Assessment Projects?
.30
Summary
31
CHAPTER THREE: TASK-BASED LANGUAGE TEACHING
.32
What Are Tasks?
35
What Is the Role of Needs Analysis in Task Selection?
.
39
What Are the Factors that Affect Task Difficulty and Sequencing?
53
CHAPTER FOUR: TASK-BASED ASSESSMENT
53
How Do We Assess Task-based Performance?
.
60
What Are the Factors that Affect Task-based Assessment Reliability?
.
What Are the Factors that Affect Task-based Assessment Validity?
. 61
. 64
What Are the Factors that Affect Task-based Assessment Practicality?
65
Developing Task-based
What Are the Steps Involved
Assessment?
in
.
69
CHAPTER FIVE: TEST AND ITEM SPECIFICATIONS
. 70
Test Specifications
82
Item Specifications
vii
CHAPTER SIX: ITEM PROMPTS
99
CHAPTER SEVEN: CONCLUSIONS
137
Immediate Plans for Research
137
Suggestions for Future Research
141
REFERENCES
143
APPENDIX: EXAMPLE ITEMS AND ITEM GENERATION
151
ABOUT THE AUTHORS AND HOW TO CONTACT THEM
227
VIII
chapter one
ALTERNATIVES IN
SECOND LANGUAGE ASSESSMENT
The primary goal of this project is to provide guidelines for designing performance
assessments for second or foreign language classes at all levels of university
instruction. Through our efforts towards this goal, we hope to achieve three
fundamental objectives: (a) to provide means whereby student performance on real-
world language tasks can be validly assessed in terms of real-world criteria, (b) to
elucidate the potential for using task-based performance assessment to generalize
about students' L2 abilities, and (c) to facilitate a direct link between classroom L2
instruction and real-world language use. Given these objectives and the consequent
scope of this project (described in chapter 7), we have decided to report on our
progress and findings in a series of publications. The following chapters have been
included in this first volume:
Chapter 1 Alternatives in Second Language Assessment
Chapter 2
Performance Assessments
Chapter 3
Task-Based Language Teaching
Chapter 4
Task-Based Language Assessment
Chapter 5
Test and Item Specifications
Chapter 6
Item Prompts
Chapter 7
Conclusions
Appendix
Example Items and Item Generation
In this first chapter, we will begin by exploring the general topic of so-called
"alternative assessments" and how they contrast with our notion of alternatives in
assessment. In chapter 2, we will turn to the issues involved in performance
assessment as it is covered in general education circles as well as in the language
testing literature. In chapter 3, we will explore the literature on task-based teaching
and demonstrate its relevance for the current project. In chapter 4, we discuss the
issues in the literature that are directly related to task-based language assessment. In
chapter 5, we will provide test specifications that describe a number of variables
which contribute to task difficulty and apply that knowledge to the process of
grading the difficulty of prototypical performance assessment items in item
specifications. In chapter 6, we will provide detailed descriptions of prototype item
Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language
performance assessments. (Technical Report #18). Honolulu: University of Hawai'i, Second
Language Teaching & Curriculum Center.
0
4t,
