Table Of ContentFM-Czaja.qxd 11/16/2004 4:04 PM Page i
FM-Czaja.qxd 11/16/2004 4:04 PM Page ii
To Beth and Cookie
FM-Czaja.qxd 11/16/2004 4:04 PM Page iii
FM-Czaja.qxd 11/12/2004 8:23 PM Page iv
Copyright © 2005 by Sage Publications, Inc.
All rights reserved. No part of this book may be reproduced or utilized in any form
or by any means, electronic or mechanical, including photocopying, recording, or by
any information storage and retrieval system, without permission in writing from the
publisher.
For information:
Pine Forge Press
A Sage Publications Company
2455 Teller Road
Thousand Oaks, California 91320
(805) 499–4224
E-mail: [email protected]
Sage Publications Ltd
1 Oliver’s Yard
55 City Road
London EC1Y 1SP
United Kingdom
Sage Publications India Pvt. Ltd.
B-42, Panchsheel Enclave
Post Box 4109
New Delhi 110 017 India
Library of Congress Cataloging-in-Publication Data
Printed in the United States of America
Czaja, Ronald.
Designing surveys : a guide to decisions and procedures /
Ronald Czaja, Johnny Blair.—2nd ed.
p. cm.
Includes bibliographical references and index.
ISBN 0-7619-2745-X (cloth) — ISBN 0-7619-2746-8 (pbk.)
1. Surveys. 2. Sampling (Statistics) I. Blair, Johnny. II. Title.
HA31.2.C93 2005
001.4′33—dc22
2004014521
05 06 07 08 09 10 9 8 7 6 5 4 3 2 1
Acquiring Editor: Jerry Westby
Editorial Assistant: Vonessa Vondera
Production Editor: Sanford Robinson
Copy Editor: Richard H. Adin
Typesetter: C&M Digitals (P) Ltd.
Cover Designer: Glenn Vogel
FM-Czaja.qxd 11/12/2004 8:23 PM Page v
Contents
Preface xi
Acknowledgments xiii
1. An Introduction to Surveys and to This Book 1
The Practice of Survey Research 2
The Uses of Surveys 3
Overview of the Survey Process 4
A Brief Summary of This Book 7
2. Stages of a Survey 11
Stage 1: Survey Design and Preliminary Planning 11
Making Sampling Decisions 14
Designing the Questionnaire 18
Determining Available Money and Time 19
Analyzing Data and Reporting Survey Results 19
Stage 2: Pretesting 20
Drafting the Questionnaire 20
Using Interviewers 21
Debriefing Interviewers 21
Deciding on Pretest Methods 21
Stage 3: Final Survey Design and Planning 24
Stage 4: Data Collection 24
Stage 5: Data Coding, Data-File Construction,
Analysis, and Final Report 26
Example of a Time Schedule for a Study 27
3. Selecting the Method of Data Collection 33
Evaluating the Advantages and Disadvantages
of the Four Survey Methods 33
FM-Czaja.qxd 11/12/2004 8:23 PM Page vi
Mail Surveys 36
Advantages 36
Disadvantages 38
Internet Surveys 40
Advantages 40
Disadvantages 42
Telephone Surveys 44
Advantages 45
Disadvantages 48
Face-to-Face Surveys 50
Advantages 50
Disadvantages 53
Combinations of Methods 55
4. Questionnaire Design: Writing the Questions 59
Questionnaire Design as Process 60
Factors in Questionnaire Development 63
Writing Questions 70
Criteria for Survey Questions 71
The Structure of Survey Questions 72
The Use of Qualifiers in Survey Questions 75
Response Categories 76
Identifying Weaknesses in Survey Questions 81
5. Questionnaire Design: Organizing the Questions 85
Introducing the Study 88
Telephone Introductions 90
Advance and Cover Letters 93
What Questions Should the Questionnaire Begin With? 93
Grouping Questions into Sections 96
Questionnaire Length and Respondent Burden 99
Avoiding Flaws in Mail Questionnaire Design 101
6. Questionnaire Design: Testing the Questions 103
Importance of Respondents’ Comprehension
of and Ability to Answer Questions 104
Conventional Pretests and Interviewer Debriefings 105
Postinterview Interviews 108
Behavior Coding 110
A Note on Intercoder Reliability 113
Cognitive Interviews 113
Respondent Debriefing 118
Expert Panel 118
FM-Czaja.qxd 11/12/2004 8:23 PM Page vii
Assessing Interviewer Tasks 119
Revising and Retesting: Deciding Which
Pretest Problems to Address 120
Deciding How Much Testing Is Enough 121
Pilot Tests 121
Some Last Advice 122
7. Designing the Sample 125
The Basics 125
Defining the Population 130
Constructing a Sampling Frame 133
Matching Defined Populations
and Sampling Frames 135
Recognizing Problems with Sampling Frames 137
Ineligibles 137
Inaccuracies 139
Missing Information 140
Multiple Listings 141
Determining Sample Size 142
Finite Population Correction 143
Probability Level 143
Variance 143
Confidence Interval 144
Putting It All Together 145
Formula Assumptions 146
What Most Affects Sample Size 147
Hypothesis Testing and Power 149
Using Census Data 155
Research Objectives 155
Analysis Groups 156
Consulting Census Data 156
Making Choices 157
8. Selecting a Sample 167
Example 1: A Community List-Assisted
Telephone Sample 167
Defining the Population 167
Selecting a Frame 168
Example 2: A Directory-Based Community
Telephone Sample 169
Solving for the Number of Sample Selections 171
Selecting the Sample 174
FM-Czaja.qxd 11/12/2004 8:23 PM Page viii
Example 3: Other RDD Telephone Samples 177
National RDD Sample 181
Selecting Respondents Within Households 184
Example 4: A List Sample of Students 186
Selecting a Frame 187
Deciding on a Sampling Method 187
Example 5: A Sample of University Classes 189
Defining the Population and Preparing the Frame 189
Solving for the Number of Sample Selections 189
Maintaining the Design Integrity 190
9. Reducing Sources of Error in Data Collection 193
The Origins of Error 193
Variance and Bias 194
Measures of Survey Quality 196
Unit Nonresponse 198
Recent Increases in Nonresponse 200
Item Nonresponse 200
Our Approach: Decisions and Procedures 201
Interviewer-Administered Surveys 202
Advance Notification 204
Reaching the Sampled Respondents 205
Minimizing Item Nonresponse 213
Interviewer Effects 214
Interviewer Selection 215
Interviewer Training 216
Conducting the Interview 217
Unit Response 218
Item Response 220
Response Errors 222
Quality Control 225
Self-Administered Surveys: Mail and Internet 227
E-mail Versus Web Data Collection 227
Unit Response 228
Prior Notification About the Survey 229
Followup Contacts 230
Response Patterns 231
Reasons for Nonresponse 231
Respondent Selection 232
Refusal Conversion 233
Samples of Nonrespondents 234
FM-Czaja.qxd 11/12/2004 8:23 PM Page ix
Item Nonresponse 234
Tracking 235
10. Special Topics 239
Ethical Issues in Survey Research 239
The Methodology Report 240
The Utility of the Methodology Report 241
What to Include in the Methodology Report 246
Sample Design 246
Sampling Errors 247
Frame Problems: Undercoverage 248
Instrument Development: Pretesting and
Special Methods 248
Data Collection: Interviewer Training,
Data Collection Procedures, and Quality Control 249
Response Bias: Questionnaire Problems 250
Unit Nonresponse: Response, Refusal,
and Cooperation Rates 250
Special Procedures: Samples of Nonrespondents
and Refusal Conversion 252
Nonresponse Bias: Patterns of Nonresponse
and the Use of Weights 252
Estimation 253
Item Nonresponse and Imputation 253
Data Entry and Coding: Entry Error, Open-Ends 253
Costs and Contingencies: Planning for the Unexpected 254
For Further Study: Suggested Readings 257
Appendix A: University of Maryland
Undergraduate Student Survey 261
Appendix B: Maryland Crime Survey 271
Appendix C: American Association for Public
Opinion Research Code of Professional Ethics and Practices 281
References 285
Glossary/Index 295
About the Authors 301
