Insightful observations on common question evaluation methods
and best practices for data collection in survey research



Featuring contributions from leading researchers and
academicians in the field of survey research, Question
Evaluation Methods: Contributing to the Science of Data Quality
sheds light on question response error and introduces an
interdisciplinary, cross-method approach that is essential for
advancing knowledge about data quality and ensuring the credibility
of conclusions drawn from surveys and censuses. Offering a variety
of expert analyses of question evaluation methods, the book
provides recommendations and best practices for researchers working
with data in the health and social sciences.

Based on a workshop held at the National Center for Health
Statistics (NCHS), this book presents and compares various question
evaluation methods that are used in modern-day data collection and
analysis. Each section includes an introduction to a method by a
leading authority in the field, followed by responses from other
experts that outline related strengths, weaknesses, and underlying
assumptions. Topics covered include:

* Behavior coding

* Cognitive interviewing

* Item response theory

* Latent class analysis

* Split-sample experiments

* Multitrait-multimethod experiments

* Field-based data methods

A concluding discussion identifies common themes across the
presented material and their relevance to the future of survey
methods, data analysis, and the production of Federal statistics.
Together, the methods presented in this book offer researchers
various scientific approaches to evaluating survey quality to
ensure that the responses to these questions result in reliable,
high-quality data.

Question Evaluation Methods is a valuable supplement for courses
on questionnaire design, survey methods, and evaluation methods at
the upper-undergraduate and graduate levels. it also serves as a
reference for government statisticians, survey methodologists, and
researchers and practitioners who carry out survey research in the
areas of the social and health sciences.

JENNIFER MADANS, PhD, is Associate Director for Science at
the National Center for Health Statistics (NCHS). She has published
numerous papers in the areas of health status measurement, chronic
disease epidemiology, and aging.

KRISTEN MILLER, PhD, is Director of the Question Design
Research Laboratory at the NCHS. She currently oversees the
question evaluation research program and designs and implements
research projects on data quality and comparability in the health
sciences.

AARON MAITLAND, MS, is Statistician at the Question
Design Research Laboratory at the NCHS, where he is the
administrator of Q-Bank, the agency's online database of question
evaluation reports for Federal surveys.

GORDON WILLIS, PhD, is a Cognitive Psychologist at the
Applied Research Program in the Division of Cancer Control and
Population Sciences at the National Cancer Institute. Dr. Willis
has more than twenty years of experience in planning, conducting,
and reporting of all phases of cognitive laboratory research.

Featuring contributions from leading researchers and academicians in the field of survey research, Question Evaluation Methods: Contributing to the Science of Data Quality sheds light on question response error and introduces an interdisciplinary, cross-method approach that is essential for advancing knowledge about data quality and ensuring the credibility of conclusions drawn from surveys and censuses. Offering a variety of expert analyses of question evaluation methods, the book provides recommendations and best practices for researchers working with data in the health and social sciences.

Based on a workshop held at the National Center for Health Statistics (NCHS), this book presents and compares various question evaluation methods that are used in modern-day data collection and analysis. Each section includes an introduction to a method by a leading authority in the field, followed by responses from other experts that outline related strengths, weaknesses, and underlying assumptions. Topics covered include:

• Behavior coding
• Cognitive interviewing
• Item response theory
• Latent class analysis
• Split-sample experiments
• Multitrait-multimethod experiments
• Field-based data methods

A concluding discussion identifies common themes across the presented material and their relevance to the future of survey methods, data analysis, and the production of Federal statistics. Together, the methods presented in this book offer researchers various scientific approaches to evaluating survey quality to ensure that the responses to these questions result in reliable, high-quality data.

Question Evaluation Methods is a valuable supplement for courses on questionnaire design, survey methods, and evaluation methods at the upper-undergraduate and graduate levels. it also serves as a reference for government statisticians, survey methodologists, and researchers and practitioners who carry out survey research in the areas of the social and health sciences.

Contributors ix

Preface xi

1 Introduction 1

Jennifer Madans, Kristen Miller, Aaron Maitland, and Gordon
Willis

I BEHAVIOR CODING 5

2 Coding the Behavior of Interviewers and Respondents to
Evaluate Survey Questions 7

Floyd J. Fowler, Jr.

3 Response 1 to Fowler's Chapter: Coding the Behavior
of Interviewers and Respondents to Evaluate Survey Questions
23

Nora Cate Schaeffer and Jennifer Dykema

4 Response 2 to Fowler's Chapter: Coding the Behavior
of Interviewers and Respondents to Evaluate Survey Questions
41

Alisú Schoua-Glusberg

II COGNITIVE INTERVIEWING 49

5 Cognitive Interviewing 51

Kristen Miller

6 Response 1 to Miller's Chapter: Cognitive
Interviewing 77

Gordon Willis

7 Response 2 to Miller's Chapter: Cognitive
Interviewing 93

Frederick G. Conrad

III ITEM RESPONSE THEORY 103

8 Applying Item Response Theory for Questionnaire Evaluation
105

Bryce B. Reeve

9 Response 1 to Reeve's Chapter: Applying Item Response
Theory for Questionnaire Evaluation 125

Ron D. Hays

10 Response 2 to Reeve's Chapter: Applying Item
Response Theory for Questionnaire Evaluation 137

Clyde Tucker, Brian Meekins, Jennifer Edgar, and Paul P.
Biemer

IV LATENT CLASS ANALYSIS 151

11 Some Issues in the Application of Latent Class Models for
Questionnaire Design 153

Paul P. Biemer and Marcus Berzofsky

12 Response 1 to Biemer and Berzofsky's Chapter: Some
Issues in the Application of Latent Class Models for Questionnaire
Design 187

Frauke Kreuter

13 Response 2 to Biemer and Berzofsky's Chapter: Some
Issues in the Application of Latent Class Models for Questionnaire
Design 199

Janet A. Harkness and Timothy P. Johnson

V SPLIT-SAMPLE EXPERIMENTS 213

14 Experiments for Evaluating Survey Questions 215

Jon A. Krosnick

15 Response 1 to Krosnick's Chapter: Experiments for
Evaluating Survey Questions 239

Johnny Blair

16 Response 2 to Krosnick's Chapter: Experiments for
Evaluating Survey Questions 253

Theresa DeMaio and Stephanie Willson

VI MULTITRAIT-MULTIMETHOD EXPERIMENTS 263

17 Evaluating the Reliabilit…