Data Collection

Data Collection is the second of six books in the Measurement and
Evaluation Series from Pfeiffer. The proven ROI
Methodology--developed by the ROI Institute--provides a practical
system for evaluation planning, data collection, data analysis, and
reporting. All six books in the series offer the latest tools, most
current research, and practical advice for measuring ROI in a
variety of settings.

Data Collection offers an effective process for collecting data
that is essential to the implementation of the ROI Methodology. The
authors outline the techniques, processes, and critical issues
involved in successful data collection. The book examines the
various methods of data collection, including questionnaires,
interviews, focus groups, observation, action plans, performance
contracts, and monitoring records. Written for evaluators,
facilitators, analysts, designers, coordinators, and managers, Data
Collection is a valuable guide for collecting data that are
adequate in quantity and quality to produce a complete and credible
analysis.

Patricia Pulliam Phillips is an internationally recognized
author, consultant, and president and CEO of the ROI Institute,
Inc. Phillips provides consulting services to organizations
worldwide. She helps organizations build capacity in the ROI
Methodology by facilitating the ROI certification process and
teaching the ROI Methodology through workshops and graduate-level
courses.

Cathy A. Stawarski is program manager of the Strategic
Performance Improvement and Evaluation program at the Human
Resources Research Organization (HumRRO) in Alexandria, Virginia.
She has more than twenty-five years of experience in research,
training and development, and program evaluation. Throughout her
nearly twenty years at HumRRO, she has worked primarily with
clients in the federal sector. Her work includes leading and
conducting the evaluation of leadership and human capital
initiatives as well as assisting organizations in developing
comprehensive evaluation strategies.

The ROI Institute, Inc., is a benchmarking, research, and
information sharing organization that provides consulting services,
workshops, and certification in the ROI Methodology. Widely
considered the leading authority on evaluation and measurement of
learning and development in organizations, the ROI Institute
conducts workshops and offers certification for thousands of
practitioners through a variety of strategic partners.

1. Using Questionnaires and Surveys.

Types of Questions.

Questionnaire Design Steps.

Pager: Please do not italicize the Contents H1 items or the sublists to this level.

Determine the Specific Information Needed.

Involve Stakeholders in the Process.

Select the Types of Questions.

Develop the Questions.

Check the Reading Level.

Test the Questions.

Address the Anonymity Issue.

Design for Ease of Tabulation and Analysis.

Develop the Completed Questionnaire and Prepare a Data Summary.

Improving the Response Rate for Questionnaires and Surveys.

Provide Advance Communication.

Communicate the Purpose.

Describe the Data Integration Process.

Keep the Questionnaire as Simple as Possible.

Simplify the Response Process.

Use Local Manager Support.

Let the Participants Know That They Are Part of a Sample.

Consider Incentives.

Have an Executive Sign the Introductory Letter.

Use Follow-Up Reminders.

Send a Copy of the Results to the Participants.

Review the Questionnaire with Participants.

Consider a Captive Audience.

Communicate the Timing of Data Flow.

Select the Appropriate Media.

Consider Anonymous or Confidential Input.

Pilot Test the Questionnaire.

Explain How Long Completing the Questionnaire Will Take.

Personalize the Process.

Provide an Update.

Final Thoughts.

2. Using Tests.

Types of Tests.

Norm-Referenced Tests.

Criterion-Referenced Tests.

Performance Tests.

Simulations.

Electromechanical Simulation.

Task Simulation.

Business Games.

In-Basket Simulation.

Case Study.

Role-Playing.

Informal Tests.

Exercises, Problems, or Activities.

Self-Assessment.

Facilitator Assessment.

Final Thoughts.

3. Using Interviews, Focus Groups, and Observation.

Interviews.

Types of Interviews.

Interview Guidelines.

Pager: Please style the following as a sublist to the previous list.

Develop the Questions to Be Asked.

Test the Interview.

Prepare the Interviewers.

Provide Clear Instructions to the Participants.

Schedule the Interviews.

Pager: end of sublist.

Focus Groups.

Applications of Focus Groups.

Guidelines.

Pager: Please style the following as a sublist to the previous list.

Plan Topics, Questions, and Strategy Carefully.

Keep the Group Size Small.

Use a Representative Sample.

Use Experienced Facilitators.

Pager: end of sublist.

Observations.

Guidelines for Effective Observation.

Pager: Please style the following as a sublist to the previous list.

Observations Should Be Systematic.

Observers Should Be Knowledgeable.

The Observer's Influence Should Be Minimized.

Observers Should Be Selected Carefully.

Observers Must Be Fully Prepared.

Pager: end of sublist.

Observation Methods.

Pager: Please style the following as a sublist to the previous list.

Behavior Checklist.

Delayed Report.

Video Recording.

Audio Monitoring.

Computer Monitoring.

Pager: end of sublist.

Final Thoughts.

4. Using Other Data Collection Methods.

Business Performance Monitoring.

Using Current Measures.

Pager: Please style the following as a sublist to the previous list.

Identify Appropriate Measures.

Convert Current Measures to Usable Ones.

Pager: end of su...