Results and discussion

Cognitive interviewing is a psychologically oriented method for empirically studying the ways in which individuals mentally process and respond to survey questionnaires. Cognitive interviews can be conducted for the general purpose of enhancing the understand-ing of how respondents carry out the task of answer-ing survey questions. However, the technique is more commonly conducted in an applied sense, for the pur-pose of pretesting questions and determining how they should be modified, prior to survey fielding, to make them more understandable or otherwise easier to answer.

The notion that survey questions require thought on the part of respondents is not new and has long been a central premise of questionnaire design. How-ever, cognitive interviewing formalizes this process, as it approaches the survey response task from the vantage point of cognition and survey methodology (CASM), an interdisciplinary association of cognitive psychologists and survey methodologists. The cogni-tive interview is generally designed to elucidate four key cognitive processes or stages: (1) comprehension of the survey question; (2) retrieval from memory of information necessary to answer the question;

(3) decision or estimation processes, especially relat-ing to the adequacy of the answer or the potential threat it may pose due to sensitive content or demands of social desirability; and (4) the response process, in which the respondent produces an answer that satis-fies the task requirements (e.g., matching an internally generated response to one of a number of qualitative response categories on the questionnaire).

For example, answering the survey question In the past week, on how many days did you do any work for pay?requires that the respondent comprehends the key elements ‘‘week’’ and ‘‘work for pay,’’ as well as the overall intent of the item. He or she must retrieve rele-vant memories concerning working and then make a judgment concerning that response (for instance, the individual may have been home sick all week, but in keeping with the desire to express the notion that he or she is normally employed, reports usual work status).

Finally, in producing a response, the respondent will provide an answer that may or may not satisfy the requirements of the data collector (e.g., ‘‘Four’’; ‘‘Every day’’; ‘‘Yes, I worked last week’’). The cognitive model proposes that survey questions may exhibit features that preclude successful cognitive processing and that may result in survey response error (in effect, answers that are incorrect). In the preceding example, the question may contain vague elements (‘‘week’’; ‘‘work for pay’’) that create divergent interpretations across respondents;

or it may induce biased responding (e.g., the socially desirable impulse to provide a nonzero response).

Cognitive Interviewing Procedures The major objective of cognitive interviewing is to identify sources of response error across a wide range of survey questions, whether autobiographical (involv-ing behavior and events), attitudinal (involv(involv-ing opinions and attitudes), or knowledge based. To this end, a specially trained cognitive interviewer administers the questions individually to persons (often referred to as

‘‘laboratory subjects’’) who are specifically recruited for purposes of questionnaire evaluation or pretesting. In departure from the usual question-and-answer sequence within a survey interview, the cognitive interview involves procedures designed to delve into the cognitive processes that underlie the production of the answers to evaluated questions, by inducing the subject to produce verbal reports.

Two related procedures are used to elicit ver-bal reports: think aloud and verver-bal probing. The 106 Cognitive Interviewing

think-aloud procedure was adapted from psychologi-cal laboratory experiments and requires subjects to verbalize their thoughts as they answer survey ques-tions. The interviewer prompts the subject as neces-sary by providing feedback such as ‘‘Tell me what you are thinking’’ or ‘‘Keep talking.’’ The researchers then analyze the resulting verbatim verbal stream to identify problems in answering the evaluated ques-tions that have a cognitive origin. For example, the subject’s verbal protocol relating to the preceding question on work status might include a segment stat-ing, ‘‘Besides my regular job, last Saturday I, uh, did help a friend of a friend move into a new apartment—

he gave me pizza and beer—and a gift card that was lying around with a little money on it still, so I guess you could call that working for pay, but I’m not sure if that’s supposed to count.’’ Given this accounting, the investigators might surmise that the meaning of

‘‘work for pay’’ is unclear, in this case concerning irregular work activities that result in noncash remu-neration. Especially if this finding were replicated across multiple cognitive interviews, the questionnaire designer could consider revising the question to more clearly specify the types of activities to be included or excluded.

However, practitioners have observed that some subjects are unable to think aloud effectively, and that the pure think-aloud approach can be inefficient for purposes of testing survey questions. Therefore, an alternative procedure, labeled ‘‘verbal probing,’’ has increasingly come into prominence and either supple-ments or supplants think aloud. Probing puts relatively more impetus on the interviewer to shape the verbal report and involves the use of targeted probe questions that investigate specific aspects of subjects’ processing of the evaluated questions. As one common approach, immediately after the subject answers the tested ques-tion, the interviewer asks probes such as ‘‘Tell me more about that’’; and ‘‘What does the term ‘work for pay’ make you think of?’’ Probe questions are some-times designed to tap a specific cognitive process (e.g., comprehension probes assess understanding of the question and its key terms; retrieval probes assess memory processes). However, probes also lead the sub-ject to provide further elaboration and clarify whether the answer provided to the evaluated question is con-sistent with and supported by a picture gleaned through a more thorough examination of the subject’s situation.

Verbal probing can be used to search for problems, proactively, when probes are designed prior to the

interview, based on the anticipation of particular pro-blems. Or, probes may be reactive, when they are unplanned and are elicited based on some indication by the subject that he or she has some problem answering it as intended (e.g., a delay in answering or a response that seems to contradict a previous answer). The proac-tive variety of probing allows the cogniproac-tive interviewer to search for covert problems that otherwise do not sur-face as a result of the normal interchange between inter-viewer and subject. Conversely, reactive probes enable follow-up of unanticipated overt problems that emerge.

Further, the type of probing that is conducted depends fundamentally on variables such as survey administration mode. For interviewer-administered questions (telephone or in person), probes are often administered concurrently, or during the conduct of the interview, immediately after the subject has answered each tested question. For self-administered question-naires in particular, researchers sometimes make use of retrospective probes, or those administered in a debrief-ing step after the main questionnaire has been com-pleted, and that direct the subject to reflect on the questions asked earlier. Concurrent probing provides the advantage of eliciting a verbal report very close to the time the subject answers the tested questions, when relevant information is likely to remain in memory.

The retrospective approach risks the loss of such mem-ories due to the delay between answering the question and the follow-up probes. On the other hand, it more closely mirrors the nature of the presentation of the tar-geted questions during a field interview (i.e., uninter-rupted by probes) and prompts the subject to reflect over the entire questionnaire. Cognitive interviewing approaches are flexible, and researchers often rely both on concurrent and retrospective probing, depending on the nature of the evaluated questionnaire.

Analysis of Interview Results

Concerning analysis of obtained data, the focus of cog-nitive interviewing is not primarily the answers to tested questions, or quantitative data, but rather qualitative data relevant to the evaluation of tested questions. Cog-nitive interviews normally produce data in the form of written notes taken by the interviewer during the course of the interview, of notes taken by observers, or of anal-ysis of (audio or video) recordings. Such analyses sometimes depend on a coding scheme that applies a particular category of outcome to subjects’ behaviors or to interviewer comments (e.g., identification of Cognitive Interviewing 107

a ‘‘vague term’’). More often, however, data derived from cognitive interviews consist of written summaries that describe the problems observed on a question-by-question basis, across a set of interviews, and that also propose modifications intended to address these problems. On the basis of these results and suggestions, the investigators may revise the questions and then con-duct further sets, or rounds, of cognitive testing. Such iterative testing rounds are useful for determining if the proposed solutions have solved identified problems without introducing additional difficulties.

Logistics of Cognitive Interviewing Because the major emphasis of the cognitive interview is not survey data collection but rather the efficient and timely development and evaluation of survey questions in an applied setting, sample sizes for a round of cogni-tive interviews are generally small; typically between 8 and 12 subjects. In departure from the random selection procedures of the field survey, cognitive interviewing most often depends on volunteers who are recruited explicitly to represent as wide as possible a range of the population to be surveyed, primarily through the use of newspaper advertisements and posted flyers, or visits by researchers to locations where eligible individuals can be located (e.g., a clinic, service agency, school, or elderly center). Cognitive interviews are often conducted within permanent questionnaire design laboratories staffed by trained and experienced professionals and recruitment specialists, but they can also be accomplished informally by a questionnaire designer for the purpose of evaluating a single questionnaire. Within a laboratory environment, cognitive interviewing is conducted as one component of a more comprehensive pretesting process that includes additional pretesting procedures such as review by sub-ject matter experts and focus groups (which normally precede cognitive interviews), or behavior coding (which is generally conducted after cognitive interviewing rounds, as part of a survey field pretest).

Variation in Practice

Although cognitive interviewing is a common and well-established pretesting and evaluation method, the pre-cise activities that are implemented by its practitioners vary in key respects. Cognitive testing of questionnaires used in surveys of businesses and other establishments places significant emphasis on information storage and retrieval, especially because relevant information is

often retained in administrative records rather than respondent memories and is distributed among multiple sources. For any type of survey, questions that focus on sensitive information (e.g., drug use, sexual behavior, or income) tend to focus on decision processes that influ-ence the truthfulness of responses.

Practitioners also vary widely with respect to how they conduct the interviews, concerning reliance on think aloud versus verbal probing, and whether the cognitive interviews are conducted by researchers who will also serve as analysts or by an interviewing team that will present the testing results to the investi-gators for further consideration. At this time it is not clear which of these approaches are most reliable or valid, although researchers have recently begun rigor-ously to evaluate the effectiveness of cognitive inter-views in various guises.

Researchers have recently focused increasingly on cultural as well as cognitive aspects of survey ques-tions. One promising new direction, therefore, is the use of the cognitive interview to assess the cross-cultural comparability of questions, especially when they are translated from a source language into one or more target languages. As such, cognitive interview-ing procedures are extended to diverse population subgroups to determine whether these questions func-tion appropriately across group or language. Further, although cognitive interviewing has mainly been applied to survey questionnaires, practitioners have also begun to use this method to assess a wide range of other survey-relevant materials, such as advance letters to survey respondents, survey introductions used by interviewers to gain respondent cooperation, research consent forms, statistical maps and graphs, and computer Web sites (in a manner very similar to usability testing). The cognitive interview is in princi-ple applicable in any case in which researchers wish to investigate the ways in which individuals under-stand and react to orally or visually presented materi-als that demand mental processing activity.

Gordon B. Willis See also Behavior Coding; Cognitive Aspects of Survey

Methodology (CASM); Focus Group; Language Translations; Pretest; Usability Testing

Further Readings

Beatty, P. (2004). The dynamics of cognitive interviewing.

In S. Presser, J. Rothgeb, M. Couper, J. Lessler, 108 Cognitive Interviewing

E. Martin, J. Martin, et al. (Eds.), Questionnaire development evaluation and testing methods (pp. 45–66). Hoboken, NJ: Wiley.

Conrad, F., & Blair, J. (2004). Data quality in cognitive interviews: The case for verbal reports. In S. Presser et al.

(Eds.), Questionnaire development evaluation and testing methods(pp. 67–87). Hoboken, NJ: Wiley.

DeMaio, T. J., & Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in the field. In N. Schwarz & S. Sudman (Eds.), Answering questions:

Methodology for determining cognitive and communicative processes in survey research (pp. 175–195). San Francisco: Jossey-Bass.

Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215–251.

Forsyth, B. H., & Lessler, J. T. (1991). Cognitive laboratory methods: A taxonomy. In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement errors in surveys(pp. 393–418).

New York: Wiley.

Willis, G. B. (1999). Cognitive interviewing: A how-to guide.

Retrieved March 24, 2008, from http://appliedresearch .cancer.gov/areas/cognitive/interview.pdf

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design.Thousand Oaks, CA: Sage.