Cognitive Interviews
We designed a cognitive interviewing process to elicit respondent feedback on all
individual items considered for the PROMIS item banks. We queried individuals on
the language, comprehensibility, ambiguity, and relevance of each item. Although
PROMIS benefited from beginning with items that had already been used in clinical
research, many of the extant items had not been subjected to formal cognitive interviewing.
Subjecting potential items to cognitive interviewing has become a standard technique
in the development of large-scale questionnaires, for example, by the National Center
for Health Statistics.(ref) Furthermore, through the item review process, most items’
structure and response options were revised. As such, the PROMIS investigators consensus
was that cognitive assessment with respondents could identify potentially problematic
items and response scales and help to clarify items that were not easily understood
and answered.
We based our cognitive interviewing protocol on the work of
Beatty (Beatty, 2007). The cognitive
interviewing process ascertained:
- comprehension of the question (ie, what does the respondent believe the question
is asking; what do specific words and phrases in the question mean to the respondent);
- the processes used by the respondent to retrieve relevant information from memory
(ie, what does the respondent need to recall to be able to answer the question;
what strategies does the respondent use to retrieve the information);
- decision processes, such as motivation and social desirability (ie, is the respondent
sufficiently motivated to accurately and thoughtfully answer the question; is the
respondent motivated by social desirability in answering the question); and
- response processes (ie, can the respondent match his/her response to the question’s
response options). Some of these processes may be "conscious," and others
are outside the awareness of the respondent.
The PROMIS cognitive interviews employed a "retrospective" verbal probing
technique. In this technique, a participant completes a paper and pencil version
of the questionnaire of interest. A trained interviewer then asks for other, specific
information relevant to each question, or "probes further into the basis for
the response (Willis, 2005). This type of "retrospective" probing or debriefing
is useful when a more "realistic" type of presentation of items is desirable,
particularly at later stages of questionnaire development (Willis, 2005). Additionally, this
method reduces probing from biasing patients’ responses to items later in the questionnaire.
As the final PROMIS item banks will be self–administered and most items have
been subjected to multiple research trials, a retrospective probing technique was
considered most appropriate.
The text above is an excerpt from the DeWalt et al, 2007 Medical Care manuscript.
For a full summary of the process please reference the following manuscript: DeWalt,
D.A., Rothrock, N., Yount S., Stone, A., on behalf of the PROMIS Cooperative Group.
Evaluation
of Item Candidates –The PROMIS Qualitative Item Review. Med Care 2007; 45:
S12–S21.
References
Beatty, P.C., Willis, G.B. (2007). Research Synthesis: The Practice of Cognitive Interviewing. Public Opinion Quarterly, 71(2), 287-311.
Willis, G.B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications.