Statistics Sunday: What Are Cognitive Interviews?

In a recent Statistical Sins post, I talked about writing surveys and briefly mentioned the concept of cognitive interviews. Today, I wanted to talk a little more about what they are and how they can be used to improve surveys and measures.

The purpose of a cognitive interview is to get into the mind of the individual, to understand their thought process. When conducting cognitive interviews for a survey or measure, your goal is to understand the respondents' thought process while completing the instrument.

There are two different ways you can conduct these interviews. The first is the think aloud technique. As the respondent completes the measure, you ask him or her to narrate the thought process. As I said in the statistical sins post, when people encounter a measure, they read an item and then determine their response. They then compare their internal response to the options given, and in essence translate their personal response to a supplied answer.

The problem I've encountered is that people can't always verbally narrate what they're thinking. Thought isn't always in words and sentences, and to communicate those thoughts, one must first translate nonverbal information into verbal information. Even when I want to do a think aloud cognitive interview, it will probably use elements of the second approach, direct probing.

For this approach, the researcher asks the respondent questions while he or she is looking at your instrument - specific questions that get at how they're approaching the instrument, and ways it could be improved. For instance, you might ask respondents what they think about when they hear a particular word or phrase that shows up in your instrument. This can help you identify potential misunderstandings or other words/phrases that would be more clear. You might ask if they like the response options and whether any options are missing. You could even ask if important items or questions are missing.

It is possible to do a mixture of the two. For example, I may have a person go through a measure while narrating their thoughts. I'll usually have a general question to help people keep narrating as they complete the measure - basically questions I use if they lapse into silence. Then, I'll have more targeted questions to help get specific responses to issues I care most about. Anytime I do interviews, I like to go from general to more specific topics, so this approach to cognitive interviews works well for me.

This approach should happen further along in the development of your survey or measure. If you're stil trying to figure out what should go on your instrument, you'd be best to use different methods like literature reviews, convening an expert committee, or focus groups. But once you get father along in your process, cognitive interviews are an important way to make sure you get good data from the instrument you've exerted a lot of effort to create.