Improving question wording in surveys of culturally diverse populations

Abstract

PURPOSE: The purpose of this paper is to briefly describe a theoretical model articulating cognitive theory and sources of potential response bias resulting from racial or ethnic cultural experience to survey questions that deal with health behavior. The theory components are then evaluated using questions obtained from national health surveys conducted by the National Center for Health Statistics and Centers for Disease Control and Prevention. The analysis explores the effects of four cognitive tasks involved in responding to questions as specified by the model: question interpretation, information retrieval from memory, judgment formation, and response editing. Implications for epidemiological research are considered.

METHODS: Data were collected from a purposive sample of 423 adults aged 18 through 50 who were recruited to ensure equal numbers of African American, Puerto Rican, Mexican American, and non-Hispanic white respondents, stratified by age, gender, and education. Individual questions were selected for evaluation to ensure variation by topic and question format. Probes related to each of the cognitive tasks were designed to obtain insight into the underlying cognitive processes used by respondents to answer survey questions. All statistical analyses used logistic regression or ordinary least squares multiple regression as appropriate.

RESULTS: Variation by race/ethnicity was found in the way respondents defined physical activity in a series of questions used in the Centers for Disease Control and Prevention Behavioral Risk Factor Surveillance System (BRFSS). Gender and race/ethnicity appeared to influence interpretation in the absence of specific cues in the question format about how to respond. Strategies used to retrieve information from memory did not appear to be influenced by respondent culture; however, frequency of the event was associated with the recall strategy in that more frequent or regular events were more likely to result in estimates about frequency, whereas unusual or seldom occurring events were counted. Effects of race/ethnicity on judgment formation seem to be reflected in the propensity of respondents' willingness to use extreme response categories. Most effects due to race/ethnicity were found in respondent editing of answers. Race/ethnicity was found to be associated with a social desirability trait; with willingness to disclose socially undesirable behavior, particularly to interviews from racial or ethnic groups that differed from the respondent; and with the tendency to overreport socially desirable behavior.

CONCLUSIONS: Overall, the results of this research suggest several ways in which the validity of questions about risk behavior can be improved. In designing such questions, the investigator should envision the interview as a structured conversation in which ordinary conversational norms apply. Thus, questions that might request redundant information or that are threatening to the respondent need to be asked in ways that minimize these effects. Using interviewers of the same racial or ethnic group is important. Attending to the order of questions to ensure that redundant information is not requested is important. Writing questions to ensure that where response cues occur they lead the respondent to answer in unbiased ways is also important. Testing questions for potential racial or ethnic bias before using them is also important, even if the questions have been used successfully with population groups other than that or those included in a study.