Hostname: page-component-8448b6f56d-c47g7 Total loading time: 0 Render date: 2024-04-17T03:38:49.654Z Has data issue: false hasContentIssue false
  • English
  • Français

Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research

Published online by Cambridge University Press:  27 February 2004

GARY KING ,
CHRISTOPHER J. L. MURRAY ,
JOSHUA A. SALOMON  and
AJAY TANDON
GARY KING
Affiliation:
Harvard University
CHRISTOPHER J. L. MURRAY
Affiliation:
World Health Organization
JOSHUA A. SALOMON
Affiliation:
Harvard University
AJAY TANDON
Affiliation:
World Health Organization
Get access Rights & Permissions [Opens in a new window]

Abstract

We address two long-standing survey research problems: measuring complicated concepts, such as political freedom and efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions—with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents' assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Because the actual (but not necessarily reported) levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can alleviate this problem.

Type
CORRECTION
Information
American Political Science Review , Volume 98 , Issue 1 , February 2004 , pp. 191 - 207
Copyright
© 2004 by the American Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aldrich John H., Richard D. McKelvey. 1977 “A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections.” American Political Science Review 71 (March): 111–30.Google Scholar
Alt James, Bo Sarlvik, Ivor Crewe. 1976 “Individual Differences Scaling and Group Attitude Structures: British Party Imagery in 1974.” Quality and Quantity 10 (October): 297– 320.Google Scholar
Baum Lawrence. 1988 “Measuring Policy Change in the U.S. Supreme Court.” American Political Science Review 82 (September): 905–12.Google Scholar
Brady Henry E. 1985 “The Perils of Survey Research: Inter-Personally Incomparable Responses.” Political Methodology 11 (June): 269–90.Google Scholar
Brady Henry E. 1989 “Factor and Ideal Point Analysis for Interpersonally Incomparable Data.” Psychometrika 542 (June): 181– 202.Google Scholar
Cantril Hadley. 1965 The Pattern of Human Concerns. New Brunswick, NJ: Rutgers University Press.
Caroll J. D., J. J. Chang. 1970 “Analysis of Individual Differences in Multidimensional Scaling.” Psychometrika 35 (September): 283– 319.Google Scholar
Cheung Gordon W., Roger B. Rensvold. 2000 “Assessing Extreme and Acquiescence Response Sets in Cross-Cultural Research Using Structural Equations Modeling (with Comments).” Journal of Cross-Cultural Psychology 31 (March): 187– 212.Google Scholar
Clarkson Douglas B. 2000 “A Random Effects Individual Difference Multidimensional Scaling Model.” Computational Statistics and Data Analysis 32 (January): 337–47.Google Scholar
Clinton Joshua, Simon Jackman, and Douglas Rivers. 2002 “The Statistical Analysis of Roll Call Data.” Unpublished manuscript. Stanford University.
Gelman Andrew, and Gary King. 1994 “A Unified Method of Evaluating Electoral Systems and Redistricting Plans.” American Journal of Political Science 38 (June): 514–54.Google Scholar
Green Donald P., Alan Gerber. 2001 “Reclaiming the Experimental Tradition in Political Science.” In Political Science: State of the Discipline, III, ed. Helen Milner and Ira Katznelson. Washington, DC: APSA.
Groot Wim, Henriette Maassen van den Brink. 1999 “Job Satisfaction and Preference Drift.” Economics Letters 63 (June): 363–67.Google Scholar
Groseclose Tim, Steven D. Levitt, James Snyder. 1999 “Comparing Interest Group Scores Across Time and Chambers: Adjusted ADA Scores for the U.S. Congress.” American Political Science Review 93 (March): 33– 50.Google Scholar
Heckman James, James Snyder. 1997 “Linear Probabilty Models of the Demand for Attributes with an Empirical Application to Estimating the Preferences of Legislators.” Rand Journal of Economics 28 (Special Issue): 142–89.Google Scholar
Holland Paul W., Howard Wainer, eds. 1993 Differential Item Functioning. Hillsdale, NJ: Lawrence Erlbaum.
Johnson Timothy P. 1998 “Approaches to Equivalence in Cross Cultural and Cross-National Survey Research.” ZUMA Nachrichten Spezial 3: 1– 40.Google Scholar
Johnson Valen E., and James H. Albert. 1999 Ordinal Data Modeling. New York: Springer.
Kahneman Daniel, David Schkade, and Cass R. Sunstein. 1998 “Shared Outrage and Erratic Awards: The Psychology of Punitive Damages.” Journal of Risk and Uncertainty 16 (April): 49– 86.Google Scholar
Kinder Donald R., Thomas R. Palfrey, eds. 1993 Experimental Foundations of Political Science. Ann Arbor: University of\break Michigan Press.
King Gary. 1997 A Solution to the Ecological Inference Problem. Reconstructing Individual Behavior from Aggregate Data. Princeton, NJ: Princeton University Press.
King Gary, James Honaker, Anne Joseph, and Kenneth Scheve. 2001 “Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation.” American Political Science Review 95 (March): 49–69.Google Scholar
Lewis Jeffrey B. 2001 “Estimating Voter Preference Distri butions from Individual-Level Voting Data.” Political Analysis 9 (Summer): 275–97.Google Scholar
Linden Wim Van Der, Ronald K. Hambleton, eds. 1997 Handbook of Modern Item Response Theory. New York: Springer.
Londregan John. 2000 “Estimating Legislator's Preferred Points.” Political Analysis 8 (Winter): 21–34.Google Scholar
Martin Elizabeth A., Pamela C. Campanelli, Robert E. Fay. 1991 “An Application of Rasch Analysis to Questionnaire Design: Using Vignettes to Study the Meaning of ‘Work’ in the Current Population Survey.” The Statistician 40 (September): 265–76.Google Scholar
Mead A. 1992 “Review of the Development of Multidimensional Scaling Methods.” The Statistician 41 (April): 27– 39.Google Scholar
Palfrey Thomas R., Keith T. Poole. 1987 “The Relationship between Information, Ideology, and Voter Behavior.” American Journal of Political Science 31 (September): 511–30.Google Scholar
Piquero Alex R., Randall Macintosh. 2002 “The Validity of a Self-Reported Delinquency Scale: Comparisons across Gender, Age, Race, and Place of Residence.” Sociological Methods and Research 30 (May): 492–529.Google Scholar
Poole Keith T. 1998 “Recovering a Basic Space from a Set of Issue Scales.” American Journal of Political Science 42 (September): 954–93.Google Scholar
Poole Keith R. Steven Daniels. 1985 “Ideology, Party, and Voting in the U.S. Congress, 1959–1980.” American Political Science Review 79 (June): 373–99.Google Scholar
Poole Keith, and Howard Rosenthal. 1991 “Patterns of Congressional Voting.” American Journal of Political Science 35 (February): 228–78.Google Scholar
Przeworski Adam, and Henry Teune. 1966–67 “Equivalence in Cross-National Research.” Public Opinion Quarterly 30 (Winter): 551–68.Google Scholar
Rossi P. H., S. L. Nock, eds. 1983 Measuring Social Judgements: The Factorial Survey Approach. Beverly Hills, CA: Sage.
Sen Amartya. 2002 “Health: Perception versus Observation.” British Medical Journal 324 (April 13): 860–61.Google Scholar
Shealy R., W. Stout. 1993 “A Model-Based Standardization Approach That Separates True Bias/DIF from Group Ability Differences and Detects Test Bias/DIF as Well as Item Bias/DIF.” Psychometrika 58 (June): 159–94.Google Scholar
Sniderman Paul M., Douglas B. Grob. 1996 “Innovations in Experimental Design in Attitude Surveys.” Annual Review of Sociology 22 (August): 377–99.Google Scholar
Stewart Anita L., Anna Napoles-Springer. 2000 “Health-Related Quality of Life Assessments in Diverse Population Groups in the United States.” Medical Care 38 (September): II–102– II–124.Google Scholar
Suchman L., B. Jordan. 1990 “Interactional Troubles in Face to Face Survey Interviews (with Comments and Rejoinder).” Journal of the American Statistical Association 85 (March): 232–53.Google Scholar
Thissen David, Lynn Steinberg, and Howard Wainer. 1993 “Detection of Differential Item Functioning Using the Parameters of the Item Response Models.” In Differential Item Functioning. ed. Paul W. Holland and Howard Wainer. Hillsdale, NJ: Lawrence Erlbaum.
Torgerson Warren S. 1958 Theory and Methods of Scaling. New York: Wiley and Sons.
Wolfe Rory, and David Firth. 2002 “Modelling Subjective Use of an Ordinal Reponse Scale in a Many Period Crossover Experiment.” Applied Statistics 51 (April): 245–55.Google Scholar