Background Accurate measurement of dementia incidence and prevalence within populations is paramount for estimating disease burden and developing social interventions and health policies. Population-based studies are the main source of evidence for dementia; however, they face some methodological challenges. A fundamental complexity in dementia research is the operationalisation of dementia as an outcome. Accurate and robust measures are needed to generate valid estimates of prevalence or incidence of dementia.

Methods The Harmonized Cognitive Assessment Protocol (HCAP) is an international collaboration funded by the National Institute on Aging (NIA) that aims to measure and understand dementia risk by utilising a carefully selected cognitive and neuropsychological battery of tests to characterize cognitive function among older people. The HCAP protocol has been implemented on a sub-sample (N=1273) of the wider English Longitudinal Study of Ageing (ELSA) cohort, aged 65 and over recruited at baseline in 2018. Using evidence from fieldwork in 2022–23, we examine the impact of the methods and criteria used for data collection and coding on individual cognitive measures during fieldwork and the impact of classification of dementia (via study-based adjudication process) within a mixed-methods approach.

Results We highlight processes in the implementation of the HCAP protocol in fieldwork and the harmonisation of the HCAP methodology with other studies in the HCAP network. The study identifies areas for potential errors, variability and introduction of artefacts in the data during test administration, coding or data entry that could result in misclassification of dementia in individuals. We demonstrate how raw data from cognitive assessments can be used to address the problems and allow meaningful operational definitions of cognitive function to be created. We present solutions (i.e., training, monitoring, accurate data capture, scoring and quality control) to mitigate against any potential biases, ensuring accuracy and validity of the data.

Discussion Standardisation of fieldwork as well as reliability and accuracy of measures is critical to ensure high-quality comparable data for dementia research across studies. How data are created is seldom described in detail. This mixed-methods study demonstrates that scrutiny of fieldwork and methods in collecting data is as important as study design and appropriate analysis of data. Lack of transparency around methodological decisions may threaten the validity and reproducibility of research. This study articulates ways to reduce methodological limitations and provides the foundation of a structured framework to guide researchers on how to improve data quality and comparability of dementia research in population-based studies.