Background Non-communicable diseases (NCDs) are the leading health burden in all but the poorest countries, and there is an increasing interest in macro-level policy responses to tackle their upstream determinants. There is also an increasing use of qualitative methods for evaluation research, in particular when evaluating multi-component and multi-level initiatives. However, methodological opportunities offered by qualitative research, such as its strength to capture interconnections, complexities and inconsistencies, are limited by challenges, such as its small scale and in-depth rather than broad reach. We report methodological considerations, challenges and solutions.

Methods We conducted a qualitative evaluation study of seven Caribbean countries in 2015 to assess the progress made in NCD policy measures. This comprised policy document analysis, and 76 semi-structured interviews with 80 stakeholders inside and outside government. Interviews were conducted by six interviewers organised in regional teams, and analysed by an expanded team under the guidance of the authors. Data collection and analysis protocols for this relatively large-scale project were developed iteratively in workshops.

Results A first consideration was to purposively sample across settings, sectors and professional roles. This was a challenge of scale, as stakeholders were initially drawn from key informants and existing networks, and then cascaded by eliciting further recommendations to cover relevant sectors (government ministries, private, civil society), organisations within these sectors, and roles (technical, executive, elected). Some stakeholders were recommended because it was perceived as ‘politically’ important to include them, even if information elicited was less ‘rich’ in terms of relevant technical expertise or topic insight. Second, it was challenging to analyse and synthesise a large qualitative dataset across similar but distinct settings. Eleven researchers coded and categorised the data pragmatically according to the WHO NCD Action Plan, and this was guided by the Multiple Streams policy evaluation framework and realist evaluation principles to compare across contexts and themes. Later, the authors expanded the analysis to capture more inductive insights. Finally, there was an ethical challenge to secure anonymity of the participants as well as settings – e.g. not attributing policy shortcomings to individuals, organisations or even countries and their governments – while retaining relevant insights for each setting and political context.

Conclusion Qualitative policy evaluation requires careful consideration and adaptation of standard research methods, the use of clear theoretical frameworks, and transparent interrogation of limitations. Its increasing popularity and use, and its detailed description and discussion among the research community, should enable the development of robust processes.