Complex interventions and their implications for systematic reviews: a pragmatic approach

Abstract

Complex interventions present unique challenges for systematic reviews. Current debates tend to center around describing complexity, rather than providing guidance on what to do about it. At a series of meetings during 2009–2012, we met to review the challenges and practical steps reviewer could take to incorporate a complexity perspective into systematic reviews. Based on this, we outline a pragmatic approach to dealing with complexity, beginning, as for any review, with clearly defining the research question(s). We argue that reviews of complex interventions can themselves be simple or complex, depending on the question to be answered. In systematic reviews and evaluations of complex interventions, it will be helpful to start by identifying the sources of complexity, then mapping aspects of complexity in the intervention onto the appropriate sources of evidence (such as specific types of quantitative or qualitative study). Although we focus on systematic reviews, the general approach is also applicable to primary research that is aimed at evaluating complex interventions. Although the examples are drawn from health care, the approach may also be applied to other sectors (e.g., social policy or international development). We end by concluding that systematic reviews should follow the principle of Occam's razor: explanations should be as complex as they need to be and no more.

Introduction

What is new?

Key findings

1. Where complexity is a focus of a review, the most important first steps are to clarify the review question and whether it is really about complexity; to identify the sources of complexity in the intervention; and to identify what type of study should be sought as evidence of those aspects of complexity.

What this adds to what was known?

1. It is a challenge for systematic reviewers to produce reviews that incorporate a “complexity perspective,” where the review question and methods take account of complexity in the intervention. This article sets out an approach to thinking about sources of complexity in interventions, and how this can be mapped onto specific types of study.

What is the implication and what should change now?

1. Much academic discussion is focused on describing aspects of complexity, rather than identifying its practical implications. Although it may be useful to identify sources of complexity in an intervention's implementation and effects, it does not always follow that it is essential to adopt correspondingly complex review methods.

There is considerable interest among practitioners, policymakers, and researchers in how evidence of the effects of complex interventions can be produced and synthesized. This interest is not new; the first workshop on systematic reviews of complex interventions was organized at the 1994 Cochrane Colloquium, with a report the next year [1]; and the first detailed guidance on the design and evaluation of complex interventions to improve health was issued in 2000 [2], [3]. This interest stems partly from the need to develop further the evidence base of the effectiveness of health care and public health interventions, along with an awareness that synthesizing this evidence becomes more challenging as one moves along the spectrum from simpler toward more complex interventions. Another driver is debates about the most appropriate methods of evaluating health systems, and the recognition that it is important to know not only just whether health system interventions work but also about when, why, how, and in what circumstances such interventions work well [4], [5].

Evaluations of interventions in health care and other systems therefore tend to involve collecting a range of qualitative and other evidence to explain processes and help understand how the intervention interacts with its context. Not all these data may be scientific: Shepperd et al. [6] noted the role of nonacademic evidence such as policy documents. The challenge for systematic reviewers is therefore to produce reviews that incorporate a “complexity perspective,” where the review question and methods take account of complexity in the intervention and then identify, analyze, and integrate heterogeneous evidence to help understand its processes and outcomes. The further challenge is to do this in a way that it results in a review that is meaningful and useful to decision-makers.

However, although it is easy to describe aspects of complexity, it is less clear methodologically what one might do about it in a systematic review. The risk here is that complexity simply becomes a descripto—we are keen to describe our interventions as complex as that attract funding and publication but are less clear on the practical implications.

This article describes a pragmatic approach to dealing with complexity in systematic reviews that focus on the research question and on research users' needs. It proposes that, where complexity is a focus of a review, the essential first steps are to clarify the review question and whether it is really about complexity; to identify the sources of complexity in the intervention; and to identify what type of study should be sought as evidence of those aspects of complexity. It also notes that although it may be useful to identify sources of complexity in an intervention's implementation and effects, it does not always follow that it is essential to adopt correspondingly complex review methods. In describing this approach, we focus on the degree of complexity of our models of reality, rather than that of reality itself, as this is beyond the scope of the article.