Introduction

In recent years, researchers have been encouraged to use natural experiments to generate better evidence to fill gaps in population health science.1 ,2 Natural experimental studies help researchers capitalise on ‘events’ that occur outside of their influence (eg, policy changes, economic shocks, natural disasters) and that change the ‘mass determinants’ of health in ways that may be impossible or unethical for researchers to manipulate deliberately.3 ,4 When events occur or are administered by chance, this may allow researchers to emulate the internal validity of randomised trials.5 Although perfect natural randomisation is rare, natural experiments can still be useful for creating comparison groups that are fairly well balanced.1 Where biases exist, a range of methodological and statistical tools have been developed to reduce bias and improve the validity of inferences. Overall, there is renewed optimism that the use of natural experiments can help to unlock answers to challenging questions in population health science.2 ,6

In 2011, the UK Medical Research Council (MRC) published guidelines for producers and users of evidence summarising a broad range of analytical techniques (eg, difference in differences, regression discontinuity, propensity score analysis) that can be employed to evaluate the impact of natural experiments.1 ,7 These approaches largely address the rationale, design elements and attention to validity threats found in randomised trials, catering to certain types of research questions (eg, ‘what works’) and generating certain types of empirical answers (eg, estimates of impact estimates). Such approaches and research questions can be useful where causal chains are short and impacts are large,1 ,6 but may be less useful where complex causal pathways exist, as is often the case in population health interventions. In particular, where complexity exists, it may not be easy to conceive of clearly distinguishable ‘exposed’ and ‘unexposed’ comparison groups.8 Large-scale changes may require further examination: What does ‘exposure’ mean or consist of?; How does it change in response to naturally occurring shocks (eg, economic recessions, policy changes, etc)?; How do changes affect behaviour?; and What degree of change is required to bring about health benefits?9–12 As Diez Roux suggests, where complexity is an issue, ‘simplification can be obfuscating rather than illuminating’.13 ,14 Additional questions are required to illuminate what, how and why changes to social or environmental factors influence health.

This article focuses on one particular part of this puzzle: how can exposure to change be characterised in situations where interventions and exposures are difficult to define, or where human interaction with changing environments is multifaceted. Exposure measurement is currently an active area of innovation and discussion in observational epidemiological studies of place effects on health.10 ,11 ,15–17 Yet, although equally pertinent, discussions about how changes in exposure are conceived and measured are lacking in the evaluation literature. This paper aims to discuss different approaches for characterising exposure in the evaluation of natural experiments in situations in which the traditional use of binary treatment conditions (eg, intervention and control) may not suffice. We situate our analysis in the context of natural experiments that affect the built environment—a topical and challenging area of population health research—although we suggest that these issues are equally relevant to other areas of population health science. We hope to prompt a discussion about how exposure can be conceptualised, measured and incorporated within an evaluative framework for assessing the health impacts of natural experiments.

Conceptualising and measuring exposure in natural experimental studies

Identifying who or what is exposed to any change presents challenges for creating reliable comparison conditions.2 ,18 ,19 Such challenges are not a new consideration for those seeking to draw causal inferences about interventions from observational studies.20 But, in order to use more advanced statistical techniques, one needs an appreciation of what the intervention is, what the exposure categories (treatment conditions) are and the extent of non-compliance.5 Being specific about how populations are exposed to interventions may be difficult where the intervention, or exposure to it, is not rigidly defined. Interventions that change the built environment present further challenges: (1) defining the causal pathways that the intervention should trigger; (2) measuring exposure to an intervention; (3) understanding the variation in exposure intensity and (4) understanding how all of this results in population health impacts.

Defining exposure in practice

This section offers non-exhaustive examples of how exposure may be characterised in natural experiments where anticipated changes to the built environment are likely to influence health. The following discussion of approaches is organised in ascending order of technical sophistication and data demands. However, less sophisticated approaches may require stronger assumptions that may or may not be justifiable.

Static or hypothetical exposure measurement

Area-based definitions

In natural experiments, geographic information systems (GIS) are commonly used to characterise exposure and minimise important cultural differences by attempting to identify or create focal local comparison units: groups comparable on observed and unobserved covariates at baseline and from the same locale.28 For practical purposes, researchers often use pre-existing administrative spatial boundaries (eg, zip codes, census tracts, etc) that correspond with the availability of other routine datasets. Where an intervention is believed to affect a specific area, researchers look for reliable comparable units that are unexposed.29 These units may be matched using conventional matching techniques, or researchers may employ methods to create synthetic units.1 ,30 Where an intervention is believed to affect a specific place (ie, address or spatial point)—such as a community park or school—it may be possible to use a concentric boundary or street network buffer zone to specify an areal circumference around the environment of interest (figure 1A). This can be used to collect data on events occurring within the vicinity of the environment (eg, on crime or injuries), or to sample individuals who live near the location of interest.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Area-based spatial units. Examples of different approaches to classifying areal spatial units for analysis. (A) A concentric buffer zone around an intervention location or area (I); (B) a contiguous (ie, neighbouring) buffer zone in which a pre-existing spatial unit is classified as ‘exposed’ if an intervention is implemented within its boundaries (represented by the dark line); and (C) represents a set of bespoke cluster units which incorporate the shape of the distribution of the intervention, or pertinent features of the natural environment.

The use of area-based exposure definitions can be attractive because the analytical requirements are relatively straightforward. However, their use requires several assumptions about the relationship between exposure and the processes that might influence health (see online supplementary appendix A for an expansion on each of these points):

1. There are reasonable conceptual grounds to believe that proximity to any change in the built environment is central to defining exposure.

2. The structural change has a ‘zone of influence’ that can be defined and justified at an appropriate spatial scale (with reasonable face validity).

3. Exposure to the intervention can be treated as being dichotomous (eg, ‘exposed’ in target areas vs ‘unexposed’ outside).

Supplementary appendix

[jech-2015-206381supp_appendix.pdf]

This approach was used in an Australian study evaluating the impact of a walking and cycling trail on physical activity.31 Here, multiple buffer zones were created around access nodes to the new trail to test whether awareness and use of the new infrastructure was greater in areas close by. Area-based units have been used widely across fields, including crime prevention,32 substance misuse,33 physical activity34 and nutrition.35

Individually computed distances

For some research questions, individual measurements can be used to create more specific population exposures. For example, it may not be appropriate to define exposure by assigning all individuals to a single geographic attribute, such as home location—populations may be members of multiple geographic units, whether ‘exposed’ or ‘unexposed’. In addition, it may not be possible to identify an ‘unexposed’ comparison area. Furthermore, exposure may vary considerably within an area, either between individuals or between groups. Where these challenges exist but proximity remains a prominent feature of a programme theory, it may be possible to develop more specific distance-based measures to allow exposure to vary between individuals who occupy the same geographic areas (figure 2). These could be used to generate ordinal or continuous measures of exposure, or could be spatially modelled to create generalised exposure surfaces to help visualise heterogeneity of exposure across space.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Individually computed distance measures. The figure demonstrates the configuration of individually computed network distances for two participants (P1 and P2). Boxes P1 and P2 represent the proximity of the individual's location (home, work or other) to the intervention (I). Using this configuration, P2 would be more likely to be exposed, or would be classified as having a higher level of exposure, to the intervention than P1.

In another natural experimental study, of new walking and cycling infrastructure in the UK, the network distance from each participant's home to the nearest access point was taken as a primary measure of exposure.36 These distances were shown to be linearly associated with awareness and use of the intervention and, subsequently, with changes in overall walking, cycling and physical activity. Conceptualising exposure as an ordinal variable had considerable face and predictive validity for this particular intervention.37 This approach involves more complex analytical requirements and also makes a number of assumptions that may not be justifiable given the intervention:

1. The proximity of the home location to the intervention site (or area) is central to classifying exposure.

2. A distance-decay effect is predictive of ‘absorbed exposure’ or uptake.

3. Computed distance-based exposure measures reflect actual or perceived distances.

Individually calibrated exposure

One way of dispensing with the inherent assumptions of the approaches outlined above is by using individually calibrated measures of exposure. With additional information about participants' pre-existing routine behaviours, such as their home and work locations and their modes of transport, it may be possible to generate ‘activity’ or ‘exposure’ spaces that determine whether exposure to a particular environmental change is likely to occur.10 ,15 For example, some individuals may live close to a new urban green space, but recorded ‘activity nodes’ (ie, home and work locations and commute route) indicate that the change to the urban infrastructure is located outside their regular ‘activity space’, which therefore makes exposure less likely. Conversely, other individuals may reside far from the site of an environmental change, but their ‘activity space’ brings them near to it and increases the likelihood of exposure (figure 3).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Individually calibrated exposure measures. (A and B) The utility of using individually calibrated exposure measures. (A) The home location of two individuals (P1 and P2) and their respective work locations (W1 and W2) prior to the building of, for example, a cycle and pedestrian ‘superhighway’ (green line in (B)). Using an individually computed distance, as discussed previously, would suggest each individual is equally exposed to the new infrastructure due to the proximity of access nodes to their home locations. However, if commute distances and times are modelled incorporating the cycle superhighway and work locations, it is possible to suggest that P2's exposure to the intervention is greater due to the likely impact on their commute options. For P2, the new infrastructure could potentially reduce the duration of a cycle commute by over 7.5 min and a pedestrian commute by 25 min, at the same time having little or no direct effect on the commute options for P1.

Activity space modelling was applied in a third natural experimental study of new transport infrastructure, again in the UK.19 Researchers used each study participant's residential and work address to build a model of their quickest route to work. Journey times were calculated for various modes of travel (ie, car, public transport, cycling and walking) before and after the introduction of the new infrastructure, and changes in modelled travel times attributable to the intervention were used to create graded measures of exposure.38 This approach requires much greater technical sophistication. Key assumptions include:

1. Exposure to an intervention is not solely dependent on residential location.

2. Relevant exposure is calculated using information about exposure at, and perhaps en route between, certain key conceptually justifiable ‘anchor points’ (eg, home or work).

3. The intervention's ‘zone of influence’ can be defined and justified at an appropriate spatial scale (with reasonable margin for error).

Discussion and conclusion

Natural experiments are becoming an increasingly popular tool to help population health researchers generate better evidence where planned experiments are not possible.1 ,2 One of the key strengths of natural experimental studies is that they use exogenous events to mimic random assignment, helping to create balanced comparison groups on the basis of chance, ‘as-if’ randomised.43 This has been useful for generating unbiased estimates of causal effects in some areas of population health.1 However, there are methodological challenges for using natural experiments in population health. These may limit a study's ability to generate valid estimates of intervention effects, generalise from these estimates or provide a more nuanced understanding of how certain exposures influence health. In studies that examine changes to an environment that may deter or facilitate healthy behaviours, it may not be obvious how an intervention changes the environment, who is exposed to these changes and where any boundary of exposure is located. Such uncertainties may make it difficult to employ more advanced statistical techniques, such as those described in the recent MRC guidance on natural experiments, if the exposure has not been conceptualised in a meaningful way.1 ,7 Questions of great interest to population health scientists may remain unanswered if natural experimental studies are designed with strict adherence to the experimental framework. New methods can provide useful estimates of the magnitude of any population health effect, but explaining why this effect occurred and how it can be replicated in other contexts requires a more systematic approach to understand the processes and mechanisms interacting along the causal pathway.

This is not to say that we discourage the application of the experimental framework or question the utility of natural experiments. On the contrary, we are optimistic about the evolution of opportunistic methods and believe they have a central role for producing better evidence in population health. In this paper, we recommend a more thorough approach to the definition of exposure in the evaluation of large-scale population health interventions, particularly those involving changes to the built environment. All too often research characterises exposure on the basis of either membership of a geographic area in which some environmental variable has changed or proximity of residential location to an environment of interest, such as a new amenity. As a growing literature in observational epidemiology has shown, exposures to health-enabling or preventing environments may be multifaceted, and mobile individuals are exposed to and absorb environmental influences from many places and at different times.10 ,11 As the tools to measure diverse routine environmental exposures advance, we should not ignore the potential implications (and opportunities) these data present for the evaluation of interventions.