Article Text
Statistics from Altmetric.com
Introduction
Measuring variation in implementation of public health interventions and levels of population exposure is an important part of evaluation.1–4 In the context of effectiveness trials, exposure data can be used to interpret the impact of the intervention on the study outcomes.4 In interventions where participation is voluntary or self-selected, measures of exposure can highlight inequity of access to the intervention among population subgroups of interest in public health.5
A substantial body of work exists on measuring exposure to media and other blanket health promotion campaigns.4 Less attention has been paid to measuring ‘exposure to’, or perhaps more appropriately ‘participation in’, complex public health interventions delivered through community engagement approaches. During the 1980s and 1990s, a handful of large-scale, multicomponent, community-delivered public health interventions were conducted in the USA, and evaluations of these programmes included extensive process measures to estimate the levels of exposure to and/or participation in different components of the intervention, ranging from mass media to small-group health education classes.6–8 These studies used the amount of potential exposure events or participation time offered and the actual number of exposure events or duration of participation as measures of implementation. They also attempted to weigh exposure to different intervention elements according to theoretically derived estimates of their relative effectiveness. More recently, Oakley and colleagues9 have suggested that quantitative and qualitative process evaluation of exposure to interventions should be considered an essential component of randomised controlled trials of complex public health interventions. This is because they can help, inter alia, to monitor the dose and reach of the intervention, to investigate contextual factors that influence effect, to explore the relationship between trial outcomes and variation in the extent and quality of implementation and to identify processes that might mediate the relationship between intervention and outcomes. However, high-quality process evaluation and measurement of exposure to intervention in complex public health interventions that use community engagement approaches is often difficult to achieve and difficult to resource.
In this paper, we describe the measures of intervention exposure used in the cluster randomised trial of the Well London programme, a public health intervention using community engagement and community-based projects to increase physical activity, healthy eating and mental health and well-being in 20 of the most deprived neighbourhoods in London.10 No earmarked resources to support the development of these measures and associated data collection were provided to either the research team or to those delivering the interventions on the ground. Instead, these were derived from contractually specified performance management information reported quarterly by partners and by inclusion of questions seeking information about participation in the follow-up questionnaires used to measure the main trial outcomes. The exposure measures are consequently considerably less sophisticated than those used in the US studies, where earmarked funding was available.
The main trial analysis did not provide evidence of neighbourhood-level effects on primary outcomes arising from delivery of the Well London programme,11 while the nested qualitative study indicated that active participation in the programme was key to improvements in subjective well-being.12
This paper aims (a) to describe the measures of exposure to Well London available for the trial analysis, (b) report the levels (and where possible sociodemographic correlates of) exposure and (c) examine the relationship between the trial outcomes and exposure measures. These analyses were prespecified in our trial analysis plan. In addition, we consider the limitations of our exposure measures and the importance and methodological challenges of measuring exposure in the evaluation of a community-level intervention.
Methods
Well London cluster randomised trial
Methods for the CRT design and analysis are described in detail elsewhere,10 ,11 but information relevant to this paper is summarised here. Twenty matched pairs of deprived neighbourhoods in London were randomised to receive the Well London programme or be a control area. Trial outcomes were measured before (baseline) and after (follow-up) intervention delivery by an adult household survey and a school-based adolescent survey in intervention and control neighbourhoods. In these analyses, we used data from the adult household survey and programme monitoring activities.
The intervention
Well London Phase-1 was supported by the Big Lottery Well-being Fund and delivered between October 2007 and March 2011. The aim of the Well London programme is to support the development of social networks and resilience in communities, with additional changes to the food supply and the neighbourhood environment, to address the local structural and social determinants of health13 ,14 and facilitate individual health behaviour change (nutrition and physical activity) and improve mental well-being. Delivery of the Well London programme was coordinated by the London Health Commission (at the Greater London Authority) and conducted in partnership by London Sustainability Exchange (LSx), Groundwork London, Central YMCA, Arts Council England, South London and Maudsley NHS Mental Health Trust (SLaM) and the University of East London.
Each of these partners led a key theme of the programme, delivering their specific projects in all of the 20 intervention neighbourhoods. Each organisation also took an oversight role in three to four neighbourhoods, coordinating delivery of projects by the other partners. This local delivery was organised in collaboration with a local cohosting statutory or voluntary sector organisation with knowledge about, and experience of working with, the community in that specific neighbourhood.
Well London comprised 14 interlinked projects (detailed in table 1), some of which specifically focused on the main health outcomes (physical activity, healthy eating, mental health and well-being), while others sought to improve the local environment (eg, green spaces), provide arts and cultural activities, or employment and training opportunities to residents. At the beginning of the programme, there was an extensive community consultation and engagement process in each neighbourhood (using World Cafe Appreciative Inquiry methodologies15 ,16) to identify community needs and tailor the suite of projects. The funding conditions of Well London required that specified projects be delivered in every neighbourhood addressing the health and well-being goals of the Big Lottery Well-being Fund (projects involving healthy eating/cooking, arts and culture, physical activity, healthy spaces, positive mental well-being), although the number and content of sessions of the individual projects delivered was tailored to local population needs and preferences, particularly the arts and culture, physical activity and healthy spaces projects. A central part of the intervention was the recruitment of a core group of volunteers in each neighbourhood who were trained to support other residents to participate in the Well London projects and to access health and social services and to improve health behaviours.
The number of sessions and spaces available varied depending on the format of the activities. Table 1 provides an indication of the volume of participation opportunities in each project, although each was locally tailored. Data on sessions and spaces were not reliably provided by all delivery organisations, and hence were not considered in the analyses presented here. The timing and frequency of activity on each project theme in each neighbourhood is shown in the online supplementary file 1.
Programme monitoring data
During intervention delivery, monitoring data were collected from participants in project activities and from the delivery organisations. Three types of data were collected: (a) what was delivered—delivery organisations provided summaries of the activities they had delivered; (b) how many people participated—delivery organisations counted and reported the total number of participants for each project activity or session; and (c) who participated—at the end of each project session, participants were invited to complete a brief evaluation form requesting age, postcode, gender, number of activity sessions attended within the project and the number of other Well London projects in which they had participated. These data were collated for each 3-month period of programme delivery.
Adult household survey data
Two separate cross-sectional surveys were conducted with no follow-up of individuals between baseline and the end of the intervention. Households were selected at random from the Post Office Address File for each neighbourhood11 and interviews requested from all adults (≥16 years) during fieldwork visits. Written informed consent was obtained from all participants. The domains covered in the questionnaire are presented in the online supplementary file 2, and the questions used to capture participation in Well London in the intervention and control neighbourhoods are shown in the online supplementary file 3. A copy of the questionnaire is available from the authors upon request.
Trial outcomes
The primary outcomes were eating at least five portions of fruit/vegetables per day, doing at least five sessions of 30 min of moderate intensity physical activity per week (or equivalent), a normal score on the GHQ-12 and the Warwick Edinburgh Mental Well-being Scale score. A range of secondary health and social outcomes were also examined (see online supplementary files 4 and 5).
Measures of exposure to Well London
Personal recall of participation
Individual-level participation of respondents in the household adult survey was represented as a binary indicator (0=no participation at any time during the 3.5-year programme; 1=individual had reported participating in any activities.) This was used in the individual-level regression analyses examining the association between self-reported participation and the trial outcomes.
The prevalence of self-reporting any participation in Well London in the household adult survey was used as a neighbourhood-level measure of participation.
Intensity of intervention delivery
Two measures of intervention delivery intensity were created:
-
Project contacts: The number of project contacts per 1000 neighbourhood residents, estimated from the monitoring data as described in the online supplementary file 6. The measure constructed from the monitoring data is not a population rate of participation because residents could take part in more than one project during a reporting period and across the lifetime of the programme and no unique identifiers were available for de-duplication. These estimates were also not adjusted for the length and capacity of specific projects because these data were not reliably reported by delivery organisations.
-
Project quarters: The number of calendar quarters (3-month periods) that projects were offered (from the programme monitoring data). The Well London programme comprised 13 separate projects and delivered over a 42-month period (14 quarters), so the total possible number of project quarters delivered was 182, although each project was not delivered in every month of the programme. The delivery of each project across the intervention neighbourhoods is summarised in the online supplementary file 1.
The majority of projects required active, identifiable involvement from participants, for example, exercise or cooking classes. Residents were passively exposed to a few intervention components, such as changes to the green space and local environment, or changes to the commercial provision of fruit and vegetables or other aspects of the local food environment. These environmental aspects of the intervention programme, and the small number of larger scale community events, are not captured in the project contact or survey-based estimates of participation.
Data analysis
Prevalence of self-reported participation in Well London across all intervention neighbourhoods was calculated overall and for different sociodemographic subgroups. Regression models were fitted to each of the individual-level trial outcomes, separately examining their association with the individual-level self-reported participation and each of the neighbourhood-level measures of exposure. All hypothesis tests and CIs were based on robust SEs to account for clustering. All analyses were conducted using complete cases, defined as those survey respondents with data for all sociodemographic variables used for adjustment and for the outcome of interest. Hypothesis tests were considered significant at p <0.01 to account for multiple tests.
Results
Survey response rate
The response rate for the adult household survey calculated at the household level was 26% and the household refusal rate was 51%.11
Exposure to the Well London intervention
The prevalence of participation (in any Well London activity) reported by household survey respondents was 3.1% (SE 0.7%) and was higher among women than men (table 2). The mean number of sessions attended across all projects was also higher among women. Self-reported participation increased with age and was highest among men and women aged 55–64 years. The greatest gender differential in participation was at age 35–54 years, with men participating substantially less than women. ‘Other Asians’ were the highest participating male group, while Black Caribbean women participated most often. Those with higher levels of educational qualifications were more likely to report participation, although the prevalence of participation was substantially higher among women with university degrees than men with the same qualifications. Self-reported participation was also higher among those who were unable to work/not seeking employment compared with those who were working or seeking work, and was higher among individuals with English as their first language. Employed women were much more likely than employed men to participate, but the pattern was reversed for men and women who were unemployed and seeking work.
There was variation between intervention neighbourhoods in project contact events per 1000 residents (data not shown), which ranged from 143 to 1204 per 1000 (mean 436, SE 58) and in the number of project quarters of delivery, ranging from 21 to 47 (mean 32, maximum possible 182). Spearman's rank correlation showed that project contact events and project quarters of delivery were correlated (ρ=0.5, p=0.03), but that neither was correlated with the survey participation rate (data not shown).
Associations between measures of exposure and trial outcomes
A matrix of associations of individual and neighbourhood-level measures of exposure with trial health and well-being outcomes is shown in table 3 and with trial social outcomes in table 4. Full results are provided in the online supplementary file 7.
Individual-level measures of exposure
After adjustment for sociodemographic characteristics and baseline measures, there were no associations between individual-level participation and the primary trial health outcomes (table 3, column 1), but self-reported Well London participants were more likely to report volunteering and taking action to improve their local neighbourhood in the last 12 months than non-participants (table 4, column 1).
Neighbourhood-level measures of exposure
After adjustment for sociodemographic characteristics and baseline measures, there were significant (at p <0.01) associations between exposure measures and some outcomes, although these were not consistent across the different exposure measures. Survey respondents living in areas where there was a higher prevalence of self-report participation from the survey (table 3, column 2) reported eating more portions of fruit and vegetables and were more likely to report meeting government guidelines on physical activity for obesity prevention (at least seven sessions of 60 min of moderate intensity physical activity per week or equivalent). They also reported (table 4, column 2) higher levels of social support and were more likely to report that they had taken action to solve a problem in their area in the past 12 months. After the same adjustments, survey respondents in areas with higher project headcounts reported (table 3, column 3) higher levels of positive mental well-being (Warwick Edinburgh scale). They reported (table 4, column 3) greater social connectedness (social network score) and were also more likely to report that people in their neighbourhood ‘pull together to improve it’, that individuals in their neighbourhoods ‘do things to help each other’ and that residents from different backgrounds in their neighbourhood ‘get on’. Finally, after adjustment, survey respondents living in areas with more project quarters of intervention delivery were more likely to report (table 4, column 4) that racial harassment was a problem in their area and to report higher levels of incivilities and social disorder.
Discussion
In this paper, we have described one individual-level measure of ‘exposure’ to the Well London programme: self-report participation at the time of trial outcome assessment survey; and three neighbourhood-level measures of exposure: neighbourhood prevalence of self-report participation (from survey); and project-contact event rate and the amount of project time delivered during the 3.5-year programme, calculated from programme monitoring data. The results from the adult household survey conducted at follow-up show a modest level of self-reported exposure and that individuals with lower levels of education, men who were in work or aged 35–54 years and women who were unemployed and seeking work, were less likely to report having taken part in the Well London programme. This indicates potential inequity of access for those who might benefit most from the intervention. At the neighbourhood level, there was correlation between the two programme monitoring-derived measures, but neither was correlated with the survey-derived neighbourhood prevalence of self-report participation.
While it is commonly reported that individuals in lower socioeconomic groups are less likely to engage with and/or benefit from health improvement interventions,17–20 it is not possible to distinguish between likelihood of participation and likelihood of reporting participation. Problems with recall may also mean that the self-report prevalence of participation from the survey under-represents the levels of participation in the programme. There were 14 different projects in which residents could participate, which were run at different times during the 3.5-year programme (see online supplementary file 2); at the time of interview it may have been several years since a survey respondent took part in a Well London healthy cooking class or went to a community festival or football tournament. Brand recognition may also have been poor in some areas, despite efforts in the survey questionnaire to provide examples of Well London activities and branding, so that respondents may have taken part in an activity but not associated it with the Well London programme.
Turning to the association between exposure measures and outcomes, at the individual level, self-report participation was only associated with volunteering and taking local action, both of which could be subject to reverse causality, because individuals who already do these things might be more likely to participate in the intervention and/or to recall doing so. While there were significant and strong associations between the neighbourhood-level/exposure measures and some secondary health and social outcomes, there was no consistent trend of association across the three participation measures or across related outcomes. Although the finding of such associations is consistent with the findings of the qualitative study12 that active participation in the programme was key to improvements in well-being, these results should be viewed with caution as they could clearly result from ecological bias, reverse causality or both, especially given that there were no effects on any of these outcomes in the main trial analysis.
There are significant limitations to all of the exposure measures that we have presented here. These mirror the challenges experienced in previous efforts to monitor and evaluate community health initiatives that used a similar philosophy and approach to Well London.6–8 When working to engage citizens in such deprived and disenfranchised communities, it might be counterproductive to ask participants repeatedly to complete forms and provide personal information to uniquely identify them. However, this information is needed to track their contact with the programme and their pathway through it to estimate prevalence of participation. Residents in these communities are often unwilling to give this information because of poor relationships with state or authoritative institutions, and to ask for it would risk their disengagement from the programme where the major aim is to reduce such alienation. In relation to the results presented here, we are also not able to say how comprehensively and consistently the anonymous programme monitoring questionnaires were administered by the project delivery staff, across different projects and across different intervention sites. It is therefore possible that the large variability in project time delivered and process evaluation participation prevalence could be due to variability in data collection, not the underlying phenomena of delivery and engagement we assume these data represent.
In multilevel interventions that combine components conceptualised as traditional behaviour change activities (where ‘exposure’ is clearly observable and quantifiable) with activities that attempt to address the more indirect but perhaps more fundamentally important influences on health and well-being such as social norms, social support and the health architecture of the local environment (food options, urban layout), there is a bias towards measuring exposure to the individual-level activities. Even where previous studies have attempted to weigh the relative impact of intervention components acting at these different levels to provide a universal measure of programme exposure,6 these weights are somewhat arbitrary and may not be grounded in any substantial health-related theory. Other studies have used more qualitative measures to represent the degree of ‘community activation’, but these do not lend themselves to the types of exposure analyses we have attempted here.7
The qualitative study embedded in this CRT12 has shown that interviewees were sometimes unsure whether they had taken part in Well London, identifying themselves initially as participants or non-participants but then revealing information during in-depth interviews that suggested the opposite. This suggests that ‘participation’ in a multicomponent, community-based intervention is exceptionally difficult to define, and therefore to measure, because it is differently interpreted by the research team, the intervention delivery staff and members of the community.
Some might suggest that attempting to measure the ‘dose’ of community engagement that individuals or communities receive completely misses the point of this approach,21 or at least that the way that individuals engage with such programmes is far less mechanistic than the way we have conceptualised participation in the measures presented here. Simple volume of contact, even when weighed for different levels of educational intensity,6 does not capture the complex and multifaceted processes by which the behaviour and self-concept of individuals may be affected and changed by such socially oriented health interventions.
Despite these criticisms, it is important to investigate levels of participation in some way, to highlight inequities of access, and whether the data from the self-report survey are taken at face value, they certainly warn us not to see deprived communities as homogeneous and point to the need to be vigilant so that the most disadvantaged and socially excluded individuals within them are not overlooked. In future studies it may be beneficial to explore the pathways that people take through complex multicomponent interventions such as Well London to help examine the ‘active ingredients’ of the programme for different individuals and different outcomes, rather than trying to capture all contact events. While there will still be limitations to these approaches (eg, Hawthorne effect, recruitment bias), recruiting a cohort of residents in intervention areas to be followed through the intervention, whether they participate or not, and/or recruiting a cohort of engaged participants after programme initiation are two possibilities. There is a need to experiment with using new mixed methods and participatory approaches to understand in more detail the ways in which individuals engage with community programmes and the way that this engagement may, or may not, impact on health and social outcomes that are the focus of public health and public policy. However, funders of interventions and evaluations of these must become more realistic about the resources that will be needed to achieve this.
What is already known on this subject
-
It has been suggested that process measures, including measures of exposure, should be considered as essential components of trials of complex public health interventions. This is because they can help to monitor dose and reach of the intervention, investigate contextual factors that influence effect, explore the relationship between trial outcomes and variation in the extent and quality of the implementation of the intervention and identify processes that might mediate the relationship between intervention and outcomes. A substantial body of work exists on measuring exposure to social marketing and other blanket media health promotion campaigns, which focus on survey recall. Less attention has been paid to measuring ‘exposure’ to complex public health interventions delivered through community engagement approaches. Previous studies of community interventions have looked at the level of provision and actual contact with the programme using management systems during delivery. However, these interventions have focused on educational and screening activities in community healthcare settings.
What this study adds
-
This study describes survey and programme management data-based approaches to measuring exposure to the Well London health improvement programme. We found sociodemographic variation in survey-based exposure measures. We found correlation between our two management data-derived neighbourhood-level measures of exposure but no correlation between these and survey-derived measures. We found some significant associations between the neighbourhood-level exposure measures and some outcomes that might suggest intervention effects not detected in the main trial but were not consistent across measures or related outcomes that might suggest reverse-causation or ecological bias . We discuss some of the implications of these findings for collecting, using and interpreting both types of data, with particular reference to community engagement in deprived urban areas.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
- Data supplement 2 - Online supplement
- Data supplement 3 - Online supplement
- Data supplement 4 - Online supplement
- Data supplement 5 - Online supplement
- Data supplement 6 - Online supplement
- Data supplement 7 - Online supplement
Footnotes
-
Contributors The study was designed by AR, RH, AD, RL, MP, DM, GP and CB. RH designed the analysis strategy. GP and GY carried out statistical analysis. GP drafted the manuscript. AR, MP and RH made critical edits and revisions to the manuscript. All authors read and approved the final version of the manuscript. PT, ES, SL and KL carried out the fieldwork for the surveys.
-
Funding This work was supported by a grant from the Wellcome Trust (Grant Number 083679), which paid the salary GY and GP. The Greater London Authority funded the fieldwork for the surveys in the intervention sites. The University of East London provided core funding for the IHHD. The views expressed in this paper are those of the authors and not necessarily those of any funding body or others whose support is acknowledged. The funders had no role in study design, data analysis, decision to publish or preparation of the manuscript.
-
Competing interests IHHD was funded by Big Lottery to deliver the Community Engagement component of the Well London Intervention.
-
Ethics approval Research Ethics Committees of University of East London, London School of Hygiene and Tropical Medicine and University of Westminster.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement All datasets will be available through the UK Data Archive.