Article Text
PDF
Abstract
Background Interventions to modify school environments are effective in promoting young people’s health across outcomes, but mechanisms are poorly understood. We assessed mediation in a trial of the Learning Together intervention, building on the recent publication of results of effectiveness for reducing bullying and benefits across secondary outcomes and generally good implementation fidelity.
Methods Within a cluster-randomised trial involving 40 English schools, we examined student-reported and staff-reported school climate and student-reported involvement with delinquent peers at 24-month and 36-month follow-up, assessing the reliability of measures and whether these mediated health outcomes at a final follow-up.
Results Response rates and reliability were good for student-reported but not staff-reported measures. The intervention increased student-reported but not staff-reported-positive school climate but, like effects on student health outcomes, these manifested only at a final follow-up. The intervention reduced student-reported contact with delinquent peers at an interim follow-up. Student-reported potential mediators measured at the interim follow-up were associated with most health outcomes at the final follow-up. Adjustment for student-reported school climate and contact with delinquent peers at the interim follow-up did not reduce the associations between trial arm and our health outcomes.
Conclusion Despite being constrained by imperfect measures and by the late manifestation of impacts on student-reported school climate undermining ability to assess mediation, our study for the first time provides tentative evidence that mediation of intervention effects via improved climate and disengagement from delinquent peers is plausible. Our study provides the first evidence from a trial that whole-school interventions may work by modifying school environments and student relationships.
Trial registration number ISRCTN10751359.
- adolescent
- schools
- environment
- trial
- behaviour
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Introduction
There is increasing interest in interventions aiming to promote young people’s health by making overall school environments more health-promoting.1 A Cochrane review of ‘health promoting schools’ interventions (with environment, community and curriculum components) reported various benefits, including reducing bullying victimisation, smoking and body mass index, and increasing physical activity.2 Another review focused on interventions with environmental and not curriculum components also reported multiple health benefits.3
But how do such interventions work? A systematic review of theories of how school environments influence health4 concluded that the theory of human functioning and school organisation is the most comprehensive theory of change for such interventions.5 This postulates that for young people to choose healthier over riskier behaviours, they must possess the autonomy and ability to reason and form relationships, to make informed, healthy decisions. These capacities are facilitated by student engagement with school: good relationships with teachers; commitment to learning; and sense of belonging and participation in the school community. A refinement of the theory suggests that students lacking such school commitments may engage with delinquent peers and risky behaviours as alternative markers of belonging and identity.6 The theory also suggests that schools can increase student commitment by modifying school organisation: distributing authority between staff; promoting good staff–student relationships; integrating academic education and broader student development; and ensuring school culture reflects that of the local community.
While there is some evidence in support of this theory from observational studies of school-level determinants of student health,7 such mechanisms remain largely unexamined in intervention studies. The only study that has examined whether the health effects of whole-school interventions are mediated by student commitments to school was the Gatehouse Project. This involved a randomised controlled trial (RCT) of a whole-school intervention delivered in Australian secondary schools. Despite reporting effects on various measures of adolescent health-related risk behaviours, the study found no evidence of effects on student’ attachment to school, suggesting that attachment was not a mediator of health effects or that the measure failed to assess attachment.8
We explore these questions in relation to our own recent RCT of the Learning Together intervention.9 This whole-school intervention aimed to support schools to implement the restorative practice, staff/student action groups, and a student social and emotional skills curriculum to reduce bullying and aggression, and promote student health across various secondary outcomes. The restorative practice aims to improve relationships to prevent and/or resolve conflicts between students or between staff and students.10 It aims to prevent incidents via methods such as ‘circle-time’ (bringing staff and students together to build relationships) and/or resolve incidents via methods such as ‘conferencing’ (bringing together conflicting parties to find ways to avoid further harms). Action groups are school meetings involving diverse students, and senior and junior staff. They coordinate whole-school intervention delivery and review school rules and policies to ensure that these support whole-school change. They aim to build better relationships between the staff and students sitting on the group and signal to the wider student body that the school cares about and intends to act on the views of staff and students to build a supportive school climate.3 Social and emotional education aims to ensure that schools teach not only academic knowledge but also attend to students’ broader social development.11 Informed by the theory of human functioning and school organisation,5 these intervention components were theorised to work synergistically within Learning Together to: distribute decision-making authority across the school; strengthen relationships between and among staff and students; and integrate students’ academic learning and broader development. The intervention did not aim to improve relationships between schools and their local communities, although this might occur as a by-product. These impacts were theorised to transform the whole-school climate and improve staff–student relationships, student commitment to learning and sense of belonging and participation in the school community, thereby reducing student engagement with delinquent peers and risk behaviours, and promoting student health (figure 1).
Logic model
An RCT of the intervention reported a range of benefits including reducing bullying victimisation, use of tobacco, alcohol and drugs, and contact with the police, and promoting mental well-being, quality of life and psychological functioning. The intervention was found to be implemented with good fidelity although this was much lower for the curriculum component.12 This paper aims to examine potential mediators of such effects. We first assess the reliability of our potential mediators. These include existing student-reported measures of views of school climate and engagement with delinquent peers. We also examine a new measure of staff-reported school organisational climate to explore whether intervention effects on the school organisational environment might explain student health outcomes. Our second aim is to assess whether these measures might be mediators of intervention effects on our primary and secondary health outcomes.
Methods
Here we provide a summary of methods for the trial. For full details, including sample size calculation, see published protocol and trial report.9 12 We undertook a two-arm parallel repeat cross-sectional cluster RCT of the Learning Together intervention in 40 secondary schools eligible to take part as state schools in south-eastern England with government inspections rated as requires improvement or above, recruited by the trial team via emails. Our eligible study population consisted of all students: at baseline at the end of year 7 in 2014 (11–12 years); at interim 24-month follow-up; and final 36-month follow-up in 2017. Student data were collected using paper questionnaires in classrooms under examination conditions by trained fieldworkers blind to allocation. Using computer-generated random numbers, schools were then allocated 1:1 to intervention or control stratified by schools: single sex versus mixed sex; school-level student free-school-meal eligibility (0%–23%;>23%) indicating poverty; and General Certificate of Secondary Education results accounting for student baseline attainment (above/below the median score for England of 1000).
All staff in intervention schools received training to implement restorative practices. Around 5–10 staff per school received in-depth training to deliver restorative conferences. All schools were provided with a manual to guide the action group comprising at least six staff and six students, led by a member of the school’s senior leadership team. Groups were supported by external facilitators in the first 2 years of intervention but in the third year, the group was facilitated by school staff only. Action groups aimed to revise rules and policies so that these supported deliveries of restorative practice and coordinate implementation. Schools were provided with lesson plans and slides to guide the delivery of a social and emotional skills curriculum targeting students in years 8–10 who received 5–10 hours of teaching per year. Schools randomised to the control group continued with the normal practice.
Primary outcomes were bullying victimisation measured by the Gatehouse Bullying Scale8 and perpetration of aggression measured by the Edinburgh Study of Youth Transitions and Crime Scale13 at 36 months. Secondary outcome included use of tobacco, alcohol and drugs, mental well-being, psychological functioning and quality of life and contacts with the police and National Health Service (NHS), as described in our protocol and the main trial paper.9 12 The focus of the present paper is on three potential mediator measures, informed by the intervention logic model (figure 1).
The first focused on student reports of school climate, assessed using an established measure, the Beyond Blue School Climate Questionnaire (BBSCQ) scale, which includes 28 student-reported items arranged in four subscales covering: staff–student relationships, student sense of belonging in the school community; student commitment to learning; and student participation at school (online supplementary table S1).14 Students were asked to rate their level of agreement with items and their responses scored between 1 (complete disagreement) and 4 (complete agreement). Scores were then averaged within the subscales to obtain the subscale scores, and across all items to obtain the overall BBSCQ score. This measure aligns closely with the key theoretical constructs from the theory of human functioning and school organisation concerning staff–student relationships, student commitment to learning, sense of belonging and participation.5
Supplemental material
The second potential mediator focused on student reports of engagement with delinquent peers, assessed by the Young People’s Development Programme single-item measure asking students whether their friends who are the same age as them have been told off, stopped or picked up by the police in the last 12 months.15 Data for these two measures were collected via baseline, interim and final follow-up student surveys.
The third potential mediator focused on staff reports of their perception of school organisational climate, using a new scale (online supplementary table S1), which was assessed for the reliability at baseline and amended to include the most reliable items.16 This was a 26-item scale with subscales measuring: whether the authority is distributed among staff; staff–student relationships; integration of students’ academic education and broader development; and school–community relationships. Staff were asked to rate their level of agreement with items, with responses scored between 1 (strongly agree) and 4 (strongly disagree). Items were recoded so that a higher score indicated what, from the perspective of our theory of change, would be regarded as a less-healthy school organisational climate. Factor scores were derived within the subscales to obtain the subscale scores and across the subscale scores to obtain the overall score. Data for this measure were collected via structured telephone interviews from staff in intervention and control schools. ‘Baseline’ data were collected in September–November 2014 from one member of each school’s senior leadership team and two other members of staff identified by this individual. Interim follow-up data were then collected in September–November 2017 from one member of each school’s senior leadership team or the staff member leading the action group.
The trial was approved by the UCL (ref 5248/001) and IoE (ref. FCL 566) Research Ethics Committees. Written, informed consent was obtained at school level (head-teacher) for random allocation and intervention, and at the individual-participant level for data collection. Information sheets and consent forms for surveys were identical in intervention and control schools. Parents of students were informed about the study and could withdraw their children from research activities. The trial was prospectively registered as ISRCTN10751359 with the ISRCTN Registry on 30 January 2014.9
For potential mediators, completion was assessed in terms of the proportion of items completed by participants and the proportion of participants who completed at least half of the items in a subscale or scale. We then assessed interitem scale reliability using Cronbach’s alpha. We used this rather than ordinal alphas for ordinal scales for simplicity and because Cronbach’s alpha provides suitably conservative estimates for such measures.17
We then used a causal steps approach to assess whether our potential mediators might have mediated intervention effects on our primary and secondary outcomes.18 Four criteria needed to be met. The first, that the intervention was associated with the outcomes, was examined in our main trial analysis where, as previously stated, we observed a range of statistically significant positive effects. The second criterion required an association between the intervention and the potential mediators. We therefore assessed associations between trial arm and: our student-reported potential mediators measured at interim and final follow-ups; and our staff-reported potential mediator measured at the interim follow-up. These analyses, like those reported on our primary and secondary outcomes, used the intention-to-treat principle that is including all schools and participants in their groups as allocated. Each measure was analysed using a separate mixed model with the outcomes from each time-point treated as a repeated measure outcome. Fixed effects of the arm, follow-up time and the interaction between arm and time were specified, and the estimated baseline measures were constrained to be identical in the two arms of the trial, equivalent to adjusting for baselines. Random effects for schools and participants were specified to allow for correlations within schools and repeated measures within participants. Explicit consideration of potential confounders and how they are controlled for is an important part of statistical mediation analysis. We, therefore, report unadjusted analyses as well as analyses adjusted for baseline measures of the outcomes, sex, ethnicity, socioeconomic status (measured using the Family Affluence Scale)19 as well as for the school-level stratifying factors (single-sex versus mixed-sex school; school level deprivation; academic attainment strata). For continuous outcomes, we report unadjusted and adjusted mean differences with 95% CIs and adjusted effect sizes (standardised mean difference). For binary and ordinal outcomes, we report unadjusted and adjusted ORs.
The third criterion required that the potential mediator was associated with the outcomes. We, therefore, examined associations between our potential mediators measured at the interim follow-up, and our primary and secondary outcomes measured at the final follow-up so that we were assured of the temporality of the associations. Adjustment for potential confounders was as per previous analyses. The final criterion required that adjusting for potential mediators reduced the association between the intervention and the outcome measures. Therefore, we assessed the effect of adjusting for our potential mediators measured at the interim follow-up on the associations between trial arm and our primary and secondary outcomes measured at the final follow-up in the fully adjusted analysis. As differences between students who completed both baseline and follow-up surveys and those who only completed baseline were observed, we conducted a sensitivity analysis using multiple imputation by chained equations to impute missing data for participants with incomplete outcome data. All analyses were completed by staff blind to allocation.
Results
For trial participation and follow-up rates, participant characteristics and overall outcomes by allocation, see the main trial report.12 Student data were available from all schools. Student-reported measures of potential mediators had good response rates and reliability with over 85% of all participants completing all items and with multi-item scales and subscales having Cronbach’s alphas of >90% for the overall scale and 70% for the subscales at the baseline and interim follow-up (table 1). Staff data were available from 40 schools at the baseline and 31 schools at the interim follow-up. The staff-reported potential mediator had slightly lower item-response rates, but still >70%. Cronbach’s alpha for the overall staff-reported measure was >80% at the baseline and over 60% at the interim follow-up, with Cronbach’s alphas for the subscales being somewhat lower. There were small differences between students completing all surveys and those completing only baselines, with attrition higher among those from smaller schools or more deprived neighbourhoods, of white British ethnicity, with non-working parents or not living with two biological parents (table 2).
Mediator measures response and reliability
Differences in characteristics between students completing all surveys and those completing baseline only
The intervention had statistically significant effects on potential mediators, being associated with increased student-reported-positive school climate as well as its constituent subscales at the final but not the interim follow-up. The intervention was also significantly associated with a decrease in student-reported measure of involvement with delinquent peers at the interim follow-up but one slightly short of statistical significance at the final follow-up (table 3). The intervention had no statistically significant effects on the overall staff-reported measure of school organisational climate or its constituent subscales measured at the interim follow-up.
Intervention effects on mediators at the interim and final follow-ups
Student reports of school climate and of friends having contact with police measured at the interim follow-up were associated with primary health outcome measures as well as with secondary health outcome measures (other than age of sexual debut and use of contraception at last sexual intercourse) at the final follow-up in unadjusted and adjusted analyses (table 4). Staff reports of school organisational climate measured at the interim follow-up were not associated with any primary or secondary outcomes at the final follow-up in unadjusted or adjusted analyses, other than for one unadjusted association between school organisational climate and students’ use of NHS in the past 12 months, which disappeared on adjustment.
Effects of potential mediators at interim follow-up on outcomes at the final follow-up
Our analysis of the effect of additionally adjusting for potential mediators measured at the interim follow-up on the associations previously found between the intervention and primary and secondary health outcomes measured at the final follow-up found no evidence that this adjustment made any difference except marginally in the case of the intervention effect on well-being, where adjustment for both student-reported school climate and friends’ contact with the police removed the previously statistically marginal intervention effect at the final follow-up (table 5).
Intervention effects on student primary and secondary outcomes at the final follow-up before and after adjusting for potential mediators at the interim follow-up
The multiple imputation analysis produced results, available on request, which did not differ from the main analysis in the pattern, size or statistical significance of the associations found.
Conclusion
Summary of key findings
The student-reported measures of potential mediators had good response rates and reliability. The staff-reported measure had somewhat lower response rates and interitem reliability. The intervention appeared to impact on: student perceptions of school climate but these (like intervention effects on primary and secondary student health outcomes) did not manifest until the final follow-up; and student contact with delinquent peers at the interim follow-up. The student-reported potential mediators measured at the interim follow-up were associated with most student health outcomes. Adjustment for student-reported school climate and contact with delinquent peers at the interim follow-up did not affect associations between intervention and health outcomes.
Study limitations
The study used a well-established, multi-item measure of student views of school climate drawing on date from all students completing surveys. However, the study used a weaker, single-item measure of student involvement with delinquent peers and used a new and less reliable staff-reported measure of school organisational climate which drew on very small samples of staff (three individuals at baseline and one individual at the interim follow-up). Therefore, our ability to assess whether student contact with delinquent peers and staff-reported organisational climate acted as potential mediators of intervention effects on student health may have been limited. The RCT included the interim follow-up at 24 months and the final follow-up at 36 months. Intervention effects on primary and secondary health outcomes as well as effects on student views of school climate manifested only at the latter time-point. This meant that our ability to determine definitely whether intervention effects on school climate might mediate effects on student health outcomes was limited. The number of statistical tests might have introduced some false positive results, but we hope that being focused on prior hypotheses limited bias.
Implications for research and policy
Our study provides the first evidence from an RCT that well-implemented whole-school interventions may be effective both in promoting student health and in improving student relationships with teachers, and sense of commitment, belonging and participation at school. Despite being constrained in its ability to assess mediation by the late manifestation of impacts on student views of school, our study suggests that mediation of intervention effects via these factors is at least plausible, in that the intervention had effects on student-reported potential mediators, which were associated with student health outcomes. The lack of evidence for mediation for the staff-reported measure of school organisational climate may have reflected the poor reliability of this measure. The fact that implementation fidelity was stronger for restorative practice and action groups rather than the curriculum further suggests that any impacts on health outcomes were likely to be achieved through changes to school environment rather than through changes to individual student health-related knowledge and skills. Further research is therefore required to investigate these matters with a better measure of school organisation and longer time scales between measurements. Nonetheless, our study offers tentative evidence that whole-school interventions might work by modifying the school climate as predicted by the theory of human functioning and school organisation.5 Regarding generalisability, our trial was carried out in a representative sample of schools in and around London. Our process evaluation identified no factors that might suggest that implementation or effects would be different in other English schools.
What is already known on this subject
Young people’s health can be improved by ‘whole-school’ interventions that aim to render schools more engaging, participative and inclusive.
But previous evaluations have not examined mediators to assess how such interventions might work.
What this study adds
We found that as well as improving a wide range of health outcomes, our intervention also improved student reports of an engaging, participative and inclusive school climate, and reduced reports of students having delinquent friends. These factors were themselves associated with health outcomes.
This suggests whole-school interventions might work by engaging young people with school and reducing engagement with pro-risk peers.
Acknowledgments
Miranda Perry played a key role in developing the intervention and coordinated its implementation in the pilot and phase III trials. The authors are grateful to the staff and students of participating schools for their dedication to the intervention and completion of the outcome surveys and process evaluation surveys and interviews. They are very grateful for the advice and support of our Trial Steering Committee and Data Monitoring Committee. They acknowledge the work of and mourn the loss of Dr Farah Jamal during the trial, whose death at the age of 30 was a tragic loss for public health research.
References
Footnotes
Contributors CB and RVM: directed the trial from which the data are drawn. CB and EA: designed the analysis for this paper. CO: implemented this design and undertook the analysis. DRE: provided additional statistical expertise. CB: drafted the paper with inputs from EA, CO, DRE and RMV. EW, LB, JM, AM and JS: worked on the trial, contributed to the design of data collection instruments and contributed to the drafting of the paper.
Funding This paper presents independent research funded by the National Institute for Health Research (NIHR) in England under its Public Health Research Board (12/153/60) and the Education Endowment Foundation.
Disclaimer The views expressed in this publication are those of the authors and do not necessarily reflect those of the National Health Service (NHS), the NIHR or the Department of Health for England.
Competing interests RV is President of the Royal College of Paediatrics & Child Health.
Patient consent Obtained.
Ethics approval UCL (reference number 5248/001).
Provenance and peer review Not commissioned; externally peer reviewed.