Schools have long been recognised as important settings for health promotion and health education. Placing health promotion in schools has been seen as attractive for two reasons: schools present an efficient method of obtaining the attention of the majority of young people and they are the major setting in which formal education takes place. Beyond providing a site for health promotion, however, schools also form a central role in adolescents’ social lives. Health promotion and health education have paid little attention to this except in terms of identifying the need to give adolescents skills to resist peer pressure to be involved in risky behaviours. Such a focus fails to take into account the impact of school as a social institution on young people’s achievements and behaviour.

More than 40 years of educational and behavioural research has consistently shown that school organisational climate is associated with educational achievement.^1–4 Furthermore, the impact of the school environment goes beyond that of academic achievement, with research also showing associations with adolescent health and health risk behaviours.^5,6

The Health Promoting Schools framework, developed in 1995 by WHO,⁷ recognised the relation between a young person’s health and wellbeing and the quality of the social environment. This framework offers an approach to school based health promotion, emphasising the importance of multilevel community health programmes in schools.^8,9

Programmes of school change such as the Manitoba School Improvement Program (MSIP)¹⁰ and the Comer Schools Project¹¹ have moved some way to addressing the issues of whole school change and school climate. These programmes identify several steps required to implement whole school change, recognising the complexity of schools and the time taken to implement change.^12,13 While promising and well evaluated in terms of the process of implementation, these programmes have not been subjected to rigorous controlled trials. The focus and the evaluation of these programmes have only been focused on assessing the impact on engagement with learning and academic achievement. Thus, despite the attractiveness of the idea of Health Promoting Schools, there has not yet been a randomised controlled trial of a more comprehensive school based health promotion strategy.^8,14

The Gatehouse Project

The Gatehouse Project was developed to address some of the limitations in earlier school health promotion work, building on whole school change programmes. In line with the Health Promoting Schools framework, the Gatehouse Project is a primary prevention programme, which includes both institutional and individual focused components to promote the emotional and behavioural wellbeing of young people in secondary schools. The intervention was based on an understanding of risk processes for adolescent mental health and risk behaviours that derive from social environments.¹⁵ The major aims were to increase levels of emotional wellbeing and reduce rates of substance use, known to be related to emotional wellbeing.^16,17

Recognising the importance of healthy attachments, the project’s conceptual framework identified three priority areas for action: building a sense of security and trust; increasing skills and opportunities for good communication; and building a sense of positive regard through valued participation in aspects of school life.^2,17

The aim of this paper was to examine the effect on mental health and health risk behaviour outcomes of this school based preventive intervention, which began when students were in their second year of secondary school (13–14 years of age).

METHODS

Sample

A cluster randomised controlled design was used for the allocation of secondary education districts to intervention or control status. In metropolitan Melbourne, 12 districts were sampled from two education regions with probability proportional to the number of secondary schools (including government, independent, and Catholic schools) and randomly allocated as intervention or control status. Using simple random sampling, 12 schools were selected from the “intervention” districts and 12 from the “control” districts. Eight country schools were randomly drawn from four regional districts. Twenty six (12 intervention and 14 control) of the 32 schools agreed to participate (fig 1). Reasons for non-participation were refusal (n = 1), involvement in another programme (n = 2), and imminent school closure or merger (n = 3). These schools were not substantially different from those agreeing to participate in terms of disadvantage or size.

Supervised by the research team, students completed a self administered questionnaire at school using laptop computers four times: twice in 1997 at the beginning and end of the participants’ second year of secondary school (year 8) and at the end of year 9 (1998) and year 10 (1999). Absent students were surveyed at school at a later date or by telephone (6%). Telephone interviews were also completed with students who had left the project schools (4%) for the subsequent waves of data collection.

Ethics approval was obtained from the Royal Children’s Hospital Ethics in Human Research Committee, the Department of Education and Training, and the Catholic Education Office. Student participation in the survey was voluntary and written parental consent was required.

Multilevel intervention

Whole school

The key elements of the whole school component were the establishment and support of a school based adolescent health team; the identification of risk and protective factors in each school’s social and learning environment from student surveys; and, using these data, the identification of effective strategies to address these issues (see Patton et al¹⁸ and Bond et al¹⁹ for more details).

Teaching resources

Implementation of the curriculum component began in the second term of the school year and focused initially on students in the second year of secondary school (year 8, mean age 14 years).

The teaching resources were designed to allow students to explore a range of common social settings where young people might experience difficult or conflicting emotional responses and to develop and practise strategies for dealing with these. The materials were designed to be taught over a 10 week period in English, health and/or personal development classes. Resources for year 9, implemented in the following year, were designed to provide more opportunities to explore and practise key strategies for managing difficult emotions.

Role of school liaison (intervention) team and professional development

The role of the school liaison team was to provide professional development and ongoing support for the schools during the implementation of the intervention (see Bond et al¹⁹ and Butler et al²⁰ for further details). Members of the school liaison team had extensive experience working in the education system(s) and they worked intensively with two to four intervention schools each. Most of the professional development was done in schools during weekly meetings with the teams and/or specific teachers.

Measures

Four waves of student data were collected. Baseline data were collected at the beginning of year 8, and subsequent surveys were undertaken at the end of years 8, 9, and 10.

Information on implementation and process were collected throughout the study period.

Mental health status—self reported anxiety/depressive symptoms

Mental health status was evaluated using a computerised version of the clinical interview schedule—revised (CIS-R),²¹ a structured psychiatric interview for non-clinical populations. It has been used as a criterion measure for the definition of caseness in teenagers, and has an ease of reading suitable for young adolescents (Fleisch reading ease 78.5). Participants were defined as having anxiety/depressive symptoms if they scored ⩾12, reflecting a level of minor psychiatric morbidity at which a general practitioner might be concerned.²¹

Social relations—availability of attachments and conflictual relationships

Indicators of perceived availability of attachments and conflictual relationships were adapted from the interview schedule for social interaction.²² Perceived availability of attachments was assessed in terms of having someone to talk to/depend on when angry or upset and having someone who can be trusted with private feelings and thoughts. Participants were categorised as having good availability of attachments, poor availability, or absent/very poor availability. For conflictual relationships, participants were categorised as reporting no arguments, arguments with one to two people, or with three or more people.

Victimisation

Participants were classified as bullied if they answered yes to any of four items addressing types of recent victimisation: being teased, having rumours spread about them, being deliberately excluded, or experiencing physical threats or violence.

School engagement

The school engagement scale originally comprised 23 items and five sub-scales.²³ These items asked about student-student relationships (for example: I like the other students in my classes; I help out other students who need it), student-teacher relationships (My teachers are fair in dealing with students, I like my teachers this year, There are lots of chances for me to work on my own with a teacher), student-learning (Doing well in school is important to me; I try hard in school), and opportunities for participation (At my school, students have a lot of chances to help decide and plan things like school activities, events, and policies). The response set for the items is “YES!”, “yes”, “no”, “NO!”. Factor analysis indicated one factor was an appropriate summary (77% of variance explained). Excluding three items with low factor loadings (<0.02), the Cronbach α was 0.87. Summing across the 20 items a total school engagement score was created. Students were classified as having low school engagement in subsequent waves if they scored below the lowest tertile at wave 1.

Health risk behaviours—substance use

Participants rated their current smoking (non-smoker, ex smoker, occasional to heavy smoker) and drinking (non-drinker or drinker) using a standard set of questions developed by the Centre for Adolescent Health (see Patton et al²⁴ for more details). A retrospective seven day diary was completed for those who had smoked tobacco in the past month or drunk alcohol in the past two weeks. Participants were classified as smokers or drinkers if they reported smoking or drinking respectively in the past month. Regular smoking was defined as smoking on six or more days in the previous week. Regular drinking was defined as drinking on three or more days in the previous week. Binge drinking was defined as drinking five or more drinks in a row. Cannabis use was defined as any use in the previous six months. Peer substance use was reported as none or some/most of a subject’s friends were smokers, drinkers, or cannabis users respectively.

Family measures

Family measures included family structure (intact family, separated/divorced, or other circumstances), language other than English spoken at home (LOTE), country of birth (Australia v other), and whether parents drank alcohol and/or smoked cigarettes.

Implementation

The median number of lessons using Gatehouse curriculum material in the first year, reported by the school liaison team was 20 (about 15 hours) with most schools using the resources in English. One school did not teach the curriculum material in the first year of the project as they were focusing on structural changes within the school.

The intervention team provided an average of 40 hours of professional development for each school each year. For the first year, half the contact time (46%) was spent providing professional development and support for the curriculum. In the second year 50% of the contact time focused on whole school planning and 25% on curriculum.

Method of analysis

Statistical analysis was performed using Stata (version 7.0, College Station, TX). All analyses were conducted using the intention to treat principle. That is, students’ data were analysed according to their assigned intervention group whether they remained in that group or not. Comparisons were made on valid responses for each wave. Prevalence estimates and univariate and multivariate logistic regressions were performed for each outcome separately for each wave using robust “information-sandwich” estimates of standard errors to account for clustering within schools.²⁵ Multivariate analyses are presented adjusting for baseline measures of each particular outcome and for variables considered to be potentially important confounders: gender, family structure, ethnicity (Australian born and LOTE), and parental daily smoking.

RESULTS

Of the sample of 3623 students, 2678 (74%) participated in the first wave of data collection. At baseline 1335 (81%) of 1652 students on the intervention school rolls and 1342 of 1971 students (68%) from comparison schools completed the questionnaire. Attrition across the subsequent three waves of data collection was less than 3%, 8%, and 10% respectively. There was no differential response rate between intervention and control groups at the subsequent waves (98% v 96%; 92% v 92%, and 90% v 89% for the respective waves). There was minimal (<3%) missing data on most measures for wave 1. Students who were surveyed subsequent to the main survey day, irrespective of whether by computer or telephone reported somewhat higher substance use and fewer reported depressive symptoms. The differences were small and those surveyed by phone were not different to those surveyed at school subsequent to the main survey day.

Intraclass correlations in the study ranged from 0.01 to 0.06. Intraclass correlations were generally higher for school engagement and substance use than the attachment measures and depressive symptoms (0.01 depressive symptoms and 0.06 peers’ smoking at wave 1). The actual intraclass correlations were often higher and certainly more variable across the years, than anticipated. All analyses were adjusted for these effects.

Table 1 summarises the baseline measures for the intervention and control groups. The intervention group reported slightly lower levels of risk factors such as parental separation and parental smoking.

Table 2 shows the associations between the school and social relationship measures and depressive symptoms and health risk behaviours at baseline. Having arguments with many others, low school engagement, and being victimised were strongly related to substance use. Relations between poor social relationships and self reported depressive symptoms were even stronger than with the health risk behaviours.

Table 3 presents the comparison between intervention and control groups on depressive symptoms and social and school relationships for subsequent waves. Both unadjusted and adjusted odds ratios are shown. Overall, there was no significant impact of the intervention on these measures although there was some indication of a negative impact on the reporting of availability of attachments in the intervention schools.

Table 4 presents the comparison of intervention and control groups for substance use and friends’ substance use for waves 2 to 4. The table shows a comparatively consistent 3%–5% risk difference between the two groups in any drinking, any smoking and regular smoking, and friends’ alcohol and tobacco use. The largest impact as assessed by the odds ratios was the reduction in the reporting of regular smoking by those in the intervention group. There was weak to moderate evidence of an impact in friends’ use of alcohol and tobacco use. While most other odds ratios were below one (indicating a protective impact of the intervention), most indicated a small effect at best. In general, adjusted odd ratios were closer to one than the unadjusted ones, possibly reflecting the lower levels of risk factors such as parental separation and parental smoking at baseline in the intervention group (table 1).

DISCUSSION

The findings from this cluster randomised controlled trial of a comprehensive school based health promotion strategy indicate that implementation of such a strategy can be effective in reducing adolescents’ health risk behaviours particularly with respect to their substance use. With differences between the intervention and control groups of 3% to 5% for alcohol, smoking, and friends’ drug use, these findings are as strong or stronger than many reported in the drug education literature.^4,26,27 Percentage reductions (difference between control and intervention rates, divided by the control rates), which range from about 5% to 40%, are somewhat lower than those reported by others⁴ however, this measure of effect is influenced by the prevalence of behaviour.

As expected from the literature strong associations were found between depressive symptoms and participants’ social relationships and school attachment. It was however, disappointing to find the intervention had no apparent effect on adolescents’ social relationships or their reporting of depressive symptoms. These findings are consistent with other studies of preventive interventions that have failed to show an effect on depression and related problems.¹⁴ It may be that the key determinants of depressive symptoms differ from those for substance use, or alternatively that the intervention was either not sufficiently specific or sustained to produce an effect on these outcomes. The individual skills focused component was taught in one subject and with limited opportunities or time to change and influence the student-teacher relationships beyond the specific class setting. Change at the system level also takes time²⁸ and the three years of implementation may not have been sufficient to significantly have an impact on the school climate to the extent of affecting the experience of social relations and emotional problems.

This study has major strengths as an evaluation of a school based intervention. Using a randomised controlled design longitudinal data were collected on a cohort of just over 2500 secondary school students. Response rates were moderately high and attrition rates exceptionally low compared with most earlier studies of health education programmes,⁴ with few exceptions.²⁷ However, there was some differential response between intervention and comparison groups at the outset with a greater percentage of students in control schools not participating. Given that non-responders are more likely to engage in health risk and socially disruptive behaviours, this may have led to some underestimation of the intervention effect.

Although larger than many other school based trials, the comparatively small number of schools in this study may have limited the effectiveness of the randomisation process. However, the demographic profiles of the intervention and control groups at baseline were not substantially different and adjusting for these factors in the multivariate analyses had little impact on the estimates of the effects.

In the analysis of this study, due consideration has been taken for the clustered design. Intraclass correlations varied across measures and between intervention and comparison schools, effectively reducing the sample size to about 500 for some outcomes and thereby reducing the power of the study and precision of many estimates. Thus, we may be overestimating or underestimating the intervention effects. While the possibility cannot be excluded that the findings indicating some benefit from the intervention may have occurred by chance, the consistency of the reductions across the measures of substance use provides some support that this is not the case.

Using the quantitative assessment of implementation, we were not able to show a “dose response” in terms of behaviour changes. This finding would have strengthened our conclusions regarding the effect of the intervention on health risk behaviours. However, the assessment of implementation was mainly focused on the curriculum component of the intervention and could not capture the complexity of the whole school changes undertaken by the schools. Establishing measures of whole school change would be an important direction for future research in the understanding of complex interventions such as the Gatehouse Project, in complex environments.

A potential source of bias in non-blinded trials such as this one is that of social desirability. However, it is unlikely that the students were aware of being subject to an intervention, which was designed to sit within the school curriculum and administrative processes. Students did not work from specific project workbooks nor was material presented with project titles. The most prominent aspect of the project for the students was their participation in the surveys. Hence, these students may have felt they were a special group in their schools but this would be equally so for both intervention and control.

The Gatehouse approach indicates that a broader focus on students’ connectedness and school climate may be equally if not more effective than specific, single issue focused education packages, in addressing health and problem behaviours. It is clear, however, that a major limitation is the fundamental complexity of implementing a multi-focused intervention. Such an intervention requires long term commitment by schools, an understanding that such interventions are not short term, quick fix solutions, and support throughout the process.

From this study it is obvious that further research and evaluation is needed to determine more fully the mechanisms that can explain the differences we have found. The lower rates of substance use in the intervention school students appear consistent with a true intervention effect, achieved through a multilevel intervention with a focus not on drugs nor on refusal skills, but on changing social processes underlying these activities. These findings are entirely consistent with the literature on resilience, connectedness, and positive youth development.^17,29–31 Importantly, this project indicates an innovative and feasible direction for sustainable multilevel school interventions with the potential for reduction of substance use and perhaps, in the long term, reduction of other health risk behaviours.