Article Text

Download PDFPDF

Is there a “school effect” on pupil outcomes? A review of multilevel studies
  1. E Sellström1,
  2. S Bremberg2
  1. 1Department of Health Sciences, MidSweden University, Östersund, Sweden
  2. 2Department of Public Health Sciences, Karolinska Institute, Stockholm, and National Public Health Institute, Stockholm, Sweden
  1. Correspondence to:
 Dr E Sellström
 Midsweden University, Department of Health Sciences, SE-831 25 Östersund, Sweden; eva.sellstrom{at}miun.se

Abstract

Study objective: The school environment is of importance for child outcomes. Multilevel analyses can separate determinants operating at an individual level from those operating at a contextual level. This paper aims to systematically review multilevel studies of school contextual effects on pupil outcomes.

Design: Key word searching of five databases yielded 17 cross sectional or longitudinal studies meeting the inclusion criteria. Results are summarised with reference to type of school contextual determinant.

Main results: Four main school effects on pupil outcomes were identified. Having a health policy or antismoking policy, a good school climate, high average socioeconomic status, and urban location had a positive effect on pupil outcomes. Outcomes under study were smoking habits, wellbeing, problem behaviour, and school achievement.

Conclusions: Despite the different pupil outcomes and the variety of determinants used in the included papers, a school effect was evident. However, to improve our understanding of school effects, presentations of results from multilevel studies need to be standardised. Intraclass correlation and explained between school variance give relevant information on factors in the school environment influencing pupil outcomes, and should be included in all multilevel studies. Inclusion of pupil level predictors in the multilevel models should be based on theoretical considerations of how schools and communities are interconnected and how pupils and their families are influenced by school contextual factors.

  • SES, socioeconomic status
  • ICC, intraclass correlation
  • school effect
  • child
  • pupil
  • outcome studies
  • multilevel studies

Statistics from Altmetric.com

Children and adolescents spend a considerable amount of their time in school, and the school environment is therefore of importance for child outcomes. Research within the framework of “effective schools” has established that factors in the school environment play a part in pupil achievement.1 In the 1970s, Rutter et al showed that pupils demonstrate greater school achievement and social adaptation in schools characterised by strong educational leadership, high expectations, and frequent evaluation by teachers.2 These findings were later confirmed in other studies.3,4 Furthermore, earlier reviews show that 10% of variation in pupils’ achievements can be explained with reference to the school they have attended.1,5

School environment also has an impact on child health and wellbeing. Three recent reviews conclude that pupils’ problem behaviours, alcohol and drug use, and crime are influenced by the school environment.6–8 Wilson et al showed that interventions focusing on school context, rather than on individual pupils, were effective in preventing problem behaviours.8 Aveyard et al claim that the most effective methods to deter smoking are bans and enforcement.6 Furthermore, Evans-Whipp et al found that more comprehensive and strictly enforced school policies are associated with less smoking.7

The association between school characteristics and child outcomes has been established mainly in ecological or individual based studies. Such study designs could, however, contain serious sources of error as they do not take into consideration the nested structure of the data.9–16 The fact that schools are situated in different neighbourhoods and the pupils come from different socioeconomic backgrounds could explain variations in their school achievement and health and wellbeing. It is a well established fact that children in families with high socioeconomic status (SES) do better in school and have fewer health related problems.17 Family SES therefore influences child outcomes, and the increased risk is not necessarily connected to the school but may instead be connected to the family.

In the study of variation in child outcomes between schools, multilevel technique is a useful method as it makes it possible to separate out school effects from family influences.18,19 In other words, multilevel analysis can establish how much of the variation in child outcomes is conditioned by individual circumstances and how much is related to differences between schools (that is, intraclass correlation). It is also possible to establish how much of the variation in child outcomes can be explained by school level factors.

An important theoretical issue is how determinants on different levels are interlinked. This includes considerations of what contextual determinants are used and how theories have been operationalised. It also raises the question of whether relevant individual level predictors are included in the statistical models and whether they should be regarded as confounders or as mediators.

The objective of this literature review is to clarify the impact of school context on any child outcomes, independently of pupil composition. The review uses systematic methodology and includes only studies that used multilevel technique. The articles were reviewed to gain information on the following questions:

  1. What is the evidence that school level factors explain between school variation in pupil outcomes (that is, explained between school variance)?

  2. How much of the variation in pupil outcomes is conditioned by differences between schools (that is, intraclass correlation)?

  3. What theoretical frameworks have been suggested to explain between school variation in pupil outcomes?

METHOD

Data sources and study selection

Literature was identified through searches from August to October 2003 in the Medline, ERIC, PsycInfo, Sociological Abstracts, and Social Citation Index databases. Search words were “multilevel” and “school” (and “environment” or “community” or “ecology*” or “context*”). The search was limited to studies of children under 18 years of age. The only studies included were those performed in high income countries (Western Europe, USA, Canada, and Australia) and where the second level units consisted of schools. In all, 411 articles were initially found, of which 17 met the selection criteria. These articles are presented in table 1 according to the outcome under study.20–36 We included studies on any child outcomes as the focus was to review effects of school environment regardless of the outcome under study. The outcomes in these studies included smoking habits and alcohol use, problem behaviours, wellbeing, school achievement, and physical achievement. Table 1 shows school effects (significant estimates from multilevel models where individual level variables were controlled), intraclass correlation coefficients (ICCs) and estimates of explained variance.

Table 1

 Multilevel studies reporting school effects on pupil outcomes

All included studies had an observational, longitudinal, or cross sectional design, and study quality was assessed based on study design, sampling technique, and inclusion of relevant pupil level variables (table 2).

Table 2

 Validity assessment of included studies

Data analyses

This review includes only studies using multilevel techniques, in which data are hierarchically structured. The basis for this choice is the assumption that pupils attending the same school are in some respects more alike than pupils from two different schools. Using multilevel approach permits identifying variability in the outcome on two levels (that is, pupil and school level). In this review the outcomes consist of continuous or binary data and, accordingly, the multilevel models are either linear or logistic. When the outcome is continuous, a random intercept model can be described with the following equations19:

Embedded Image

where Yij is the value of the outcome of the ith pupil in the jth school; βjis the overall constant (intercept) and β1 X1ij +…+βp Xpij are the effects of individual level variables on pupil outcome; eij is the variation in outcome at the individual level.

Embedded Image

γ00 is the average value of the outcome across all schools and γ01 Z1j +…+γ0q Zqj are the effects of school level variables; u0j is the variation at the school level.

The degree of resemblance between pupils belonging to the same school can be expressed by the ICC. ICC is the proportion of variance that is accounted for by the school level. For studies where ICC was not presented, we have calculated it when components of variance were available. For continuous outcomes the following formula was used:

Embedded Image

When the outcome was binary the following formula was used:

Embedded Image

“σu2” denotes the school level variance and “σe2” is the variance at the individual level in a linear model, whereas the logistic distribution for the individual residual implies a variance of π2/3 = 3.29.19

Where possible, we present between school variation in pupil outcome explained by school level variables for each study when individual level variables were controlled (table 1). When this proportion was not calculated in the reviewed study we made the calculation if components of variance were available. Explained school variation can be calculated as the residual between school variance explained by school level variables after the variance explained by pupil level variables is taken out.19

RESULTS

What is the evidence that school level factors explain between-school variation in pupil outcomes?

In schools without health and antismoking policies, smoking was more prevalent among pupils (OR 1.20−2.77).22–24 In two of these studies, peer smoking habits were controlled in the statistical model. Moreover, it was shown that school norms and values influenced pupils’ smoking habits and alcohol use. A competitive climate meant a higher risk of smoking initiation (OR 1.17−1.22), and pupils from Catholic schools were more likely to smoke than were pupils from non-Catholic schools.20 The frequency of alcohol use was higher in schools with a more pronounced drug subculture.26 School level determinants explained 4%−40% of between school variation in pupils’ smoking habits and alcohol use.20,22,23,26

School climate had a significant effect on pupils’ wellbeing in three papers.31,32,34 In schools where the relationship between teachers and pupils was good and where bullying did not occur, pupils’ wellbeing was improved.31 Van den Oord and Rispens showed that, in schools where a low proportion of pupils had plans for future education, feelings of fear and uncertainty were more common.34 In schools with a small number of girls, it was less common for pupils to be victims of physical violence, and in schools that participated in pupil exchange programmes with other schools, disruptive behaviour among pupils was less common.32 In this study, a large number of correlations between school level variables and outcome were investigated and it is possible that some correlations will reach significance by chance. The findings by Mooij were not interpreted, and the inclusion of the variable on exchange programmes was not motivated by theory or by results in earlier studies. Aspects of school social climate explained 5%–8% of variance.32

School average SES was correlated to different aspects of pupils’ problem behaviours or wellbeing.33–35 Schools with low average SES had higher rates of pupil victimisation.33 In schools with high poverty, more pupils carried weapons.35 Van den Oord and Rispens, however, showed an opposite correlation, reporting that in schools with high average SES pupils felt more fear and uncertainty.34 The authors attribute this finding to random variation. School average SES explained 19% of the variation between schools in the study on pupils carrying weapons.35 None of the other studies reported explained variance or presented components of variance, and further studies are therefore needed to verify the result in Wilcox’s study.

Pupils from high SES schools perform better than pupils from low SES schools. This was established in all included studies on school achievement.27–30 Furthermore, in two Australian papers a strong and negative effect of rurality on pupil achievement in mathematics and science was seen.29,30 This effect was independent as average SES was controlled in both studies. Between school variation explained by school SES and localisation was estimated to be between 12% and 98%.

In the single study on physical achievement it was shown that children in schools with physical education specialists, and those in schools where fitness tests were performed, had better cardiovascular endurance than pupils in other schools.36 These variables explained 10% of between school variance.

How much of the variation in pupil outcomes is conditioned by differences between schools (ICC)?

ICC varied between 9% and 12% regarding pupils’ smoking habits and alcohol use.22–24,26 In studies on school achievement, estimates of ICC varied considerably.29,30 In three studies on mathematic achievement ICC varied between 5% and 28%.21,31,32,34 In four studies on pupils’ problem behaviour and wellbeing, ICC did not exceed 8%.35 In a single study on carrying weapons in school ICC was estimated to be 25%.36 Finally, in a study on health related fitness ICC was reported to be 22%.28,30 Thus, the variation conditioned by school context, ICC, differed according to pupil outcomes. Health related behaviours such as physical exercise, smoking, and alcohol use seemed to vary between schools to a greater extent than did pupils’ problem behaviours and wellbeing.

What theoretical frameworks have been suggested to explain between school variation in pupil outcomes?

A complete report on the theoretical frameworks used is not presented here. Instead, some aspects are highlighted to illuminate problems and possibilities. Only one of five studies on smoking habits developed hypotheses drawn from theories.20 Johnson and Hoffmann refer to “social learning and strain theories”.37 Theoretical concepts operationalised include self esteem, negative peer association, and positive school attitude as individual and as aggregated measures. It seems that the other four studies on pupils’ smoking habits derived their measures from similar theoretical considerations, although these are not presented in the papers.

Theories on community aspects were used in two studies on school achievement. Blau et al rely on theory derived from neighbourhood research to elaborate hypotheses on neighbourhood attributes as determinants of pupils’ school achievement.28 Neighbourhood research shows that deficits in poor communities result in poor learning environments.38,39 The homogeneous character of deprived neighbourhoods creates enclaves denying young people social and cognitive challenges. Battistich et al27 take their point of departure in Durkheim’s theory on anomie (that is, feelings of alienation and normlessness).40 Also, Wilcox and Clayton’s study on pupils’ weapon carrying is based on theories of community ecological perspective.35 They refer to Jencks and Mayer’s idea that organisational effectiveness of institutions within neighbourhoods strongly affects the rates of harmful behaviours in these areas. Viable intracommunity institutions can serve as an indicator of informal social control and consequently of decreased neighbourhood problems.41

DISCUSSION

This review points out important school level determinants on pupil outcomes. There is some evidence that school health and antismoking policies affect pupils’ smoking habits, which confirms findings in earlier reviews.6–8 Furthermore, pupils in high SES schools performed better than pupils in low SES schools. Three studies showed that a school’s social climate affects pupils’ wellbeing. This is an interesting finding as earlier research had established that a school’s climate affects pupil achievement.1,2 Similar measures of school climate were used in the study of pupils’ achievement as in the study of pupils’ wellbeing. Determinants such as high expectations on pupils and a strong educational leadership are important not only for pupil achievement but also for the pupils’ sense of wellbeing. Finally, there is some evidence that rurality has a negative effect on school achievement. This observed effect was not confounded by average school SES.

The evidence from the literature is limited. We cannot conclude that the observed effects are causal. However, the school level determinants reviewed here can be considered fairly stable over time and the observed associations may therefore be interpreted as causal. Furthermore, the studies included in this review investigate different pupil outcomes within diverse frameworks and traditions. Therefore, drawing conclusions is not straightforward. Some methodological and theoretical aspects require attention.

An important result in empirical studies within public health is the extent to which between school variance can be explained by school level variables. In the reviewed studies, explained variance differed both regarding the same outcome and comparing different outcomes. This variation could be explained by which individual level variables were controlled. Inclusion of multiple, inter-correlated individual level variables could decrease variance explained by school level variables.42 Furthermore, individual level variables such as peer attitudes or influence may be mediators rather than confounders. When such predictors are controlled in the statistical model they may act as proxies for contextual effects and can therefore be misinterpreted and possibly dilute the school effect. Peer attitudes and norms could instead be regarded as the path on which pupils’ smoking habits are influenced.6 Hence, cross level interactions in the statistical model could provide valuable information on how school context may differentially affect pupil outcomes. In the reviewed studies, such interactions were not explored, however. Moreover, if variance components were always partitioned to the school and pupil level in the null model as well as in the full multilevel model, it would be possible to calculate explained variance from results in a single study. Such meta-analyses would be valuable to practitioners in education and public health.

The extent to which variation in pupil outcomes was conditioned by differences between schools varied according to the outcome under study. Studies on pupils’ health behaviours reported an ICC of 7%–12%, while studies on pupils’ wellbeing generally reported much lower ICCs. Regarding school achievement, considerable disparities in ICC were seen. This may have been because of sampling bias. Two of the studies on school achievement used a random sampling technique.28,30 In the remaining two studies the sampling was deliberate or unclear.27,29 It is therefore not possible to rule out sampling bias as an explanation of disparities in observed ICC in the included studies on school achievement. Also, error in outcome measurement can cause biased results regarding ICC. However, school achievement was measured with similar tests in the included studies and thus such errors are not plausible. Nevertheless, there is a clear need for more standardised presentation of results from multilevel studies. In educational research, where multilevel statistical methodology has a long tradition, ICC is always presented. This is important as information on ICC focuses attention on the contribution of the school environment to pupil outcomes.

Studies carried out within a public health conceptual framework often lack theoretical justifications. Deriving hypotheses and operationalising variables from theory would facilitate interpretation of results. For example, how should the observed effect of average school SES be interpreted? In two studies on school achievement, the school SES variable was used to control for effects of other contextual variables such as rurality. In studies where other contextual were not included, it seems relevant to interpret school SES as a proxy for unmeasured school contextual features that are potentially relevant to pupil outcomes such as school achievement. Likewise, the evidence of an effect of school SES on pupil wellbeing may be interpreted in a similar way. Hence, school SES may suggest a potential for prevention. It is therefore important to find true contextual variables to develop intervention strategies.

What this paper adds

The rationale for carrying out the review is to summarise findings in studies where hierarchical data are analysed with multilevel technique. Such studies are still rare but the findings are highly interesting in public health as there is a possibility that earlier research has overestimated contextual effects.

Policy implications

Pupil outcomes vary between schools and targeting interventions to the school environment could have effects on pupil outcomes. Consequently, there is a potential for school based prevention of negative pupil outcomes.

Thus, relying on theoretical grounds would improve interpretation of results in studies on variation in pupil outcomes attributable to school environment, as the choice of explanatory variables would be more consistent. Yet, it remains a challenge to researchers to further examine how schools and communities are interconnected and how certain chains of events can lead to negative or positive pupil outcomes.

REFERENCES

View Abstract

Footnotes

  • Funding: the Swedish National Institute of Public Health funded this study.

  • Competing interests: none declared.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • In this issue
    Carlos Alvarez-Dardet John R Ashton