Article Text
PDF
Abstract
Background We systematically reviewed and meta-analysed evaluations testing the effectiveness of positive youth development (PYD) interventions for reducing violence in young people.
Methods Two reviewers working independently screened records, assessed full-text studies for inclusion and extracted data. Outcomes were transformed to Cohen's d. Quality assessment of included evaluations was undertaken using the Cochrane risk of bias tool. Effect sizes were combined using multilevel meta-analysis. We searched 21 databases, including MEDLINE, PsycINFO, CINAHL and CENTRAL, and hand-searched key journals and websites. We included studies where the majority of participants were aged 11–18 years and where interventions were delivered in community (not clinical or judicial) settings outside of normal school hours. We excluded studies targeting predefined physical and mental health conditions or parents/carers alongside young people. We defined violence as perpetration or victimisation of physical violence including violent crime.
Results Three randomised trials were included in this systematic review. Included evaluations each had design flaws. Meta-analyses suggested that PYD interventions did not have a statistically significant effect on violence outcomes across all time points (d=0.021, 95% CI −0.050 to 0.093), though interventions did have a statistically significant short-term effect (d=0.076, 95% CI 0.013 to 0.140).
Conclusions Our meta-analyses do not offer evidence of PYD interventions in general having effects of public health significance in reducing violence among young people. Evaluations did not consistently report theories of change or implementation fidelity, so it is unclear if our meta-analyses provide evidence that the PYD theory of change is ineffective in reducing violence among young people.
- CHILD HEALTH
- VIOLENCE
- SOCIAL SCIENCE
Statistics from Altmetric.com
Introduction
Preventing youth violence continues to be a public health, education and criminal justice priority.1–3 Evidence from a UK survey suggests that by age 15–16 years, a quarter of young people have carried a weapon and 19% report attacking someone with the intention to hurt them seriously.4 Violence is subject to marked social inequalities5 and is associated with an increased risk of: physical health problems;6 engaging in health risk behaviours such as substance use;7–9 long-term emotional, behavioural and mental health problems6 ,10 ,11 and self-harm and suicide.12 Moreover, gang involvement is associated in longitudinal studies with acute health risks and strongly correlated with later-life offending and serious adverse mental health outcomes.13 Economic costs associated with youth aggression are extremely high.14–16
Positive youth development (PYD) interventions have recently been the target of increased investment in the UK, as proposed by the UK government's Positive for Youth report and the devolved governments in Scotland and Wales, and recent investments by the London mayor.17–20 PYD is a complex intervention with varying definitions, though a review by the National Youth Agency in the UK has articulated a definition that focuses on promotion of positive assets in young people, rather than traditional ‘risk reduction’ approaches.21 This promotion of positive assets, according to a definition from the USA, includes developmental skills, such as self-regulation, bonding and resilience; prosocial norms, including academic achievement, acquisition of cognitive and vocational skills, and community involvement; development of positive social identities; strong connections with peers and adults and caring for others.22–25
In addition, it is important to focus on community-delivered interventions over school-delivered interventions for several reasons. School-delivered interventions may no longer be tenable as a growing focus on academic metrics means that schools have a decreased ability to focus on broader social development. Community-based interventions also have the potential to divert young people from antisocial behaviours outside of school hours, but they may also have greater potential for iatrogenesis due to the potential for social deviancy training—that is, bringing young people with a variety of risk profiles together may induce greater risk-taking in otherwise lower-risk young people. Finally, PYD is intended to be a voluntary activity, which is not amenable to a school setting, where attendance is compulsory.
Previous reviews of PYD interventions for violence outcomes in young people are now out of date. Specifically, Roth and Brooks-Gunn,26 in their narrative review, flagged the need for more evaluations, though they found early evaluations of PYD interventions for violence outcomes encouraging. In Catalano et al's27 systematic review across different outcomes, PYD interventions were described as associated with decreased aggressive and violent behaviour. In the face of increasing investment in PYD interventions for what continues to be a pressing public health concern, there is a need for evidence as to the effectiveness of these interventions. Thus, we present here an up-to-date systematic review of community-delivered PYD interventions and the first focused specifically on violence outcomes in young people.
Methods
The systematic review of PYD effects on violence reported in this study was part of a larger set of linked syntheses addressing theory, process evaluations and outcome evaluations of PYD interventions on substance use or violence.28 We registered our methods a priori in a protocol (PROSPERO CRD42013005439).29 This project was approved by the research ethics committee of the Institute of Education's Faculty of Children and Learning (ethics approval reference number FCL 544).
Studies were included in the broader evidence synthesis if they met the following criteria:
Published after 1985 and up to the point of searching;
Written in English;
Reported a theory of change, process evaluation or outcome evaluation that was experimental (ie, randomised) or quasi-experimental (ie, non-randomised, but employing a prospective comparison group) in design;
Focused on youth aged 11–18 years;
Focused on prevention of violence or substance use and
Focused on PYD.
For the systematic review reported in this study, we only included evaluations of PYD interventions that included measurement of violence outcomes. We defined violence as perpetration or victimisation of physical violence including violent crime. We defined PYD based on prior research30 as voluntary education outside of school hours aiming to promote generalised (beyond health) and positive (beyond avoiding risk) development of assets (bonding, resilience, social, emotional, cognitive, behaviour or moral competence, self-determination, spirituality, self-efficacy, clear and positive identity, belief in the future, recognition for positive behaviour, opportunities for prosocial involvement and/or prosocial norms). We judged that interventions were focused on PYD if they promoted an asset characteristic of PYD in multiple domains (eg, family, school or community), or multiple assets applied to one domain. We included interventions that were delivered in community (not clinical or judicial) settings outside of normal school hours. We excluded studies targeting predefined physical and mental health conditions or parents/carers alongside young people, as this would have detracted from this review's focus on primary prevention and introduced population heterogeneity.
Between October 2013 and January 2014, we searched 21 databases, free-text searched websites and hand-searched journals (see online supplementary file 1 for details of search strategies). Working in pairs, we initially screened sets of the same references in batches of 100 until 90% agreement was reached. We repeated this process for assessing full-text studies. We then conducted data extraction and study appraisal in duplicate and independently using, respectively, an extraction form that was initially piloted on two studies (see online supplementary file 1) and a modified version of the Cochrane risk of bias tool.31
Supplementary file
We extracted relevant effect sizes into a spreadsheet and converted them into standardised mean differences (Cohen's d) using available study information. We adjusted direction as necessary so that positive effect sizes would indicate an effect size in favour of the intervention. Where additional imputation of outcome-related data was necessary, we flagged a range of reasonable assumptions about p values that were not explicitly reported for sensitivity analysis (available on request).
We meta-analysed outcomes using a two-level multilevel meta-analysis method with random effects at the between-study (ie, programme) level and at the within-study (ie, outcome) level. The pooled effect size generated by a multilevel meta-analysis includes all the information from multiple effect sizes while correcting for non-independence between observations. We specified one model including all intervention follow-up measurements and one including just postintervention measurements (ie, excluding one study's long-term follow-up measurements). While we planned initially to undertake a multivariate meta-analysis, the diversity of outcomes and unavailable variance–covariance matrix meant that an alternative method was necessary.
Results
Our searches returned 32 394 deduplicated abstracts (figure 1). We assessed 689 of these in full text and identified four study reports of three distinct studies that met our definition of outcome evaluations of PYD and evaluated violence outcomes: Big Brothers Big Sisters (BBBS),32 ,33 Quantum Opportunity Project (QOP)34 ,35 and National Guard Youth ChalleNGe Program (NGYCP).36–38
Flow of studies through the review. Inappropriate study design refers to studies that were not outcome evaluations with randomised or quasi-experimental designs, process evaluations or reporting a theory of change. PYD, positive youth development.
Characteristics of included studies
All three studies used a randomised evaluation design. In all cases, participants were randomised within-site rather than by cluster. The comparator in all evaluations was no additional intervention—that is, for BBBS24 ,25 and NGYCP,36–38 control group participants did not receive the intervention, and for QOP,30 ,31 control group participants attended high school alongside those who were receiving the intervention. All included studies were conducted across multiple sites in the USA. Follow-up was 18 months postrandomisation for BBBS28 ,29 (considered in this analysis to be ‘postintervention’), and at postintervention and 18 months postintervention for NGYCP.36–38 The evaluation of QOP30 ,31 had multiple follow-ups, but the postintervention follow-up is the only one that presents violence outcomes.
Study quality was variable (table 1). None of the evaluations provided enough information to determine the risk of bias in sequence generation, though evaluators of BBBS28 ,29 explained that allocation concealment was achieved by randomisation through an external survey contractor. Blinding was impossible in two of the interventions, though we were unclear as to whether participants in the NGYCP36–38 were blinded as to intervention assignment. Trials of QOP30 ,31 and NGYCP36–38 used weighting analyses to account for missing data and accounted for clustering using ‘fixed-effects’ models, but the evaluation of BBBS28 ,29 used only complete case analysis and did not appear to account for clustering. We could not determine selective outcome reporting, and we did not observe that the included outcome evaluations had other significant flaws that placed them at high risk of bias.
Risk of bias judgments for included studies
Characteristics of included interventions
BBBS28 ,29 was a mentoring programme targeted to youth who generally lived in single-parent households and, along with their parents, agreed to the match, though specific eligibility criteria varied by site and generally aimed to identify ‘at-risk’ youth. This specific evaluation included young people between the ages of 10 and 16 at baseline. In BBBS,28 ,29 potential adult mentors drawn from the community were evaluated by programme staff, who were often professional social workers, and then matched with a young person for regular (generally several times a month) meetings over a long-term relationship. These lay mentors were trained in recognising and reporting abuse and, though not required, also often received training in communicating with youth. Volunteers received monthly supervision for the first year of the match and quarterly thereafter. The intervention included no formal education, but rather the ongoing relationship with a trusted adult was intended to develop specific positive assets such as improved self-esteem, life coping skills, academic performance, social relationships with family and friends, and cultural awareness, though experiences provided by the mentor.
NGYCP36–38 was delivered to adolescents aged between 16 and 18 years who either had left school or been excluded, who were unemployed and who were not involved in the correctional system. It was run as a 5-month military-style ‘boot camp’ including a ‘pre-ChalleNGe’ and a residential component that included life skills education, work preparation and completion of the secondary school diploma. After the military boot camp, participants completed job placements and structured mentoring. Mentoring was provided by programme staff and by mentors from the community nominated by participants. In an unusual feature, the intervention was primarily delivered by the National Guard, a branch of the US military run at the state level. Though the intervention did not set out an explicit theory of change, the key principle was that a ‘wraparound’ approach that addressed underlying issues in youth achievement and exposed them to the structure of a military context would be more effective than other less intensive approaches. While the programme did not appear to include a large amount of prevention education, the intervention promoted the positive assets of job skills and life skills training, academic excellence, leadership and citizenship skills, community service, physical fitness, and health and hygiene. The military boot camp was designed to incorporate these activities, which evaluators called ‘positive youth development’, to increase self-efficacy and self-esteem.
Finally, QOP30 ,31 was delivered in schools with dropout rates of 40% or more. It was further targeted to students who were in the bottom two-thirds of the grade distribution in the entering class of their secondary school and who did not have special educational or disability needs that would prevent participation. Since the programme enrolled students on initiation of secondary school, the average age of participants was 14. The intervention was delivered by staff from community-based organisations in a school context, including a substantial case management component tied in with mentoring provided by programme staff; academic assessment, planning and tutoring; community service and leisure activities and, when necessary, support over the summer vacations. The specific positive assets promoted as a core part of the intervention were cultural awareness, community service and academic achievement, though some sites also included health and hygiene and life skills. Staff members were youth workers who assumed ‘round-the-clock’ on-call responsibilities for participants assigned to them as part of the case management model. The intervention's theory of change was not explicit, but appeared to be premised on completion of secondary school education as a way to prevent antisocial behaviours and to attain employment. There appeared to be little specific prevention education.
Meta-analysis of included studies
We included 10 effect sizes from 3 distinct studies in an overall meta-analysis and 7 effect sizes from 3 distinct studies in a meta-analysis of short-term outcomes (ie, outcomes measured at postintervention). We did not look at longer term outcomes alone because they were derived from only one report. Findings were mixed across studies, but tended towards the null (table 2). None of the included studies reported outcomes relating to violence victimisation. While BBBS28 ,29 and QOP30 ,31 measured violence outcomes that were self-explanatory, evaluations of NGYCP36–38 defined violence incidents as those involving ‘any type of physical aggression’.
Outcomes reported by included studies
PYD interventions did not have a statistically significant effect on violence outcomes across all time points (d=0.021, 95% CI −0.050 to 0.093) (table 3 and figure 2). There was no meaningful programme-level heterogeneity in this finding (I2=0%). Short-term outcomes did yield a statistically significant effect (d=0.076, 95% CI 0.013 to 0.139), though this finding was marginally significant (p<0.10) in sensitivity analysis and should thus be regarded with caution. Again, there was little meaningful programme-level heterogeneity (I2=0%).
Meta-analysis of violence outcomes in PYD interventions
Violence outcomes for included studies. Positive values indicate a beneficial effect of the intervention. BBBS, Big Brothers Big Sisters; NGYCP, National Guard Youth ChalleNGe Program; QOP, Quantum Opportunity Project.
Discussion
This is the first systematic review of PYD interventions, specifically addressing and meta-analysing violence outcomes in young people. Though our search and selection criteria were rigorous and extensive, we were only able to locate three examples of PYD interventions with published outcome evaluations. These examples were diverse, though all met the definition of PYD that we created based on prior research. All promoted positive assets as a core of their interventions. One common way in which they did this was through extensive mentorship as a core component, delivered by a variety of people. Programmes, however, varied in how they operated. While NGYCP36–38 essentially functioned as a ‘school replacement’ programme, QOP30 ,31 acted to supplement school attendance and BBBS28 ,29 functioned separately from school.
Our meta-analysis yielded a pooled effect for violence outcomes that was not statistically significant over all time points, and was of marginal statistical significance immediately postintervention. Furthermore, the size of the pooled intervention effect was of questionable public health significance. Given the diversity of programmes, it is surprising that there was little statistical heterogeneity in either analysis, either between studies or within studies, though assessment of this was hampered by the small number of studies.
Another possible source of heterogeneity in effect is the age range that interventions target. The interventions included in this systematic review addressed young people at different stages of development including those not yet in adolescence (in part, BBBS28 ,29) to those in the later stages of adolescence (NGYCP36–38). Though we were not able to assess for moderation of intervention effect by age range of participants, future interventions may wish to consider how their approach to reducing violence via PYD is influenced by the age range being targeted.
It is possible that this non-significant pooled effect size is not necessarily an indication that interventions based on a PYD theory of change are ineffective in reducing violence. It is possible that other forms of PYD intervention might be effective. PYD focused more specifically on violence might be more effective. Although evaluated in terms of their effects reducing violence, these interventions included in this review did not specifically seek to target violence outcomes, though the process evaluation of NGYCP36–38 did report36 ,39 that staff members aimed to address gang membership. It is possible that PYD interventions focusing especially on violence may show stronger effects. It is also possible that the PYD interventions may have differential impacts on violence outcomes depending on the type of violence considered. That is, PYD interventions specifically seeking to reduce perpetration of violent crimes may have different effects than those seeking to address social and emotional learning skills to prevent fighting. We also note that we were unable to locate any measures of violence victimisation in the included studies. We did not find any evidence of a harmful effect of PYD interventions on violence outcomes. This is a concern because other interventions aiming to reduce juvenile delinquency, such as Scared Straight, have shown harmful effects on young people's engagement in criminal behaviours40 due to the potential for ‘social deviancy training’, in which programme participants model antisocial behaviours for peers, and social modelling effects from prisoners.
In addition to the limited number of studies our review included, our review may have been subject to publication and retrieval bias. We were unable to assess publication bias because of the few studies we included, and our extensive search and retrieval procedure was protective against the potential for retrieval bias.
Finally, in considering PYD interventions for adoption in the UK context, policymakers and commissioners should consider that interventions may not be readily generalisable between contexts. This is particularly given that all three included interventions were designed, conducted and evaluated in the UK context. In particular, the UK and the USA have considerably different service systems for supporting high-risk youth, suggesting that pathways to service referral may be different as well. Moving forward, PYD interventions considered as part of a strategy to reduce violence in young people—especially in social and service contexts where these interventions have not yet been evaluated—should be implemented as part of a carefully designed, rigorous evaluation strategy, preferably including randomised evaluation. However, it is important to acknowledge that this ‘gold standard’ approach to evaluation may not be possible in interventions that seek to target high-risk youth. Communities may not be amenable to randomisation. Regardless, there is a need for rigorous evaluation of PYD interventions. And in general, there is a need for more research on whether PYD interventions affect violence outcomes in young people, as well as how PYD interventions affect these outcomes, for whom these effects are strongest, and which configuration of assets characteristic of PYD is most effective. As PYD interventions continue to be a popular choice for policymakers, research that establishes whether interventions based on this theory of change are effective and that offers a guidepost for implementation will be of critical importance.
What is already known on this subject
Positive youth development (PYD) interventions focus on promotion of positive assets over more traditional risk reduction interventions.
It is unclear whether PYD is of use for preventing violence in young people.
What this study adds
PYD interventions may have a short-term effect, but not a long-term effect, in preventing violent behaviours in young people.
The scarcity of published evidence suggests that additional research is necessary before funding to these programmes is increased.
Acknowledgments
The authors thank Hayley Reed, the Involving Young People Research Officer at DECIPHer, for her advice on this project, and all the young people who worked with us on this project during our consultations with ALPHA young people's public input advisory group based in DECIPHer.
References
Footnotes
Contributors GJM-T contributed to conducting the review, led the meta-analysis of outcome evaluations and contributed towards drafting this paper. KD managed the review day-to-day, and contributed to searching and report drafting. AF contributed to project planning and commented on report drafts. JT contributed to project planning, advised on meta-analysis and commented on report drafts. KH contributed towards report drafting. RC contributed to project planning and commented on report drafts. SM contributed to project planning and commented on report drafts. CB planned the project, and led on synthesis and report drafting.
Funding This project is funded by a grant from the National Institute for Health Research Public Health Research programme (grant PHR 12/153/19). The work was undertaken with the support of The Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer), a UKCRC Public Health Research Centre of Excellence. Joint funding (MR/KO232331/1) from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, the Welsh Government and the Wellcome Trust, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged.
Competing interests None declared.
Ethics approval Institute of Education Faculty of Children and Learning.
Provenance and peer review Not commissioned; externally peer reviewed.