Health outcomes of youth development programme in England: prospective matched comparison study
BMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b2534 (Published 07 July 2009) Cite this as: BMJ 2009;339:b2534
- Meg Wiggins, senior research officer1,
- Chris Bonell, senior lecturer2,
- Mary Sawtell, research officer1,
- Helen Austerberry, research officer1,
- Helen Burchett, research student2,
- Elizabeth Allen, lecturer 2,
- Vicki Strange, senior research officer1
- 1Social Science Research Unit, Institute of Education, University of London, London WC1H 0NR
- 2London School of Hygiene and Tropical Medicine, London WC1E 7HT
- Correspondence to: C Bonell chris.bonell{at}lshtm.ac.uk
- Accepted 16 March 2009
Abstract
Objective To evaluate the effectiveness of youth development in reducing teenage pregnancy, substance use, and other outcomes.
Design Prospective matched comparison study.
Setting 54 youth service sites in England.
Participants Young people (n=2724) aged 13-15 years at baseline deemed by professionals as at risk of teenage pregnancy, substance misuse, or school exclusion or to be vulnerable.
Intervention Intensive, multicomponent youth development programme including sex and drugs education (Young People’s Development Programme) versus standard youth provision.
Main outcome measures Various, including pregnancy, weekly cannabis use, and monthly drunkenness at 18 months.
Results Young women in the intervention group more commonly reported pregnancy than did those in the comparison group (16% v 6%; adjusted odds ratio 3.55, 95% confidence interval 1.32 to 9.50). Young women in the intervention group also more commonly reported early heterosexual experience (58% v 33%; adjusted odds ratio 2.53, 1.09 to 5.92) and expectation of teenage parenthood (34% v 24%; 1.61, 1.07 to 2.43).
Conclusions No evidence was found that the intervention was effective in delaying heterosexual experience or reducing pregnancies, drunkenness, or cannabis use. Some results suggested an adverse effect. Although methodological limitations may at least partly explain these findings, any further implementation of such interventions in the UK should be only within randomised trials.
Introduction
Youth development programmes aim to promote overall personal development, self esteem, positive career and other aspirations, and good relationships with adults among vulnerable young people, in order to promote motivation to avoid teenage pregnancy and other negative health and social outcomes.1 Alongside education on sex and drugs, these programmes offer activities such as education, social skills development, mentoring, arts, sports, and volunteering. The most notable of these, the Children’s Aid Society’s Carrera programme was an intensive after school intervention combining youth development, sexuality education, and regular sexual health clinic check-ups. This was reported as delaying young women’s sexual experience, increasing their use of contraception, and reducing pregnancies when implemented in New York, but no such benefits were seen for young men.2 Several influential reviews subsequently identified youth development as a promising approach to reducing teenage pregnancies.3 4 However, studies of attempted replications of the Carrera programme in the United States did not report benefits for young women or men, and recent reviews have called for further evaluation.1 5 6 We report on sexual health and other outcomes of a youth development intervention implemented in England. We also provide some key findings on process, although this will be covered further in a forthcoming paper.
The Carrera programme included tutoring as well as work preparation, sex and drugs education, arts and sports, and referrals to health interventions. Informed by the Carrera programme and other youth development programmes, the Young People’s Development Programme (YPDP) was a three year (April 2004-March 2007) initiative funded by England’s Department of Health and targeting young people aged 13-15 at entry deemed by teachers or other care professionals to be at risk of teenage conception, substance misuse, or exclusion from school. These participants were recruited in three annual cohorts. The programme aimed to reduce teenage pregnancy, substance use, and other outcomes (table 1⇓) through an intensive programme focused on overall personal development. Content was to include education, training/employment opportunities, life skills, mentoring, volunteering, health education (particularly sexual health and substance misuse), arts, sports, and advice on accessing services (such as family planning and substance misuse services).
Aims and outcomes of Young People’s Development Programme
Through competitive tendering, the Department of Health identified 27 existing projects to deliver the YPDP, which received additional funding and support from the National Youth Agency, a non-governmental agency. Tenders were judged on the quality of the proposed work, local deprivation, and teenage pregnancy rates and to ensure geographical spread and ethnic/gender diversity of participants. Although the YPDP was influenced by the Carrera programme, it intentionally differed from the outset in that provision was less tightly defined and young people were targeted on their perceived behavioural risk and were to be involved for 6-10 hours a week for one year, compared with up to 15 hours a week for three years in the Carrera programme.
Methods
The Department of Health commissioned us to independently evaluate the process, outcomes, and costs of the YPDP. A trial randomising individual participants was not feasible because groups of young people were to be referred to the YPDP together. A cluster randomised trial, whereby agencies would be randomly allocated to provide the YPDP or continue normal work, was ruled out by the competitive tendering outlined above. We therefore opted for a prospective matched cluster comparison with pre-intervention/post-intervention data from the young people in 27 intervention sites and 27 comparison sites matched by region, local deprivation, teenage pregnancy rates, area (urban, rural, or seaside) and sector (voluntary or statutory). In comparison sites, we recruited from youth service providers that had bid and were shortlisted for, but did not receive, YPDP funds, as well as pupil referral units providing education to young people not attending mainstream schools (which in YPDP sites were referrers to the programme). Youth service providers in comparison sites also worked in deprived areas with high rates of teenage pregnancy and had failed to be selected for YPDP primarily on the basis of quality of tender. We recruited young people at these sites by asking workers to identify young people aged 13-15 who were at risk of teenage pregnancy, substance misuse, or exclusion from school (that is, as for the YPDP), although in practice field workers sometimes asked workers to identify “vulnerable” young people. We thus aimed to recruit young people in comparison sites who might have been referred to the YPDP had it been delivered in their area.
As required by the funders, we aimed to examine the effects of YPDP on multiple outcomes to reflect the broad aims of the programme and pre-hypothesised measures before analysis (table 1⇑). We collected self reported questionnaire data from young people in the YPDP and comparison groups at baseline (shortly after joining) and approximately nine and then 18 months later. Our original statistical power calculation was based on 35 YPDP sites (as was originally planned) and 35 comparison sites, with an estimated 2300 young people participating in each arm. This sample size, with a 5% level of significance, 80% power, and an estimated intracluster correlation coefficient of 0.02, would enable us to detect a 50% change in teenage pregnancies on the basis of an assumed 4.5% teenage pregnancy rate informed by a previous trial.7 We based our sample size calculations on teenage pregnancy, as other outcomes were likely to be more prevalent. With the subsequent reduction of YPDP sites to 27 by the Department of Health (and therefore similarly the number of comparison sites) and lower than expected average numbers of young people at each site (around 40) the study was powered to detect a reduction in pregnancy rates of 62% at 5% significance.
Young people completed questionnaires at the programme site or a nearby site (for example, a minibus) or by telephone interview. In telephone interviews or for participants with literacy problems, a researcher read out questions and indicated response options. Of the 2371 young people who participated in the YPDP, 1637 (69%) completed baseline questionnaires, as did 1087 young people in comparison sites (table 2⇓). Follow-up 1 questionnaires were completed by 1054 (64%) in YPDP sites and 599 (55%) in comparison sites. Timing of the evaluation meant that follow-up 2 could be completed only by young people recruited in the first two (2004/5 and 2005/6) recruitment cohorts. In YPDP sites, 566 (43% of the baseline pool) completed follow-up 2 questionnaires; 338 (39%) of comparison participants did so. The vast majority of those who did not complete questionnaires were missed because of irregular attendance (baseline, follow-up 1) or because contact details changed and they could not be located (follow-up 2). Monitoring data indicated that YPDP participants who did not complete baseline questionnaires spent less time on the YPDP (mean 89 v 225 hours), were the same age at entry (mean 14.2 years), and were significantly more likely (P<0.05) to be of black/minority ethnicity (25% v 22%) or female (38% v 37%). Participants gave informed, signed consent to data collection, and data were stored in anonymised form.
Response rates and baseline characteristics. Values are numbers (percentages) unless stated otherwise
We examined outcomes by logistic regression, reporting odds ratios, both unadjusted and adjusted for age and measures of pre-hypothesised potential confounders that differed significantly between arms at baseline (table 2⇑). We accounted for the clustering of data in all analyses and excluded participants from analyses if relevant data were missing. We present results for pregnancy stratified by gender, but for other outcomes we present overall results, only stratifying by gender if interactions were significant.
To check whether differential attrition at the follow-ups might explain our findings, we also did a weighted analysis. We identified baseline variables that predicted whether participants provided data at each follow-up (available on request) and created inverse probability weights so that outcomes reported by participants who did report data at that follow-up but had a low probability of doing so would be given more weight (www.lshtm.ac.uk/msu/missingdata/weighting_web/index.html). As an additional check, we did an analysis of follow-up 2 outcomes by using propensity scores to balance covariates in the two groups. We included a categorical propensity score covariate in models that also accounted for the clustered nature of the data.8 We computed the propensity score by using logistic regression with the dependent variable being receipt of the intervention and the independent variables being the baseline exposures listed in table 2⇑. We also did a small number of sensitivity analyses to examine whether the way the programme was implemented had any effect. These analyses compared participants who attended agencies where participation was voluntary versus those where participation was required by the school, attended agencies judged by the National Youth Agency to deliver high versus moderate versus adequate quality work, participated for more versus fewer overall hours, and experienced a more versus less holistic package of participation (all versus comparison group). Our methods for evaluating these processes are reported elsewhere.9
Results
Some overall baseline differences existed (table 2⇑). For some (such as housing, family structure) YPDP participants were more vulnerable, and for others (such as alcohol consumption, heterosexual experience) comparison participants were more vulnerable. We also checked for baseline differences by gender, and in general the results were consistent with the differences found overall. Overall, 60/622 young women in the YPDP group reported using no contraception at most recent sex compared with 33/477 comparison participants. Overall, 66/1015 young men in the YPDP group reported using no contraception at first sex compared with 65/610 in the comparison group. These differences were statistically significant (P<0.05) in analyses that included only those with baseline heterosexual experience but not analyses including, respectively, all young women and all young men at baseline.
Tables 3⇓ and 4⇓ show the outcomes at the two follow-ups. Regarding health outcomes, significantly more pregnancies were reported post-baseline among young women in the YPDP group (38) than in the comparison group (13); this association remained significant after adjustment for all pre-hypothesised confounders (table 4⇓) and also after confirmatory adjustment for no contraception at most recent sex at baseline. The difference was greater among the first cohort but remained among subsequent cohorts. Pregnancies were reported in 16 YPDP sites (median 1 per site, interquartile range 0-2). We found no significant differences in the proportion of young men who reported awareness of causing a pregnancy. Although we found no significant differences between young people in the YPDP and comparison groups overall in those expecting to be a parent by age 20, female YPDP participants were significantly more likely than comparisons to report this expectation at follow-up 1; this remained significant in all models.
Outcomes at follow-up 1. Values are numbers (percentages) unless stated otherwise
Outcomes at follow-up 2. Values are numbers (percentages) unless stated otherwise
At first and second follow-up, non-significantly more young people in the YPDP group than in the comparison group reported heterosexual sex (tables 3⇑ and 4⇑) but with a significant gender interaction: significantly more young women in the YPDP group than in the comparison group reported heterosexual sex at follow-up 2 (table 5⇓); this remained in all models. We found no differences in the proportion of young people who reported two or more partners or regular use of condoms at follow-up 1 or 2. No significant differences existed between YPDP participants and comparison participants in any measure of substance use or in worry, anger, low self esteem, or difficulty talking about personal matters to friends.
Outcomes at follow-up 1 and 2 stratified by sex. Values are numbers (percentages) unless stated otherwise
In terms of social outcomes, at the first follow-up significantly more young people in the YPDP group than in the comparison group reported truanting in the previous six months; this association remained in all analyses. This was largely explained by increased truanting among young women in the YPDP group in the first year cohort of the project. At follow-up 2, the proportion of the YPDP participants who truanted fell and the difference between the intervention and comparison groups was not significant. At the first follow-up, the number reporting temporary exclusion from school in the previous three months was lower than at baseline, with no significant differences between the YPDP and comparison groups. However, among young women, those participating in the YPDP were significantly more likely to report temporary exclusion from school at follow-up 1. We found no differences overall or by sex in temporary exclusions from school or non-participation in education, employment, or training at follow-up 2. We found no significant differences between the YPDP and comparison groups in the number of participants who reported contact with the police at follow-ups 1 and 2 or reported official warnings or convictions at follow-up 2. No significant differences existed in the proportion of YPDP and comparison participants who reported at follow-up 1 that they expected to have a steady job by age 20. Our exploratory subgroup analyses found no differences in any health or social outcomes by programme implementation (data available on request).
Implementation of the programme will be reported in a separate paper, but to provide context for our findings on outcomes we report the key findings here. YPDP projects had initial challenges—for example, in recruiting staff and participants and modifying their practice to embrace the YPDP approach. By the end of the first year, nearly all projects operated a programme that offered the key YPDP components. Projects were able to recruit, retain, and engage many vulnerable young people in relatively intensive provision for a prolonged period (on average 173 hours over 40 weeks), although the average amount of time young people spent on the YPDP was less than targeted. Projects delivered the YPDP in diverse ways. Several projects responded to pressure to meet targets relating to recruitment, attendance, and retention by cooperating with schools to offer education to disaffected students as an alternative to attending some or all of the normal school day both on and off site.
Education about sex and drugs was delivered to different extents and in varying styles across sites. This was generally delivered by youth workers and aimed to enable participants to make informed decisions to delay sex/refuse drugs or reduce associated risks. Sites diverged as to whether they segregated sessions by age or gender. All sites aimed to refer participants to sexual health services when necessary, but according to monitoring data only six did so (although we suspect that not all referrals were recorded). Some but not all projects distributed condoms. Most young people, staff, parents, and other stakeholders rated the YPDP highly. Young people were especially positive about the activities on offer and their relationships with staff. Staff liked working in a more holistic way with young people and thought that through the YPDP they were offering a better service to their participants.
Discussion
Unexpectedly, our analysis suggested that participation in the YPDP was associated with higher rates of some outcomes than occurred at comparison sites. Among young women, YPDP participants more commonly reported teenage pregnancies, early heterosexual sex, and expectation of becoming a teenage parent, as well as temporary exclusion from school and truancy; these associations remained in all models. Our exploratory subgroup analyses found no explanation for these outcomes in terms of the type of agency or the “dose,” contents, or quality of the programme.
Strengths and weaknesses of the study
Our outcome findings might, at least in part, reflect methodological limitations. In practice, young people were sometimes recruited to our comparison group through use of a slightly more general criterion (“vulnerable young people”) than those used in recruitment to the YPDP (see above). However, our comparison group was recruited from sites matched on deprivation and rates of teenage pregnancy and was not systematically at lower risk at baseline. We adjusted for a broad range of pre-hypothesised potential confounders where these significantly differed overall between arms at baseline, checked whether gender specific baseline differences might account for the associations found, and did analyses incorporating propensity scores for data from follow-up 2. We cannot, however, eliminate the possibility of unmeasured/incompletely measured confounding, although this would have to be large to account fully for the unexpected associations found.
Attrition was a major challenge in this study, and our findings might result from participants at higher baseline risk being followed up more completely in the YPDP group than in the comparison group. However, our results suggest that this was true for only some baseline measures; the pattern was reversed for other measures. Weighting for losses to follow-up did not change our findings. However, differential attrition between arms in terms of unmeasured baseline risk factors might explain our findings. Some caution should also be exercised in interpreting our tests of significance, as we did a large number of these. However, our finding of consistent associations regarding sexual health outcomes among young women indicates that random error may not explain these.
Other limitations are likely to have biased the evaluation towards overestimating the benefits of the YPDP. YPDP sites had successfully tendered to participate in the programme, whereas comparison youth work agencies had tendered but not been chosen. Most YPDP agencies were likely to have been selected because they had better capacity, experience, or preparedness, which might have been expected to produce better outcomes regardless of added value from the YPDP. Our outcomes relied on young people’s self reports, so information bias might have led to some overestimation of the benefits of the YPDP. Furthermore, completion of questionnaires was greater among YPDP participants who spent longer on the programme, possibly leading to overestimates of benefits.
Loss to follow-up, as well as the greater than expected intracluster correlation coefficients (for example, 0.12 for heterosexual sex and 0.08 for teenage pregnancy), reduced our power to detect small associations between participation in the YPDP and our key outcomes. Although this cannot explain our unexpected findings of significant associations between participation in the YPDP and, for example, teenage pregnancy among young women, it might mean that some real associations were not detected. This might be the case, for example, with the non-significant associations between participation in the YPDP and weekly cannabis use and official warnings/convictions at follow-up 2. Finally, although our study involved multiple sites and a large sample of young people across England, its results cannot be assumed to be generalisable to other countries.
Meaning of the study
Our finding of no added benefits for YPDP participants over comparison participants for some outcomes, particularly among young men, is in line with findings from some recent studies of youth development programmes.1 5 6 The lack of added benefit may reflect both the high quality of work under way in some comparison sites and the challenge of bringing about detectable effects on health behaviours and outcomes where these are influenced by profound socioeconomic and educational inequalities, peers, and mass media. Some outcomes, such as boys’ temporary exclusions, were less commonly reported at follow-ups than at baseline, perhaps suggesting that young men in both intervention and comparison groups experienced benefits.
However, higher rates of pregnancy, heterosexual sex, truancy, and temporary exclusion from school among young women in the YPDP group do not tally with previous evaluations of youth development. These might be explained by the above methodological limitations or by some YPDP providers experiencing initial disruption as a result of participation in the YPDP, which may have negatively affected initial outcomes, such as young women’s truancy. However, our exploratory analyses found no evidence that outcomes were better in YPDP providers rated as high quality.
The unexpected sexual health outcomes are unlikely to be attributable to the sex education within the programme, because this was a relatively small and variably delivered component and because of the lack of previous evidence for harms arising from sex education.1 Another possibility arises from the potential effects of targeting young people deemed to be at risk. Previous studies suggest that some interventions targeting people at risk can expose participants to the influence of new peers who are more supportive of or more engaged in behaviours associated with risk, thereby spreading risk.10 11 12 13 Whereas the Carrera programme targeted young people in deprived areas, the YPDP aimed to bring together young people deemed to be at risk of teenage pregnancy, substance misuse, and school exclusion. This was not the case in comparison sites, where youth work did not generally target in this way and where pupil referral units targeted only on the basis of young people’s exclusion from school. Our evaluation of the process did not aim to examine the plausibility of this pathway but did find a few cases of participants experiencing bullying during their participation and a few parents who suggested that bringing together badly behaved children might spread misbehaviour.
Additionally, previous studies of interventions targeting vulnerable youths have suggested that adverse outcomes can arise from “labelling.”14 YPDP participants deemed to be “at risk” may have felt labelled as problematic despite the YPDP’s explicit emphasis on young people’s potential. This may have been the case especially where the YPDP was in effect a form of alternative education. Young women in the YPDP group more often reported that they expected to be a teenage parent, possibly reflecting lowered expectations arising from labelling. However, these are speculations and do not explain why effects seemed to differ between young men and women.
We believe that a combination of reasons best explains our findings. Informed by the precautionary principle in public health,15 we recommend that any future implementation of targeted youth development in the United Kingdom should occur only within the context of a randomised trial and with more emphasis on the definition and consistency of the intervention. Although we cannot be certain whether peer group and labelling effects explain our results, we would suggest that any future implementation should ensure that the intervention does not inadvertently bring participants, in particular young women, into contact with more risky peers. This might be achieved by separating provision for young women and men, or for those of different ages, and targeting participants in terms of social disadvantage (as the Carrera programme did) rather than behavioural risk. To minimise labelling, we recommend that any future youth development should be an addition rather than an alternative to school. We also recommend more attention to tackling the wider socioeconomic and educational influences on young people’s health.16
What is already known on this topic
Youth development delivered by the Children’s Aid Society’s Carrera programme to young people in disadvantaged areas of New York city was effective in reducing teenage pregnancies
Subsequent studies of attempted replications elsewhere in the United States did not find such benefits
What this study adds
A youth development programme was delivered to young people in England targeted as being at risk of teenage pregnancy, substance misuse, and school exclusion
More young women participating in this programme reported teenage pregnancies as well as early heterosexual experience and expectation of teenage parenthood than did comparison participants
Methodological limitations may explain at least some of the effects seen, but plausible causes may involve participants encountering more risk oriented peers and feeling labelled as problematic
Notes
Cite this as: BMJ 2009;339:b2534
Footnotes
Contributors: MW co-directed the study and oversaw all data collection and analysis. CB co-directed the study, advised on design and methods, and led the drafting of the paper. MS coordinated the study’s surveys and participated in data collection and analysis. HA coordinated qualitative fieldwork and participated in data collection and analysis. HB coordinated collation of monitoring data and participated in data collection and analysis. EA advised on statistical analysis. VS managed the study and coordinated data collection. All authors contributed to drafting the paper. CB is the guarantor.
Funding: The research was funded by England’s Department of Health. The funders required to see a draft of this paper 28 days before submission but in no way influenced the decision to publish or the contents of the paper. The funders were consulted about the design and methods used in the study and commented on drafts of the interim and final project reports. The research team was independent of the funders.
Competing interests: None declared.
Ethical approval: The Institute of Education research-ethics committee approved the study. Participants gave informed, signed consent to data collection, and data were stored in anonymised form.
Copyright statement/Licence for publication: The corresponding author has the right to grant on behalf of all authors and does grant on behalf of all authors, a non exclusive licence on a worldwide basis to the BMJ Publishing Group Ltd and its licensees, to permit this article if published in BMJ editions and any other BMJPG products and to exploit all subsidiary rights, as set out in the licence.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.