Effects of Sure Start local programmes on children and families: early findings from a quasi-experimental, cross sectional study
BMJ 2006; 332 doi: https://doi.org/10.1136/bmj.38853.451748.2F (Published 22 June 2006) Cite this as: BMJ 2006;332:1476
- Jay Belsky, professor of psychology and director (j.belsky{at}bbk.ac.uk)1,
- Edward Melhuish, professor of human development1,
- Jacqueline Barnes, professor of psychology1,
- Alastair H Leyland, director2,
- 1 Institute for the Study of Children, Families and Social Issues, Birkbeck, University of London, London WC1B 3RA
- 2 MRC Social and Public Health Sciences Unit, University of Glasgow, Glasgow G12 8RZ
- Correspondence to: J Belsky
- Accepted 9 May 2006
Abstract
Objective To evaluate the effects of Sure Start local programmes (SSLPs) on children and their families. To assess whether variations in the effectiveness of SSLPs are due to differences in implementation.
Design Quasi-experimental cross sectional study using interviews with mothers and cognitive assessment of children aged 36 months who speak English.
Setting Socially deprived communities in England: 150 communities with ongoing SSLPs and 50 comparison communities.
Participants Mothers of 12 575 children aged 9 months and 3927 children aged 36 months in SSLP areas; mothers of 1509 children aged 9 months and 1101 children aged 36 months in comparison communities.
Outcome measures Mothers' reports of community services and local area, family functioning and parenting skills, child health and development, and verbal ability at 36 months.
Results Differences between SSLP areas and comparison areas were limited, small, and varied by degree of social deprivation. SSLPs had beneficial effects on non-teenage mothers (better parenting, better social functioning in children) and adverse effects on children of teenage mothers (poorer social functioning) and children of single parents or parents who did not work (lower verbal ability). SSLPs led by health services were slightly more effective than other SSLPs.
Conclusion SSLPs seem to benefit relatively less socially deprived parents (who have greater personal resources) and their children but seem to have an adverse effect on the most disadvantaged children. Programmes led by health services seem to be more effective than programmes led by other agencies.
Introduction
Sure Start local programmes (SSLPs) represent a large scale, area based effort by the government of the United Kingdom to enhance the health and development of children under 4 years and their families who live in socially deprived communities in England. These programmes aim to improve services and create new ones in small areas with average populations of just under 13 000 people, including about 700 children aged 0-3 years. The first SSLPs began in 1999, and a total of 524 existed by 2004. SSLPs are a unique approach to enhancing the life prospects of disadvantaged children, in that all children aged 0-3 years and their families living in a prescribed area are “targets” of intervention, and thus of evaluation of effectiveness.
Because of their local autonomy, SSLPs do not have a “protocol” to promote adherence to a prescribed model, as do other early interventions that are known to be effective.1–3 All SSLPs are expected, nevertheless, to provide core services of outreach or home visiting; family support; support for good quality play, learning, and childcare experiences; primary and community health care; advice about child and family health and development; and support for people with special needs, including help in accessing specialised services. Community participation is central to the mission of these programmes, through local partnerships that bring together all people who are concerned with children in the local community, including health, social, and education services; the private sector; the voluntary sector; and parents.
Our report aims to evaluate the impact of SSLPs on children and their families by investigating differences between children and families in 150 communities with ongoing SSLPs and 50 comparison communities designated to become SSLPs at a later date. Because the government ruled out a randomised control trial, we have used a quasi-experimental, cross sectional study with extensive statistical controls. We sought to answer five questions on the effectiveness of SSLPs. Firstly, does the use of services differ between SSLP and comparison communities, and do parents rate SSLP communities more positively? This compound question reflects the “theory of change” that underlies SSLPs, which stipulates that enhancing services and fostering change in the community will benefit children and their families. Secondly, do families in SSLP and comparison areas function differently? Thirdly, does child health and development differ between SSLP and comparison areas? Because children can be affected directly (for example, by enhanced health care) or indirectly (by effects on parents), question four asks whether the effects of SSLPs on parenting mediate effects on child functioning. The final question asks whether variations in the effectiveness of SSLPs are due to certain differences in the implementation of these programmes. To answer the first four questions we compared SSLP and comparison communities, whereas to answer the fifth question we performed a within group analysis of the 150 SSLP areas. (See www.surestart.gov.uk/_doc/P0001867.pdf for full details on all methods and results.)
Methods
Design and participants
We randomly selected 150 of 260 SSLP areas after stratification across nine government regions within England; we used 50 areas waiting to become SSLP areas as comparison communities. Because all children aged 0-3 years and their families can benefit from SSLPs, we used an intention to treat design. We randomly selected for study families with children of 9 or 36 months from child benefit records during calendar years 2003-4. In each SSLP, we sought to recruit 80 children aged 9 months and 20 aged 36 months; in comparison areas 25 children were sought for each group. We gathered data on 12 575 children aged 9 months and 3927 aged 36 months (and their families) in SSLP areas and 1509 children aged 9 months and 1101 aged 36 months in comparison areas. The response rate was 84.4% and 73.4% for families of children aged 9 and 36 months, with no differences across comparison groups. No information was gathered on non-respondents. Mothers or guardians provided written informed consent.
Procedures and measurements
During a 90 minute home visit, mothers or guardians participated in a computer assisted interview; only English speaking children aged 36 months were given a standardised assessment of cognitive and linguistic functioning. Data collectors received extensive training.
Table 1 lists outcome variables and the relevant research questions. Tables 2 and 3 list family background variables that were covariates for outcome. Social deprivation was greater in several domains (such as income and education) in families in comparison areas than in SSLP areas. We gathered area information from diverse sources (such as hospital episode statistics (Department of Health), census statistics, benefit records (Department of Work and Pensions), and school achievement records (Department for Education and Skills)4) to provide additional statistical control. Factor analysis of community characteristics yielded summary scores that reflected the high versus low prevalence of community characteristics indicative of Indian subcontinent populations and young children; black (African or Caribbean) population and adults of working age; lone and teenage parents; unemployed adults; ill and disabled children; infant mortality; key stage 1 (age 7) school achievement; household crowding; council housing; and adults in poor health. These scores served as control variables if significantly related to dependent variables.
Outcome variables by research question in a study of Sure Startlocal programmes
Background characteristics of families with 9 month old children in a study of Sure Start local programmes. Values are number (%)
Background characteristics of families with 36 month old children in a study of Sure Start local programmes. Values are number (%)
We measured three variables in the implementation of SSLPs that might explain variations in the effectiveness of these programmes. “Reach” represented the number of eligible families contacted by SSLPs; “cost” reflected the annualised expenditure per child; and “lead agency” reflected the agency leading the partnership board.
Statistical analysis
We used multilevel models to evaluate the effects of SSLPs and accommodate hierarchically structured data (children and families nested within communities).5 All analyses were adjusted for child and family background variables and area characteristics before comparing SSLP and comparison areas; we analysed data on children aged 9 and 36 months separately.
To determine whether SSLPs differentially affected subpopulations, we tested selected two way interactions involving sex of the child, maternal employment, mother's age at birth of the child, lone parenthood, all members of the household unemployed, and income deprivation (table 1). Significant two way interactions resulted in comparisons between SSLP and comparison communities for selected subgroups, but not for data on children aged 9 months owing to low rates of significant interactions.
Whenever findings suggested possible mediational effects (SSLP→parenting→child), we tested mediation using the Sobel test.6 To investigate whether the three variables in the implementation of the programme accounted for variations in the effectiveness of SSLPs, we fitted multilevel models to the data collected from families living in SSLP areas only.
The findings presented are based on multiple imputed data sets in which missing values of all independent and dependent variables were estimated based on standard multiple imputation procedures.7 Across all available data, imputation of missing data on children aged 9 and 36 months resulted in 3% and 6% increases of the data, respectively.
Although results derived from imputed data and complete case data did not differ, we report significant findings only when present in analyses of both sets of data (table 4).
Imputed data mean scores and confidence intervals of measures that show significant differences between Sure Start local programmes (SSLPs) and comparison groups
Results
Effect of SSLPs on use of services and rating of communities
SSLPs did not seem to affect mothers' reports of service use or usefulness in either age group. Mothers of children aged 36 months (but not 9 months) living in SSLP areas rated their communities a little less favourably than mothers in comparison areas.
Effects of SSLPs on family function
Mothers of children aged 9 months living in SSLP areas reported less household chaos and mothers of children aged 36 months showed greater parental acceptance. Non-teenage mothers (86% of sample) of children aged 36 months reported less negative parenting when living in SSLP areas (see table 4).
Effects of SSLPs on children
SSLPs seemed to affect only children aged 36 months and effects varied across subpopulations. Such children of non-teenage mothers had fewer behavioural problems and greater social competence when living in SSLP areas, but the reverse was true for teenage mothers. Children of teenage mothers, like those who lived in workless or lone parent households, also scored lower on tested verbal ability (see table 4).
How do effects on children come about?
Results of the Sobel test indicated that the effects of SSLPs on children of non-teenage mothers were mediated by the effects of these programmes on parenting (P < 0.006 for child social competence, P < 0.006 for behavioural problems).
Do implementation features of SSLPs account for their effectiveness?
The lead agency correlated consistently with the effectiveness of programmes: SSLPs led by health services had better outcomes than programmes led by other agencies (table 5). Health led SSLPs resulted in greater involvement by fathers of children aged 9 months than programmes led by local authorities (P = 0.02) and other agencies (P = 0.05); fewer accidents for children aged 36 months than local authority led programmes (P = 0.009); more positive area ratings by mothers of children aged 9 months than local authority led programmes (P = 0.03); and more positive area ratings by mothers of children aged 36 months than programmes led by other agencies (P = 0.02). SSLPs that achieved greater reach also scored higher on supportive parenting by mothers of children aged 9 months (parameter estimate 0.006, 95% confidence interval 0.000 to 0.011, P = 0.03).
Imputed data mean scores and confidence intervals of measures that show significant differences as a function of Sure Start local programme lead agency. Values are differences between groups (95% confidence interval) unless otherwise stated
Discussion
Most families in socially deprived SSLP and comparison areas were disadvantaged and our results indicate that the small and limited effects of SSLPs varied with the degree of social deprivation. Children from relatively less socially deprived families (non-teenage mothers) benefited from living in SSLP communities, probably because of the beneficial effects of SSLPs on parenting. In contrast, children from relatively more socially deprived families (teenage mothers, lone parents, workless households) were adversely affected by living in SSLP areas. The recent American evaluation of Early Head Start, a home visiting and childcare programme (or both) for disadvantaged children under 4 years, reported similarly divergent results.2
Socially deprived families with greater personal resources may have been better able to take advantage of SSLP services and resources, which may have left those with fewer personal resources (such as young mothers and lone parents) with less access to services than would otherwise have been the case. Relatively more socially deprived parents may also find the extra attention of service providers in SSLP areas stressful and intrusive.
More children and families were affected beneficially than adversely, as teenage mothers formed a minority of the sample (14%), as did lone parent families (33%) and those living in households where nobody worked (38%). However, because the most socially deprived groups account disproportionately for many problems in society (such as school problems and crime), the apparent adverse effects of SSLPs might have greater consequences for society than the beneficial effects.
Health led SSLPs were more effective than other SSLPs, indicating that better access to birth records and better integration of health visitors as part of a ready made system of home visiting may facilitate the success of SSLPs. This may explain why health led programmes spent the money available to SSLPs more quickly on services than did other programmes.8
Because this evaluation was quasi-experimental, cross sectional, and evaluated the impact of a programme that had been in place for only a few years, the detected effects of SSLPs and the conclusions must be treated with caution. Ongoing follow-up of the 9 month old children should enhance understanding. Nevertheless, the study indicates that health services led SSLPs appear more effective than those led by other agencies, and that improving parenting is one of the mechanisms by which SSLPs promote child wellbeing.
What is already known on this topic
Large scale, area based programmes have been initiated by the current government in the United Kingdom to enhance the wellbeing of children aged 0-3 years and their families who live in disadvantaged communities
What this study adds
These programmes seem to have had limited and small effects: beneficial effects on the least socially deprived families and adverse effects on the most disadvantaged families
Programmes led by health services seem be more effective than programmes led by other agencies, probably because of better access to children and established health visitor networks
The national evaluation of Sure Start research team: Pam Meadows directed the collection and analysis of the cost data. Jane Tunstill directed the collection of implementation data. Martin Frost managed the geographical information system used to define SSLP areas, to enable community level data to be organised in terms of the small SSLP areas. Mog Ball and Angela Anning helped to collect and interpret the implementation data. Juliet Henderson and Katrina Wilkins trained and supervised the team of home visitors who collected the child and family data, and they also gathered such data.
Footnotes
-
contributions JB helped plan the national evaluation and interpret results, guided data collection, supervised data analysis, was the main writer of the manuscript, and is guarantor. EM helped plan the national evaluation, interpret results, write the manuscript, supervise data analysis, and guide data collection. JB helped plan the national evaluation, edit the manuscript, design and direct collection and analysis of community level data. AHL designed and oversaw the multilevel modelling used to analyse the data and helped to interpret results and edit the manuscript. HR analysed the data and helped interpret the findings and edit the manuscript.
-
Funding The Department for Education and Skills (DfES) funded the research, after competitive tender. DfES shaped the research design by ruling out a randomised clinical trial and designating SSLPs as serving entire communities, thereby necessitating a quasi-experimental, intention to treat design. The DfES's Sure Start unit and the unit's scientific advisory board reviewed all major research decisions (instrumentation, sampling, analysis) and helped interpret the findings, but those presented here represent the views of the authors. The government reviewed this report before submission, for accurate representation of findings only.
-
Ethical approval multi-regional ethics committee and Birkbeck's ethics committee.