Article Text
PDF
Abstract
Background There have been few evaluations of national area-based interventions. This study evaluated the early effects of Commmunities for Children (CfC) on children and their families and whether the effectiveness differed for more disadvantaged families.
Methods A quasi-experimental cohort study in socioeconomically disadvantaged communities in Australia. Mothers of children aged 2–3 years participated at wave 1; 1488 children in CfC communities and 714 children in comparison communities. Outcome measures included child health and development, family functioning and parenting, and services and community.
Results After controlling for background factors, there were beneficial effects associated with CfC. At wave 3, in CfC areas children had higher receptive vocabulary (mean difference (MD) 0.25, 95% CI −0.02 to 0.51; p=0.07), parents showed less harsh parenting (MD −0.14, 95% CI −0.30 to 0.02; p=0.08) and higher parenting self-efficacy (MD 0.11, 95% CI 0.00 to 0.21; p=0.04). Fewer children living in CfC sites were living in a jobless household (OR 0.56, 95% CI 0.32 to 0.95; p=0.03) but children's physical functioning (MD −0.26, 95% CI −0.53 to 0.00; p=0.05) was worse in CfC sites. For children living in households with mothers with low education there were reduced child injuries requiring medical treatment (MD −0.61, 95% CI −0.07 to −1.13; p=0.03) and increased receptive vocabulary (MD 0.57, 95% CI 0.06 to 1.08; p=0.03).
Conclusions CfC showed some benefits for child receptive vocabulary, parenting and reducing jobless households and two adverse effects. Children living in the most disadvantaged households also benefited.
- Child development
- child welfare
- early intervention (education)
- parenting
- quasi-experimental
- social class
Statistics from Altmetric.com
Communities for Children (CfC) is a large-scale area-based initiative designed to enhance the development of children in 45 disadvantaged community sites around Australia. The CfC initiative aimed to improve the coordination of services for children aged 0–5 years and their families, identify and provide services to address unmet needs, build community capacity to engage in service delivery and improve the community context in which children grow up.1
The initiative involved providing funding to a large non-government organisation in each area (the facilitating partner).1 The facilitating partner established committees that included other local service providers and community representatives to decide on the services required in communities and to allocate funding for these services to local providers. The local service providers then delivered these services.
CfC did not include any prescribed services. The type of the services delivered in each community was decided by the local committees based on a needs assessment.1 The services included home visiting, programmes on early learning and literacy, parenting and family support, child nutrition and community events. The facilitating partners were also funded to increase service coordination and cooperation between service providers. The explicit focus on service coordination and cooperation in communities was a novel aspect of the initiative.
While a number of countries have implemented area-based interventions designed to improve outcomes for children in disadvantaged areas, few have been rigorously evaluated.2 An exception in the UK is Sure Start Local Programmes (SSLP), which was an area-based intervention that targeted all children aged under 4 years and their families. Each SSLP had extensive local autonomy in terms of the services that were developed,3 a feature CfC shares with Sure Start. The evaluation of the early effects (short-run) of SSLP found mixed effects, with beneficial effects for some groups, but adverse effects for children from families with higher needs and experiencing greater disadvantage.3 Results from the second phase of the evaluation were more positive. Children in Sure Start local areas were more likely to be immunised, less likely to have accidents requiring treatment, and had significantly higher scores on measures of positive social behaviour and independence/self-regulation.4
The present study aims to evaluate the impact of CfC on children and their families by investigating differences between children and families living in 10 CfC sites and five comparison sites that were similar in location, size and socioeconomic status. A quasi-experimental, cohort study was used. Data were collected before the CfC initiative in CfC communities and comparison sites (wave 1), at the time of implementation (approximately 1 year after baseline, wave 2) and approximately 2 years after the baseline (wave 3) with statistical controls. Some outcome measures were only collected at wave 3, due to the age of the study child.
The impacts of CfC are estimated using the longitudinal data (wave 1 and wave 3) and wave 3 cross-sectional differences. After controlling for demographic characteristics there were no statistically significant differences in outcomes at baseline and also no evidence of differential attrition.
This paper addresses two questions. First, did child and family outcomes differ between CfC and comparison communities? Second, what are the effects of CfC for more disadvantaged children and their families? The study provides insights into the early evidence of the effectiveness of a national area-based initiative that emphasised service cooperation and coordination to enhance the life chances of disadvantaged children and their families. This evaluation has the added methodological rigour of longitudinal data; an improvement on the Sure Start evaluations, which were quasi-experimental, cross-sectional studies.
Methods
Design and participants
Ten of the total 45 CfC sites were selected. Sites were chosen by geographical area, start date of the initiative and the number of 2-year-old children.
The 10 CfC evaluation sites were selected across seven of the eight Australian states and territories and divided between major cities and regional areas. Sites were also only included if they were due to commence the initiative in the 12 months following wave 1. The five comparison communities were selected to be similar to CfC sites in location (the same states and territories as the CfC sites) and in socioeconomic status of the local area (as measured by the socioeconomic index for areas, CfC site mean 941.58, 95% CI 916.34 to 966.82, comparison site mean 970.11, 95% CI 920.15 to 1020.06; p=0.29). Of the five comparison sites, three were communities considered for CfC funding but not funded, and the other two were comparable areas to the CfC sites. Both CfC and comparison sites needed to have a sufficient number of 2-year-old children living in them.
As all 2-year-old children and their families living in CfC sites could potentially benefit from the intervention, an intention-to-treat design was used. The sampling unit for the study was 2-year-old children in 2005 in CfC and comparison sites. The sampling frame was obtained from administrative data on recipients of Australian government payments to families with children (family tax benefit (FTB) parts A and B). All lone parent families and most couple families are eligible to receive an FTB payment. For couple families, FTB payments are income tested and were not received by high-income families.
In wave 1, 3379 families were approached and 2202 families were interviewed, representing a 65% response rate (724 families refused and 453 families could not be contacted). The final sample represented 42% of the total population of eligible families with a 2-year-old child in the CfC and comparison sites. The waves 1 to 2 and waves 2 to 3 response rates were 91.5% and 90.3%, respectively. Data collection covered the periods June to August 2006, March to July 2007 and February 2008 to May 2008.
Procedures and measurement
The longitudinal study involved face-to-face interviews with the person in the family who knew most about the child (parent 1). In most cases parent 1 was the study child's mother. In wave 3, the study child was required to be present during the interview to undertake the direct assessments (height and weight and a language and vocabulary test). Interviewers did not know whether they were interviewing a family in a CfC or comparison site.
The research was approved by the Australian Institute of Family Studies Human Research Ethics Committee and ratified by the University of New South Wales Human Research Ethics Committee. All parents or guardians provided written informed consent to participate in the study and were free to withdraw from the research at any time.
Table 1 describes the outcome variables, their source and the waves they were collected. Table 2 lists child and family background variables that were covariates for CfC and comparison sites.
Outcome variables in a study of communities for children
Demographic characteristics of families in CfC and comparison sites at wave 1
Statistical analysis
Two approaches were used to estimate the impact of the CfC intervention. First, multilevel modelling was used to estimate the impact of CfC by comparing the difference between CfC and comparison sites in the outcome measures at wave 3 after taking account of demographic variables (see table 2). For jobless households several covariates were excluded, as these variables would have confounded the estimates (ie, household income, maternal labour force status and whether the father was present and working, present and not working or absent). We used xtreg in Stata for continuous outcomes and xtlogit for dichotomous outcomes. The multilevel models estimated take into account the clustering of the data in the calculation of standard errors. These analyses are very similar to those conducted in the Sure Start evaluation.3 4
Second, because baseline data were available for many outcomes, a second multilevel model was run that included baseline functioning. Given that CfC and comparison sites are matched on the socioeconomic index for areas, which comprises over 30 area-level variables, it was not necessary to include area characteristics as control variables.
We tested whether CfC affected families who were more disadvantaged than others by using an interaction between the CfC variable and the disadvantaged variable. The more disadvantaged families were defined by two variables present before the intervention: low maternal education (year 10 or less schooling to approximately 15 years of age; 17.5% of sample); and low income (parental income was $A485 per week or less; 16.0% of sample). Each interaction was tested in a separate model.
The validity of all methods of estimating the impact of the intervention rests upon the assumption that outcomes in CfC and comparison sites would have been the same in the absence of the CfC initiative (ie, the construction of a credible counterfactual). The validity of the counterfactual was established by testing for statistically significant differences in outcome variables between CfC and comparison sites at wave 1 and by testing for differential rates of sample attrition between waves 1 and 3.
There were no statistically significant differences in average outcomes between CfC and comparison sites pre-intervention (wave 1) after taking account of the demographic variables using ordinary least squares and logit regression with robust standard errors for clustering in the sites. When these demographic variables were not included, only two of the 14 outcomes variables were statistically significant (at p<0.05): children's physical health was better in comparison sites than in CfC sites, and the reverse was true for maternal mental health at wave 1. Of the eight demographic variables in table 1 there were two statistically significant differences between CfC and comparison sites—mothers were almost 1 year older in contrast sites and there was a greater percentage of children who were aboriginal or Torres Strait Islander in CfC sites. Differences between CfC and contrast sites on the demographic variables in table 2 were tested using χ2 and independent t tests that account for the clustering using the svy command in Stata. Means and percentages that were adjusted for demographic variables are shown in supplementary table A1, available online only.
There was no difference in sample attrition between CfC and comparison sites for key demographic variables and baseline child and family outcomes in CfC and comparison sites. This was tested using logistic regression (with robust standard errors) to model the extent to which wave 1 demographic characteristics and outcomes affect the probability of a family participating at wave 3.
The CfC initiative was evaluated approximately 12 months after implementation. Therefore any effects were likely to be small.
Results
Overall effects of the CfC intervention
Table 3 describes the estimated effects of the CfC initiative on the 19 outcome variables from multilevel models with demographic variables and multilevel models with demographic variables and the baseline as a control. The estimated effects of continuous outcomes are presented as a standardised effect size and dichotomous variables as OR. Four of the 19 variables were statistically significant and, when baseline functioning was controlled for, three were statistically significant. Parents living in CfC sites reported significantly lower levels of hostile or harsh parenting and higher levels of parental self-efficacy than parents in comparison sites, but not when baseline parenting was controlled. Children's receptive vocabulary achievement and verbal ability was also significantly higher in CfC sites (receptive vocabulary was not collected before wave 3 and therefore baseline functioning could not be controlled).
Standardised effect sizes and CI for the overall effects of the CfC intervention of measures that show significant differences between communities for children and comparison communities
The CfC intervention was estimated to reduce significantly the likelihood of a child living in a jobless household, based on both multilevel models. Further analysis revealed that this result was not explained by changes in a particular geographical site, such as a factory closure. The effect of CfC on joblessness was related to the intervention as a whole, rather than a site-specific outcome.
There were two negative findings from the evaluation. Children residing in CfC sites had significantly lower reported physical functioning than children in comparison sites even in the multilevel model that controlled for baseline functioning. Parent-rated general health was also significantly worse in CfC sites when baseline functioning was controlled.
Differential effects of the CfC intervention on specific subpopulations
Differential effects of CfC according to maternal education and parental income were tested using interactions and significant interactions are presented in tables 4 and 5. The effect sizes refer to the impact of CfC for two subpopulations: low education and parental income. In both situations, the subpopulation groups in CfC sites are compared with the subpopulation groups in the comparison sites.
Standardised effect sizes and CI for differential effects of the CfC intervention for low levels of maternal education of measures that show at least one significant difference between CfC and comparison communities
Standardised effect sizes and CI for differential effects of the CfC intervention for low-income households of measures that show at least one significant difference between CfC and comparison communities
Maternal education
For children living in households where their mothers have low education there was a large reduction in child injuries requiring medical attention, and this was also evident when baseline child injuries were controlled. The CfC intervention also had a large positive impact on their child's receptive vocabulary and verbal ability for children with mothers with low levels of education. Multilevel model estimates with and without the baseline as a control suggested that CfC had a positive effect on involvement in community service activity and reduced the rate of household joblessness for households with low education mothers. Support in raising children was also significantly higher in CfC sites, but not when baseline functioning was taken into account.
For children living in low and higher maternal education households, multilevel model estimates suggest that the CfC intervention had a negative impact on child physical functioning.
Low-income households
Low-income parents' involvement in their children's learning significantly improved in CfC sites even when controlling for baseline functioning (table 5). CfC also increased low-income mothers' ratings of the neighbourhood as a good place to bring up children when baseline functioning was taken into account. Finally, the estimates from both sets of multilevel models suggest that CfC had the effect of reducing the number of jobless households for those in low-income and not low-income households.
The multilevel models that did not control for baseline functioning suggest that children in low-income and those not in low-income households had significantly lower levels of physical functioning than children in CfC sites than in comparison sites. However, when baseline functioning was controlled for, only children in not low-income households in CfC sites had lower levels of physical functioning.
In relation to households who did not have a low income, there was a statistically significant effect of the CfC intervention on hostile or harsh parenting and parenting self-efficacy, but not when baseline functioning was controlled for.
Discussion
There is evidence that the CfC intervention has had a positive effect on a range of outcome measures (four of 19 outcomes), specifically increasing children's receptive vocabulary and verbal ability, reducing the number of jobless households and improving parenting practices (reduction in harsh or hostile parenting and greater parenting self-efficacy). Children in CfC sites had lower physical functioning than children in comparison sites based on parental reports, and parents' general health was also worse. There was also evidence that CfC had beneficial effects for both disadvantaged and more advantaged families. Children of mothers with year 10 education or less had higher receptive vocabulary and verbal ability, experienced a reduction in injuries requiring medical attention, and they were less likely to be living in a jobless household, while their mothers were more involved in community service activities in the community (four of 19 outcomes). Beneficial effects were also found for low-income households (three of 19 outcomes) with the quality of home learning environment higher and lower rates of jobless households and mothers reporting that the neighbourhood is a better place to bring up children. There was some evidence that mothers with low levels of education had worse health and mothers from lower income households reported worse general health in CfC sites.
The size of the CfC impacts on most outcomes was small, but can be considered positive relative to what was observed in the early phase of the UK Sure Start evaluation.3 The current results are also comparable in size to those found in the later impact evaluation of the Sure Start programme, in which 3-year-old children were exposed to mature SSLP throughout their entire lives.4 Reviews of the effectiveness of early childhood interventions have found that most studies reported effect sizes on parenting and child outcomes that were small to moderate.14 15 When comparing CfC and SSLP with other interventions, it is important to remember that the evaluations of these interventions measured effects on an entire population, rather than on programme participants, as is the case in the evaluation of many other interventions. The CfC service model was also assessed at a very early stage of implementation. Indeed, programmes and services in CfC communities were only likely to have been fully operational for a maximum of 2 years at wave 3, or from the time study children were approximately 3 years old.
The beneficial effects of the CfC initiative are unlikely to be solely due to a greater number of services. The other key components, better coordination of services and a focus on improving community ‘child-friendliness’ or social capital are likely to have played an important role. Positive change in relation to parental involvement in community activities, reduction in joblessness and increased support for parents, supports the idea that ‘community embeddedness’ may have an additional effect on children and families, and that the provision of increased services on their own would not have achieved this aim. For instance, greater community engagement and participation of mothers, and denser networks of relationships surrounding families in CfC sites, could have led to opportunities for employment.16
Across most of the groups examined (low and higher education and higher income), there was evidence that the intervention had a negative effect on child physical functioning. One possible explanation is that the CfC intervention may have increased the likelihood that parents came into contact with health professionals and others who may recognise undiagnosed health conditions, ailments or developmental issues for the child. Consequently, parents may have been more attuned to their children's needs and the child physical functioning scale may be sensitive to this as parents were asked how much of a problem their child had with gross motor skills and indicators of illness or a health condition. Other studies do suggest that parents from lower socioeconomic backgrounds have greater difficulties recognising health problems,17 and those with lower levels of income and education are more likely to believe that child health problems will improve on their own.18 CfC may thus have been successful in getting parents to recognise physical functioning problems in their young children, but there had been insufficient time for these functioning problems to be addressed once they were identified.
The optimal design for an evaluation of this kind would have been a randomised controlled trial. However, like the evaluation of Sure Start, this was not possible because of how the initiative was implemented. Yet, unlike the Sure Start evaluations, the CfC evaluation has the advantage of both pre-intervention baseline data and longitudinal follow-up data. Moreover, there was no evidence to suggest that families living in CfC and comparison sites were different at baseline, low rates of attrition at follow-up and no evidence of differential attrition based on baseline characteristics. Two multilevel models were estimated, one without baseline functioning and one including baseline outcome variables when they were collected with the first multilevel model similar to the analysis conducted in Sure Start. There were fewer statistically significant effects when baseline functioning was controlled, particularly with regard to parenting; however, for the high-risk groups almost all of the positive effects remained when baseline functioning was controlled. The results were also largely unchanged in analyses controlling for individual site differences and suggest that a single intervention site did not drive the results. The outcome measures used were valid and reliable, although with the exception of receptive vocabulary and verbal ability and children's body mass index they were primarily from parent self-report. The self-report measures of parenting were validated by interviewer observations of parenting.
The current findings suggest that the early stage of the implementation of the CfC service model has been modestly successful, particularly when seen in the light of early findings from Sure Start, and when one considers that children who participated in this study would have only received CfC services from approximately 3 years of age. As the CfC programmes continue to operate and improve in terms of access, outreach and effectiveness, there is the potential that there may be larger benefits for children and families exposed to the initiative.
What is already known on this subject
A number of countries have implemented area-based interventions to enhance the wellbeing of young children in disadvantaged communities.
The national evaluation of Sure Start was one of the few rigorous evaluations. The early evaluation found mixed early effects with adverse effects for children from families with higher needs and experiencing greater disadvantage.
A later evaluation of Sure Start with 3-year-old children who had grown up in these areas demonstrated several benefits for children and parents and no adverse effects for any subpopulations.
What this study adds
The current study suggests that the early stages of a national large-scale area-based intervention that has a focus on the coordination and integration of early childhood services can have beneficial effects for child receptive vocabulary, parenting and reducing jobless households.
Children living in the most disadvantaged households also benefited.
References
Supplementary materials
Web Only Data jech.2010.118133
Files in this Data Supplement:
Footnotes
Funding The Department for Families, Housing, and Community Services and Indigenous Affairs (FaHCSIA) provided funding after a competitive tender.
Competing interests None declared.
Patient consent Obtained.
Ethics approval This study was conducted with the approval of the Australian Institute of Family Studies Human Research Ethics Committee and ratified by the University of New South Wales Human Research Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.