METHODS

Safe Living Program

The Safe Living Program aimed to increase public awareness of prevention measures, reduce injury hazards, and reduce the incidence and severity of injuries in the Shire of Bulla. The Shire of Bulla was responsible for program development, employment of program staff, program management, and budget administration.¹⁴ Working parties, comprising individual community members and representatives from organisations within and external to the community, contributed to specific interventions. Program strategies targeted injury reduction in multiple settings, particularly home, school, sports and road, with a focus on high risk groups, such as children and older persons, in these environments.^14–16 Timelines for selected components of the Safe Living Program are shown in fig 1.

Subjects and measures

The evaluation of the Safe Living Program (Shire of Bulla population 32 257 in 1990) was a quasiexperimental study employing a comparison community design.¹³ The selected comparison community (Shire of Melton population 33 592 in 1990) was matched to the intervention community on relevant demographic variables. Both shires were within the then relatively small catchment of the Victorian Injury Surveillance System (VISS) for emergency department childhood injury presentations.¹⁷

The Safe Living Program evaluation encompassed process (fidelity of implementation as planned), impact (effects of program on injury risk factors), and outcome (effects of program on injury occurrence). The evaluation was formative with regular program and community feedback. Measures included program reach, community involvement, working party participation, program publicity, receipt and use of the home safety package, education and training course participation, safety product outlets, bicycle helmet wearing rates, implementation of footpath cycling legislation,¹⁶ and safety audits of school playground equipment and arterial roads. Data for many of these measures were collected from shire records and by using random sample pre-intervention (1990), interim (1993) and post-intervention (1996) household telephone surveys. The range for response rates for these three telephone surveys was 50%–62%, each involving a representative sample of over 2% of both populations.

Program outcome was evaluated by comparing injury deaths, hospitalisations, emergency department presentations, and self reports.¹³ Sources of injury data included: the Victorian Coroner's Facilitation System (all reported injury deaths to 1994/95); the Victorian Inpatient Minimum Database (VIMD) (all public hospital admissions for Victoria to 1996)¹⁸ and the VISS database (emergency department injury presentations) for children under 15 years.^17,19 The VISS collection (five major Victorian public hospitals) was replaced from 1995 by the all age Victorian Emergency Minimum Dataset (26 hospitals encompassing 80% of all Victorian public hospitals) with data available for 1996. Injury readmissions within 30 days to the same hospital, and injury caused by adverse effects of medical treatment were excluded. Interhospital transfers (5.6%) were not excluded because of technical difficulties.

Analyses

Data were analysed using SPSS for Windows and Excel.^20,21 χ² Tests and differences in proportions were used to test significance of pre-intervention and post-intervention differences. Injury rates for the two shires were age standardised by the direct method using Australian census (1986, 1990) population projections for 1992–96 assuming linear growth between 1986 and 1991. Time trend models of injury in the shires were derived using injury frequency as the dependent variable in GLIM (generalised linear models).^13,22 Sensitivity analyses were conducted for a range of comparative hospital admission rates.

RESULTS

Program implementation

The Safe Living Program implemented 113 activities in the first three years (box 3).^14,23 Some activities were short term while others were implemented over a longer period or repeated intermittently.

Information about the Safe Living Program appeared once a week on average in the local newspaper throughout 1991–96. Fifty issues of the Safe Living Program newsletter were produced and sent to key stakeholders. A total of 345 persons participated in volunteer working parties.

Program reach of interventions ranged from <1% to 27%. The 1996 telephone survey found higher rates in the Shire of Bulla compared with the Shire of Melton for awareness of the program (47% v 3%) and receipt of the Safe Living News (17% v 3%). The two home safety packages delivered to all Bulla households in 1991 and 1996 were reportedly received by 27.5% and 9.3% of households, respectively. In Bulla, 37.4% of residents recalled safety displays at fairs compared with 26.6% of Melton residents. Pre-existing and widely available preschool and primary school traffic safety education programs were promoted. Altogether 75% of preschools and 86% of primary schools used the program in the Shire of Bulla in 1991 and 1993, compared with 50% and 56% respectively in the Shire of Melton.

Program impact

The Safe Living Program impact evaluation showed effects on risk factors associated with road and car occupant safety for children, bicycle safety, school safety, and safety training (table 1). Attempts to measure sales of safety products other than those listed in table 1 were generally unsuccessful.

Five per cent of households reported making safety changes as a result of receiving the home safety package in 1991 compared with 0.3% in 1996. Changes in other injury risk factors (installation of smoke detectors, first aid courses) were similar to those observed in the comparison community. No effect on the Bulla community could be identified related to high profile safety display homes which incorporated many safety design features.

Program outcomes

No significant changes occurred in rates of deaths, hospitalisations (fig 2), duration of hospitalisation (fig 3), or emergency department presentations attributable to injuries. Subanalyses by age identified no significant differences between the Shires of Bulla and Melton. Regression modeling for cause of injury hospitalisations found significant reductions for the Shire of Bulla compared with the Shire of Melton for motor vehicle occupant injury only.

Household report of injury in the Shire of Bulla declined from 19% to the same level as Melton (13%) by the interim survey in 1993. There was no difference in reported rates (both 13%) in the 1996 post intervention survey. Most self reported injuries were minor, requiring no medical treatment.

DISCUSSION

For the period 1991–96, the Safe Living Program was unable to replicate the reduction in injury rates reported in a Swedish community based study.⁹ Although program reach was modest, successful initiatives were demonstrated in specific areas where evidence based recommendations were made directly to both the shire and schools (road and playground equipment audits, footpath cycling law). However, many results were mixed even where promotional and educational initiatives directly targeted the Bulla population.

Evaluation method

The evaluation was complicated by changes to geographic boundaries and the restructuring of both local government¹⁵ and the VISS. Interpolated census data may have added further uncertainty. Budget restrictions limited process evaluation to selected measures.

The statewide change in funding structure for Victorian hospitals in July 1993²⁴ was associated with a rapid increase in Victorian hospitalisations in the following year, possibly distorting injury rates and masking small rate changes. These effects were not uniform by age or hospital so could not readily be included in modelling analyses. Sensitivity analyses for hospitalisation rates indicated that if Melton rates remained static, Bulla would have to experience at least a 25% reduction over six years to be statistically significant. Since the numbers of deaths were small, meaningful comparisons of frequencies or manner of death between the shires were not possible. Similar data quality issues have been reported for some of the controlled studies of community based injury prevention programs elsewhere.^6,9

The comparison community had similar demographics and socioeconomic status to Bulla and was also located within the catchment of the VISS, a unique advantage for identifying childhood injury presentation to emergency departments. Despite their proximity, there was little evidence of program contamination in the Shire of Melton as measured by program awareness and receipt of Safe Living News. Nevertheless, general injury prevention initiatives across the state of Victoria such as safety displays at fairs, and school traffic safety education programs, showed effects in both shires. However, the greater effects in the Shire of Bulla were associated with enhanced promotion of these initiatives through the Safe Living Program.

More appropriate designs for community based injury prevention programs using cluster randomised trials^6,7,25 would provide robust evidence for effective strategies. Logistical problems and substantial costs are major barriers to this approach.³ Indeed most community based injury prevention programs do not employ even a comparison community⁸ and few of the current 61 World Health Organisation Designated Safe Communities²⁶ have published evaluations in the international literature.

Despite these methodological issues, data from a range of sources indicated a limited effect of the Safe Living Program, with results lower than the range suggested for an acceptable positive effect.²⁷ As the rate of injury presentation per person was low, the binomial model provided as good a fit to the data as the Poisson model. Any substantial findings would be unlikely to differ greatly regardless of analytical method employed.^5,27

IMPLICATIONS FOR PREVENTION PROGRAMS

Our results and those of others, indicate that a high level of program reach and uptake of targeted and sustained efficacious countermeasures is essential for achieving program objectives.^6,7 Community based injury prevention may work better in cohesive, homogenous, stable, and isolated communities with good organisational involvement.⁹ The “snowball” effect gained by training professionals and community volunteers to implement the program was a vital tool in the Swedish programs. This suggests that insufficient recruitment and training for implementation may be a factor in the low effect achieved for the Safe Living Program.

Institutionalisation involving incorporation into the routine function of a responsible organisation such as local government may be an important component in maintaining community based injury prevention programs. The long term effects of such institutionalisation should be evaluated, particularly for program sustainability and injury prevention outcomes.

Major lessons

Comprehensive standardised documentation of intervention strategies, research methods, and results is essential for program evaluation. Systematic record keeping by organisations responsible for intervention programs is necessary not only for quality assurance management but also for evaluation. In addition to data on severe injury, the availability of both emergency department and, where possible, reliable measures of lower severity injuries seen in primary care, would enhance the scope of evaluation.^28,29 Improved measures of hospitalisations beyond number of bed days would help clarify issues related to injury severity in evaluation studies, though accurate coding may be costly. As reporting on every available intervention measures is expensive and inefficient, there is a need to focus on those risk factor measures which are most strongly associated with injury reduction.

In addition, telephone surveys may be decreasing in value due to the high prevalence of answering machines and the increasing proportion of households declining participation. Key informant interviews may provide a useful contribution to explanations of successful and less successful program effects.

Comparison communities may be non-equivalent in a number of ways and differences may evolve over time (for example baseline injury rates, methods of injury data collection, size, and nature of community). To the extent possible, alternative data sources should be established as a contingency for unanticipated changes to data systems over time. Importantly, intervention populations with high baseline injury rates are likely to be more susceptible to reductions in rates due to regression to the mean. In future community based injury prevention programs, consideration should be given to: fewer, well developed and more targeted, efficacious interventions; greater community reach; achieving organisational change; and enhanced evaluation design, including benefit/cost analyses.^1–3,7 In particular, larger communities or a number of randomised communities are required to demonstrate significant reductions in serious (hospitalised) injury rates.⁶