Policies and interventions that tackle the root causes of poor health have recently been promoted by the UK and other EU governments as an important component of national strategies to improve health and reduce health inequalities.^1–6 The need to ground these strategies on evidence has also been highlighted.^2,7,8 Most recently the Wanless report stated that “every opportunity to generate evidence from current policy and practice needs to be realised”, and pointed to the value of systematic review methods in this regard.² National programmes of urban regeneration, or area based initiatives (ABIs), are one example of large scale investment tackling urban deprivation and the socioeconomic determinants of health, for example, employment, education, income, and housing; in the UK £11 billion has been spent on these initiatives over the past 20 years. The potential for this significant investment to lead to health improvement may seem obvious and indeed is currently used as a justification of such large scale investment (box 1).^1,9–11 However a systematic review of the impacts of ABI programmes on health or the socioeconomic determinants of health has not yet been done.

The dearth of data validating links between regeneration¹² or housing investment within regeneration programmes¹³ and subsequent health improvement has already been established in both systematic¹³ and non-systematic reviews.¹² But these reviews have relied largely on the results of formal research studies. Other relevant data and valuable lessons from previous policy interventions may remain hidden within government reports of policy evaluations. For example, large scale evaluations of ABIs are commissioned by government departments but their findings are rarely published in academic journals and the public health value of the evaluations’ findings seems to have been overlooked. In addition, evaluations of ABI programmes may be more likely to prioritise assessments of socioeconomic outcomes, over health outcomes. Impacts on socioeconomic outcomes have been recommended as a pragmatic and more immediate alternative to assessments of health impacts where health impact data are absent or difficult to obtain.⁴ A systematic examination of both the health and the socioeconomic impacts reported in national ABI evaluations may therefore allow exactly the type of synthesis called for by Wanless.²

WHAT IS THE EVIDENCE THAT NATIONAL PROGRAMMES OF URBAN REGENERATION (ABIS) IMPROVE HEALTH?

We carried out a synthesis of evaluations of national ABI programmes in the UK over 24 years (1980–2004) to examine the evidence that such major investments can have an impact on population health, the socioeconomic determinants of health, and health inequalities. We used existing systematic review methods for this synthesis.¹⁴

METHODS

Search strategy

We searched for the original reports of national evaluations of all the UK government’s nine national ABI programmes since 1980. (A brief description of each ABI programme’s activities, focus, years of implementation, and level of funding in the UK since 1980 is provided in table 1.) Eight electronic databases were searched (Bath Information and Data Services International Bibliography of the Social Sciences (BIDS IBSS, 1980–2004), COPAC (1980–2004), Health Management Information Consortium (HMIC, 1988–2004), IDOX Information Service (1980–2004), INSIDE (1980–2004), Ovid Medline (1980–2004), Urbadisc/Accompline (1980–2004), Web of Knowledge (1980–2004)). Because of the specific nature of the review topic, the databases were searched for any text containing the programme names or their commonly used abbreviations (for example, SRB for single regeneration budget). Relevant government departmental libraries were contacted for details of archived reports. Bibliographies of located documents and identified relevant web sites were also searched (http://www.odpm.gov.uk/,http://www.landecon.cam.ac.uk/urban/urgsrb.html). Authors of national ABI evaluations and an author’s (AK) own experience in this specialist field were drawn on to identify experts; identified experts were contacted to ask about further documentation available that may not have been identified by our search strategy.

Inclusion and exclusion criteria

Evaluations that reported achievements or impacts drawing on data from at least two target areas of a national ABI programme in the UK were included. Evaluations of single target areas or of projects within programme areas were excluded as the review aimed to assess the general impacts of a national programme; we assumed that single area evaluations may be less able than multi-area evaluations to account for local peculiarities that may influence outcomes. Annual reports and routine audits of programme activity were excluded unless they were presented as an evaluation or assessment of the programme’s achievements. Where it was clear that the document reported on a process or strategy for delivering urban regeneration rather than on the outcomes of ABI investment these documents were excluded (for example, the use of inter-agency partnership working in the delivery of ABI programmes). All methods of evaluation were included (for example, qualitative, quantitative case study, retrospective or prospective studies). Evaluations reporting only business and enterprise outcomes were not included.

Screening and selection

Titles of identified documents were screened by one reviewer to exclude obviously irrelevant or duplicate documents, after which titles and abstracts were screened independently by two reviewers. Where there was disagreement or uncertainty the full document was obtained and screened independently by two reviewers. Data extraction was carried out by RA and HT.

Data extraction

Impact data, defined as a measure of change in a given outcome over time, were extracted for health and selected socioeconomic outcomes. Health outcomes were any direct measure of health (quality of life, wellbeing, health, morbidity, mortality) or intermediate measure of health (for example, registration/use/satisfaction with local health services). Socioeconomic outcomes relevant to the determinants of health were defined as outcomes pertaining to housing, education, training, income, or employment. These included both direct measures (for example, household income, housing quality) and intermediate measures (receipt of welfare, satisfaction, with housing). Impacts on crime and neighbourhood outcomes (for example, satisfaction with local shops) were also extracted. Gross output data (reports of monies spent and investment activity, for example, number of dwellings built or improved, use of new sports centre) were not extracted.

Data synthesis

Impact data on direct health outcomes and direct measures of socioeconomic determinants of health were synthesised. Stakeholders’ and evaluators’ overall assessment of impacts on direct outcomes were not included in the synthesis. Intermediate outcomes were not included in the data synthesis.

RESULTS

A total of 896 references were identified of which 86 initially appeared relevant; 35 were included in the final review (fig 1). Sixteen evaluations used gross outputs exclusively to report programme achievement. Nineteen evaluations assessed health and social impacts and were included in the first stage of the review (see table w1 on line http://www.jech.com/supplemental).^15–34

Impact evaluations: methods, data quality and choice of outcome measures

Nine evaluations were carried out prospectively.^{23,24,26–28,30,31,34} All but two^20,26 of the impact evaluations used a case study approach, where the evaluators selected a few sites to represent the national programme. Detailed reporting of evaluation methods, data sources, and sample sizes was poor; in two evaluations some impacts were reported without any supporting data.^23,24 Furthermore, evaluators frequently reported that data on included outcomes were unavailable, resulting in non-reporting^17,23,24,29 or presentation of incomplete data in the final document.^16,19,28,34

Evaluations assessing impacts relied heavily on routine statistics collected by the UK government as well as stakeholders’ perceptions or the evaluators’ overall estimates of impacts. Six evaluations included a prospective survey of residents,^{23,24,26,28,32,34} one of which was a panel survey of the same residents at both time points.³² Ten of the 19 impact evaluations reported impacts on direct health or socioeconomic outcomes (table 2).^{18,22,25–28,30–32,34}

Data synthesis of direct impacts on health and socioeconomic status

Impacts on direct health and socioeconomic outcomes reported in the evaluations were self reported health status, mortality rates, employment (long term unemployment, employment, unemployment), household income, educational attainment, housing quality, and housing costs (rent) (table 2). A narrative synthesis of these impacts is presented below.

Impacts on self reported health and mortality rates

Impacts on self reported health or mortality rates were reported in three evaluations.^26,31,32 In one evaluation that surveyed the same residents before and after the programme, three of four measures of self reported health deteriorated, typically by ±3.8%.³² Two other evaluations reported overall improvements in mortality rates (standardised mortality rate 131 v 114²⁶ and 122 v 118,³¹ crude mortality rate −0.6%³¹) although standardised mortality rates increased in some case study areas in one of these evaluations.²⁶

Impacts on employment and unemployment

Employment measures were the most frequently included outcome measure and data were reported in nine evaluations.^{18,25–28,30–32,34} Improvements were reported in all but one evaluation.¹⁸ However, this simple tally of positive impacts conceals the specifics of type of outcome assessed, negative effects, and missing data.

Three evaluations reported improvements in employment (% working age in employment +6%²⁶ +4%³² and number of households with at least one person economically active +9%²⁷), but in one of these evaluations employment rate fell in two of the four case study areas²⁶ and in another evaluation there was no additional improvement when compared with the national trend in employment rates.³²

Eight evaluations reported impacts on unemployment outcomes; in six of these positive impacts were reported (% unemployed −1.3%,³¹ unemployment rate −3.8%,³⁴ −10.8%³⁰ numbers of unemployment claimants −32%,³⁴ −29.5%,²⁵ and % working age economically inactive −5.3%,²⁶ −4%,³²). In two evaluations overall impact on employment outcomes were negative (unemployment rate +0.3%,²⁸ % unemployed- +3.35%¹⁸). While improvements in unemployment measures were regularly reported, in two evaluations a mix of negative and positive impacts on unemployment measures were reported across case study areas^26,28 and in a further three evaluations the improvements reported were similar to national or regional trends over the same time period.^25,31,34

Impact on long term unemployment was reported in three evaluations (% of unemployed who have been unemployed >12 months,^28,30 and % of (unemployed + employed population) who have been unemployed >12 months³¹). In two evaluations of the SRB long term unemployment fell (−1.6%³¹ and −17%³⁰), although in one of these evaluations rates of long term unemployment increased relative to standardised English rates.³¹ In one evaluation of City Challenge an overall increase in long term unemployment was reported, although both increases and decreases were reported within individual case study areas (range −4.1% to +5.8%).²⁸

Impacts on educational attainment

Five evaluations (1988–1999) reported impacts on school achievement. Improvements in proportions of “pupils obtaining >4 GCSEs” or “>2 standard grades” (Scotland) were consistently reported in the four evaluations that included this outcome (mean impact +6.25%).^26,28,30,31 However, similar improvements in the proportion of “pupils obtaining >4 GCSEs” were also reported across England over this time and two evaluations reported little or no improvement when the findings were compared with national data.^30,31 Despite overall improvements, both negative and positive impacts on the proportion of respondents reporting “any member of household with CSE/GCSE/O level”³² or “school leavers with no GCSEs”²⁸ were reported across case study areas in two evaluations.

Impacts on household income

The number of households with incomes below £100 per week was assessed in two evaluations^26,32 and an overall improvement was reported. However, in one of these evaluations a range of negative and positive impacts on this outcome were reported across the four case study areas (−34% to +3%).²⁶

Impacts on housing quality and rent

The proportion of original residents living in improved housing after ABI investment was only reported in one evaluation (42.5%).²² Another evaluation assessed changes in housing costs; average social housing rent doubled over the period of investment, seven to eight years.²⁵

DISCUSSION

This review is a direct response to Wanless’s call to tap “every opportunity to generate evidence from current policy and practice”.² The use of conventional systematic review methods to synthesise impact data for both socioeconomic outcomes as well as health outcomes is a novel attempt to present evidence tailored to inform healthy public policy. The data synthesis suggests that previous ABIs may have small positive impacts (median size of positive impact reported ±5.5%, range 1.0% to 32.0%, for example, unemployment rate −3.8%,³⁴ households with income of less than £100 −4%³²) across a range of key socioeconomic determinants of health, although these impacts may mirror national trends. Small positive health impacts are also reported, but adverse health impacts remain a real possibility.

However, reports of impacts in the evaluations of ABIs are rare. In the UK, evaluation of ABI achievement has relied heavily on reports of gross outputs and monies spent (for example, number of new houses built), rather than reports of the actual impacts effected by the investment (for example, change in the proportion of residents living in poor quality housing). Even when an impact evaluation has been attempted this has often been unsuccessful. Evaluators frequently reported difficulties with data collection, preventing clear conclusions around impacts. This made identifying relevant evidence to synthesise for this review difficult. Common problems reported by evaluators included a lack of baseline data, lack of routine data that conform to target area boundaries, incomparable data between case study areas and a limited time scale in which to observe change in key outcomes.^{19,27–29,34–37} Data were often collected at an area level rather than an individual level, and panel surveys to assess impacts on the original residents before and after the ABI investment were used in only one evaluation.³² The potential, therefore, for this significant public investment to ameliorate deprivation and improve health and reduce inequalities remains unknown. Moreover, the possibility of adverse impacts of ABI investment on residents is also largely unknown.

Implications for evidence based healthy public policy

The dearth of health impact data to inform the development of healthy public policy has already been established across a number of policy areas.^13,38–40 In this review, the lack of socioeconomic impact data questions assumptions that ABI investment will reduce socioeconomic deprivation. In addition, the lack of data on both health impacts and socioeconomic impacts may undermine the rhetoric that links such investment to health gains and reductions in health inequalities.^{1,9–11,42,43} However, the absence of impact data does not provide grounds for inaction,^8,41 and it would be wrong to conclude that there is no research evidence to support hypothetical links between ABI investment and health impact. For example, in the UK both the Black Report and the Acheson Report presented data from a wealth of cross sectional and longitudinal studies to establish clear links between socioeconomic circumstances and poor health.^42,43

Improving the evidence base for healthy urban regeneration policy

Evaluations of ABIs need improving if they are to be used to inform the development of healthy public policy or to inform prospective health impact assessments of regeneration programmes. Detailed descriptions of variations in programme delivery and contextual factors that may account for variations in outcomes between areas are essential,⁴⁴ and are already available in most ABI evaluations. In addition, evaluation of complex programmes, like ABIs, requires clear theories or hypotheses specifying pathways through which health and social outcomes might improve.⁴⁵ To date these have been missing from both evaluations and programmes, even where health improvement is a key objective.

While health impact data remain on the public health “wish list”, “best available” evidence should be exploited.² This will typically entail rigorous syntheses of socioeconomic impact data as a proxy for health impact data (the approach taken by this review). The extreme heterogeneity of interventions, contexts, methods, and outcomes is an inherent characteristic of this type of systematic review and synthesis will be methodologically challenging as well as producing findings that may often draw attention to uncertainty rather than offering tangible policy recommendations; however, establishing what is not known is essential to good practice.⁴⁶ In the face of such uncertainty alternative sources of data can also provide evidence to direct policy and practice. Systematic reviews of cross sectional research evidence may help prioritise interventions and develop research informed theories for possible health impacts of policies which can then be tested through evaluation.

CONCLUSION

Despite significant public investment in national ABI programmes there is still little evidence to demonstrate the impacts on socioeconomic or health outcomes. Where impacts have been assessed, a small overall positive impact is suggested though adverse impacts are also possible. The few impacts reported rarely related to the original residents of target areas, thus the potential for ABI investment to improve the health or socioeconomic status of people or impact on inequalities remains uncertain.

Future evaluations need to incorporate clear theories of change informed by existing research evidence. In addition, an assessment of the actual impacts on original residents of target areas is required if the potential of such programmes to improve health and reduce health inequalities is to be confirmed. In the meantime, evidence syntheses that exploit best available data may be the best way to develop healthy public policy which is evidence informed.

CONTRIBUTORS

HT and RA were the principal reviewers, carrying out searches, screening, selection, and data extraction. HT and MP prepared the data synthesis. All authors contributed to planning the review and writing the paper. HT is the guarantor for this paper.