Background Economic evaluation of public policies has been advocated but rarely performed. Studies from a systematic review of the health impacts of housing improvement included data on costs and some economic analysis. Examination of these data provides an opportunity to explore the difficulties and the potential for economic evaluation of housing.
Methods Data were extracted from all studies included in the systematic review of housing improvement which had reported costs and economic analysis (n=29/45). The reported data were assessed for their suitability to economic evaluation. Where an economic analysis was reported the analysis was described according to pre-set definitions of various types of economic analysis used in the field of health economics.
Results 25 studies reported cost data on the intervention and/or benefits to the recipients. Of these, 11 studies reported data which was considered amenable to economic evaluation. A further four studies reported conducting an economic evaluation. Three of these studies presented a hybrid ‘balance sheet’ approach and indicated a net economic benefit associated with the intervention. One cost-effectiveness evaluation was identified but the data were unclearly reported; the cost-effectiveness plane suggested that the intervention was more costly and less effective than the status quo.
Conclusions Future studies planning an economic evaluation need to (i) make best use of available data and (ii) ensure that all relevant data are collected. To facilitate this, economic evaluations should be planned alongside the intervention with input from health economists from the outset of the study. When undertaken appropriately, economic evaluation provides the potential to make significant contributions to housing policy.
Statistics from Altmetric.com
Economic evaluation of health technologies (drugs, devices, etc) has become widespread across the world, with government agencies (eg, National Institute of Clinical Excellence (NICE) in England and Wales) employing these techniques on a routine basis to make decisions about which healthcare to fund.1–4 But there has been very little progress in economic evaluation for public health and in particular within public policy.5–8 In 2005, the remit for NICE was expanded to include public health, reflecting a growing desire to broaden the scope of such evaluations and address issues of resource allocation across all sectors impacting on health.9 ,10
Provision of acceptable housing conditions may be regarded as a cornerstone of healthy public policy, representing a major public investment with the potential to improve health and contribute wider public health strategies to improve population health and reduce health inequalities.11–13 In 2009, a systematic review of the health impact of housing interventions, including studies from around the world, concluded that housing improvements can lead to health improvements.14 This was especially the case for warmth improvements targeted at those with poor health, living in poor housing. Some of the studies in the 2009 systematic review included reports on the costs associated with the interventions and a small number reported having undertaken economic evaluations.14 Given the growing desire of policy makers to demonstrate value for money from interventions, we undertook a further review of these studies to identify and extract data which could be used to inform estimates of the relative costs and benefits of housing improvement, and illustrate the challenges of economic evaluation in this field.
This paper presents the results of this review, providing details of the cost and economic analyses reported alongside housing intervention studies and accompanying health impacts. The paper uses these data to examine the economics of housing investment and also to reflect on the current state of health economic analysis in housing. As an example of health economic evaluation of a substantial public investment and policy area, the lessons may have a wider methodological relevance to topics of interest to healthy public policy, such as welfare reforms or transport initiatives.
Prior to detailing the process undertaken for this systematic review of economic data and analysis, we detail the definitions used to distinguish between differing types of cost studies and economic evaluations (table 1). These definitions are routinely applied in the field of health economics and have been proposed for the economic appraisal of public health interventions.4 ,5 ,7 ,15
Table 1 details three forms of cost study which are common within the literature, namely cost-offset, cost-minimisation and cost-consequence. Cost-offset studies simply detail the costs of the intervention alongside the cost savings achievable (eg, days in hospital averted). There is no measurement of health outcome. Cost-minimisation studies compare the costs (including any cost savings) for an intervention with the costs of the status quo under the assumption that the outcomes are equivalent between the two. Cost-consequence studies present the costs (including any cost savings) associated with an intervention and the status quo alongside a list of the various possible outcomes achieved. There is no attempt to identify or value the collective outcomes within a single metric. As such, while cost consequence studies provide a useful descriptive summary, and first step towards a full economic evaluation, they cannot be used to determine value for money or identify priority interventions.
These approaches are not formally economic evaluation techniques because they do not allow for a formal comparative analysis in terms of costs and outcomes. In a situation where policy involves the provision of a specific amenity (eg, a new heating/insulation system) then a comparison of costs is all that is required to establish the least costly way to provide the desired amenity. However, this assumes that provision of the specific amenity is, by some definition, a good thing (eg, it improves health outcomes).
Where budgets are constrained there is an unavoidable opportunity cost of undertaking any policy, as funds spent on one intervention will limit the funds available for other interventions. The aim of economic evaluation is to assist policy makers to identify interventions/policies which represent good value for money. The value for money associated with a new intervention is determined by comparing the additional costs required and additional outcomes achieved, with the status quo or current intervention. Where an intervention/policy provides greater outcomes at lower costs, it is said to ‘dominate’ the current intervention/policy and a decision about adopting the new intervention/policy is straightforward. However, where an intervention/policy provides greater outcomes at greater cost, a decision must be made about whether to spend these additional resources to achieve these additional outcomes. Where outcomes of different interventions/policies are valued using common metrics (see table 1), the value for money associated with them can be directly compared to indicate where these outcomes can be achieved at the lowest price. These interventions represent the best value for money compared with the alternatives available and, where resources are scarce, could be considered ‘economically worthwhile’.
There are three main forms of economic evaluation: cost-effectiveness analysis, cost-utility analysis and cost-benefit analysis (table 1). Each method involves a ‘comparative analysis of alternative courses of action in terms of both their costs and consequences’.4 With each method costs are measured in monetary units, while the measurement and valuation of outcomes differs. As such, the methods are traditionally classified by outcome.15
Cost-effectiveness analysis typically involves measuring a specific, one-dimensional, health or clinical outcome, for example asthma attacks averted. Cost-utility analysis is a special type of cost-effectiveness where multidimensional health outcomes are reduced to a single dimension reflecting individuals’ preferences for the diverse health outcomes. The most commonly used outcome in cost-utility analysis is the quality adjusted life year (QALY). For both cost-effectiveness and cost-utility studies, value for money is identified using a measure of the additional cost per additional outcome ratio (eg, an incremental cost/QALY ratio) and comparing that to an external threshold or to the ratio achieved by alternative policies. In contrast, cost-benefit analysis involves the measurement and valuation of all outcomes of interest in monetary terms. Here the value for money is identified by positive net economic benefit associated with the interventions (ie, the monetary value of the outcomes exceeds the net costs of the intervention less any costs savings achieved elsewhere). This Paretian definition of cost-benefit analysis is the established, standard definition used within health economics. It allows a broad spectrum of outcomes (all those of importance to the individual) to be included within the metric, but requires the, often complex, valuation of outcomes in monetary terms.
Data extraction and analysis of economic data in housing intervention studies
All housing intervention studies and their associated papers included in the 2009 systematic review of the health impacts of housing improvement14 were examined for reports of costs and economic analyses. Details of the scope of the review (inclusion and exclusion criteria), and evidence appraisal are available in the 2009 publication along with the findings of the review.14 Forty-five medical and social science databases, as well as websites and grey literature were searched to identify studies of housing improvement which assessed change in any health outcome. A separate search for economic studies was not undertaken but economic studies which included health outcomes following housing improvement would have been identified in the broad search. Two independent reviewers screened 27 082 citations to select the included studies. All available data on costs and, where available, details of any economic analysis were extracted by one reviewer (CM) and checked by a second reviewer (EF or HT). The cost data were tabulated alongside a summary of reported heath impacts and an indication of overall study quality as used in the original review (A=minimal bias, B=some bias, C=considerable bias).14 Where a study reported plans to undertake economic analysis, the authors were contacted for an update on progress and available data, or reasons for not completing the economic analysis.
Studies were allocated into two groups based on the type of data reported. Studies which only presented cost data (table 2) were further examined for the potential to have conducted an economic evaluation, that is, presence of a suitable health outcome which could be linked to cost and compared to an alternative (table 1). Studies which reported having undertaken an economic evaluation (table 3) were examined to determine the precise form of that analysis (table 1).
Forty-five studies were identified in the original review. The study designs varied and included five randomised controlled trials (RCTs), and 23 non-randomised controlled studies. The better quality RCTs and controlled studies were used to draw conclusions about effectiveness. The health outcomes reported included validated measures, for example SF-36, and self-reported measures, and covered four main domains: general health; mental health; respiratory health; and other/illness and symptoms. Twenty-nine studies reported costs or an economic analysis; 25 of these studies16–41 presented only basic cost data (table 2) while four studies reported having undertaken an economic analysis (table 3).42–46
Studies which present cost data without economic evaluation
Details of the 25 studies reporting cost data without economic analysis are presented in table 2 (see supplementary table 1 for full details of interventions, economic data and health impacts). Eleven of the studies examined warmth and energy efficiency interventions,16–26 eight examined rehousing or retrofitting,27–35 two focused on pre-1965 rehousing from slums40 ,41 and four focused on provision of basic housing needs.36–39 Over 75% of the studies (n=19) were from the UK,17–35 ,41 with one study each from New Zealand,16 the USA,40 Mexico,36 Philippines,37 Pakistan38 and Malawi.39 The study design and methodological quality of the studies varied, as assessed by the original systematic review criteria. These studies reported a diverse range of health outcomes, and the reported impacts suggest either improvement or no change in health status following housing improvement during the study period (see supplementary table 1). One study reported a deteriorated health status following housing improvement.22
Six studies24 ,26 ,33 ,40 ,47 ,48 presented data on both the cost of providing the intervention and other costs, including those incurred by the recipient, nine16 ,23 ,28 ,30 ,34–36 ,39 ,49 presented only costs relating to the provision of the intervention and ten studies18–19 ,25 ,27 ,29 ,31 ,37 ,38 ,41 presented only other costs, primarily to the recipient. The cost measures used varied and require different interpretations. For example, some studies reported intervention costs of a major housing led regeneration programme for an area beyond the included study population, while other studies reported average costs of the specific housing intervention per household. Measures used for recipient costs did not always report direct data and some studies used residents’ assessments of these changes, for example changes in fuel consumption or bills, making it difficult to interpret. The measures of other costs to recipients were reported, nine studies reported changes in fuel use or costs,17 ,19 ,25–27 ,31 ,38 ,47 ,48 six changes in rent,17 ,26 ,27 ,31 ,40 ,41 three changes in healthcare spending,24 ,33 ,38 two changes in household costs17 ,29 and one changes in income.37 One study compared the ability to manage financially between the intervention and control group.18 Drawing on the most commonly reported recipient costs, all six studies which reported numerical data for changes in fuel costs reported a reduction following energy efficiency improvements.17 ,19 ,25 ,26 ,31 ,47 Each of the five studies which reported numerical data for changes in rent reported increased rent following housing improvement.26 ,27 ,31 ,40 ,41
Of the 25 studies which presented only cost data, 11 (8 from the UK) were assessed to have sufficient data for an economic evaluation. Analysis linking cost data to health outcomes could have been conducted to present a cost-effectiveness, cost-utility or cost-benefit analysis (table 2).16 ,20–23 ,26 ,28 ,30 ,35 ,36 ,39
Two studies, both from the UK, reported plans to conduct an economic evaluation but this was either not conducted or not publicly available at the time of this review. Neither of these studies was among the 11 reporting data amenable to economic evaluation; Eick et al24 did not have data for a comparator group and Caldwell et al25 did not have data on intervention costs. Eick et al24 presented data on medical costs before and after the intervention and reported plans for a cost-benefit analysis. Caldwell et al (2001) originally planned to examine issues of cost-effectiveness but this was not undertaken due to poor quality data.
Studies reporting an economic evaluation
Four studies reported undertaking an economic evaluation (table 3). These studies all involved warmth and energy efficiency interventions since 2000 or later. Two studies were from the UK44–46 and two from New Zealand.42 ,43 ,50 The methodological study quality, with respect to assessment of health impacts in the original systematic review, varied; two studies were assessed to have a minimal risk of bias (grade A),42 ,45 ,50 one study was assessed to have some risk of bias (grade B)44 ,46 and one study was assessed to have considerable potential for bias (grade C).43
Three of these studies reported undertaking a cost-benefit analysis. These studies fall short of full cost-benefit analysis, as defined above, as they did not include a monetary valuation of all important outcomes. They are more accurately described as having presented a ‘balance sheet’ type approach. This hybrid approach involves identifying and listing the costs and benefits associated with an intervention or policy change, in much the same way as in a cost-consequence study.7 ,51 ,52 The costs and some of the benefits are then measured in monetary units where appropriate values are either available or can be postulated, otherwise they are simply listed in their natural units (eg, time). For example, a UK study44 presented cost data for medical treatments, prescriptions and fuel use and imputed a monetary benefit of reduced school absences. A study from New Zealand42 ,50 presented a benefit-cost ratio based on the cost of the intervention, changes in the costs of medical service use, and the economic value imputed for reduced CO2 emissions and the reduction in lost days of school and work. Another study from New Zealand43 presented a feasibility study for a cost-benefit analysis with a direct benefit to cost ratio, however the authors provided no details of the methods used or the outcomes measured. The findings from all three studies suggested net economic benefits associated with the interventions based on the outcomes measured in monetary terms (ie, the monetary value imputed for these outcomes exceeds the net costs of the intervention less any costs savings achieved elsewhere). In addition, both New Zealand studies indicate small, but positive, benefits to cost ratios associated with the intervention.42 ,43 ,50
One study reported undertaking a cost-effectiveness analysis which met the criteria employed by this review.45 This UK study used health indicator data (SF-36) to conduct a cost-effectiveness analysis. Although data were not presented numerically (it was narratively reported that the results were not statistically significant), a bootstrapped cost-effectiveness estimate presented graphically suggested that the intervention was dominated. This means that the intervention (improved housing) was more costly and less effective than the status quo.
Two further economic studies of housing interventions were identified when contacting study authors about completion of ongoing housing studies which might contribute to an update of the original housing review. These studies were published recently and were not included in the original systematic review.53 ,54 Edwards et al53 report an economic evaluation, as defined here, presenting an estimate of the additional cost per point improvement on the PedsQL asthma specific scale. Grimes et al54 present a ‘balance sheet’ approach with a calculation of net economic benefit.
In addition, we are aware of the economic evaluation undertaken alongside the Scottish Housing and Regeneration Project (submitted to JECH Lawson, Kearns, Petticrew, Fenwick, Investing in health: is social housing value for money? A cost-utility analysis.); however at the point at which our review was undertaken, results for this analysis were not available.
Studies investigating the health impacts of housing improvement have frequently provided some details on costs or economic analysis (n=29/45).14 However, the majority of these (n=25/29) present data on intervention and/or recipient costs only and, despite sufficient data, opportunities to conduct economic analysis have been missed. Where studies report conducting economic evaluations, the majority of the reported analyses would be more accurately described as a ‘balance sheet’ approach.
Findings from the studies which report costs only, suggest that fuel costs may reduce following provision of warmth and energy efficiency improvements, and that rents may increase following housing improvement. These findings need careful interpretation. Changes in fuel costs are largely dictated by the unit cost of fuel and may not be directly linked to changes in fuel use or levels of warmth. In addition, changes in housing costs to recipients, including rent and fuel, may be mediated by welfare provision and changes in the individual's eligibility for welfare benefits such as housing benefit. As such, neither of these changes can necessarily be taken to indicate an improvement, or not, for the recipient of the intervention.
The three studies which presented a ‘balance-sheet’ approach reported a positive net economic benefit following the intervention, based on the outcomes valued in monetary terms.42–44 ,50 One cost-effectiveness study45 reported that the intervention was more costly and less effective, in terms of SF-36 score, than the status quo,45 indicating that the intervention was not cost-effective. This may reflect the fact that mental and physical health (measured by SF-36) deteriorated following housing improvement; or, and perhaps more likely, that the disruption during housing upgrading led to deterioration in health outcomes and the relatively short period of follow-up (maximum of two years) failed to capture the longer term impacts of the intervention.
The absence of long term health impacts limits the potential for economic analysis. The longest follow-up in this group of studies was 3.5 years after the intervention (range 1 month to 3.5 years). Expectations that health impacts will be observed in this short timescale may be naive and it may be more realistic to hypothesise that the potential for health benefits could be many years after the intervention, perhaps only in the next generation of residents. A full economic evaluation should ideally consider the impacts over the lifetime of the intervention. However, attributing longer term impacts to a historical intervention, even in large scale datasets with minimal attrition, introduces an additional level of confounding. Uncertainty due to immeasurable confounding is an important issue even for short term studies. As such, data on longer term health impacts may be useful but requires careful interpretation.55
The near absence of economic evaluation of housing improvements cannot solely be explained by difficulties in collecting suitable data. Over 40% of the studies which presented cost data alone, had sufficient data to conduct an economic evaluation, but had not done so. This comparative data on costs and health outcome for the intervention and status quo could have been presented to provide a measure of value for money of the intervention, for example in terms of the additional cost per unit of effectiveness achieved. The specific form of the evaluation would depend on the measure of health outcome collected with data such as SF-36, EQ-5D, etc used to determine QALYs for a cost-utility analysis, monetary values of health outcome used within a cost-benefit analysis and other measures of health/clinical outcome used within a cost-effectiveness analysis (see table 1). Where economic analysis was conducted, most (n=3/4) of the studies claimed to have undertaken a cost-benefit analysis but had in fact presented a ‘balance sheet’ approach. This hybrid approach can be helpful for policy-makers by identifying the costs and outcomes associated with a policy/intervention and who bears/receives these impacts.7 ,51
In each of these three studies, the authors calculated the net economic benefits associated with the interventions of interest, as required to establish value for money in a full cost-benefit analysis. However, none of these studies included changes to health outcomes within the benefits assessment in their calculation, despite all three collecting data on health outcomes. As such, none of these studies provides a full monetary assessment of the benefits associated with the interventions of interest. Instead the economic benefits calculation was restricted to a monetary valuation for increased school and/or work attendance and reductions in CO2 emissions. All three included changes in health service utilisation within the calculation, although these cost changes were frequently misreported as benefits.
These results suggest three important factors. First, the importance of and need for collecting data over a reasonable period of follow-up to allow detection of long term health improvements. Second, the importance of employing wider perspectives through inclusion of costs and savings in other sectors (eg, education or the environment) to give greater potential to show cost offsets and/or cost-effectiveness of housing interventions. Third, and perhaps most importantly, a lack of familiarity with the techniques of economic evaluation such that studies with relevant data often fail to make best use of it, while other studies fail to collect relevant data or misrepresent analyses that are undertaken.
Conclusions and lessons for the future
Rigorous economic evaluation of public health interventions, including those in housing, is seen as a priority. Future studies planning an economic evaluation need to make best use of all available data as well as ensuring that all relevant data are collected. To facilitate this, economic evaluations should be planned alongside the intervention with health economists included from the outset. When undertaken appropriately, economic evaluation provides the potential to make a significant contribution to healthy housing policy.
What is already known on this subject
Economic evaluation assessing the costs and health benefits of healthy public policy interventions has been advocated but has rarely been undertaken.
Housing investment represents a substantial policy investment area which is considered to have potential to contribute to a wider public health strategy to improve population health and reduce health inequalities.
Some studies assessing the health impact of housing improvement have reported some data on costs and economic analysis.
Examination of the nature of cost data and economic analysis reported in these studies may help to explain why economic evaluation has rarely been conducted and provide tangible examples of the potential to develop economic evaluation of healthy public policy investment.
What this study adds
Despite availability of cost data, opportunities to conduct economic evaluations of housing interventions have often been missed.
A small number of studies have reported ‘economic evaluation’ but the term has often been misappropriated.
Future studies planning an economic evaluation should ensure that it is planned alongside the intervention to ensure that all relevant data are collected and available data are fully utilised.
Studies should include a health economist from the outset.
Increasingly policy-makers, seeking to spend money from limited budgets, want evidence of value for money associated with changes in public policy.
Appropriately conducted economic evaluations have the potential to identify public policies which represent good value for money.
Existing economic analyses of the health impacts of housing improvement have often fallen short of undertaking economic evaluations.
Improved planning and better use of available data should ensure that economic evaluations are undertaken which can provide policy-makers with the evidence they require.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Contributors HT led the selection of studies, CM extracted the data, EF and HT acted as co-reviewers to check the data. CM and HT prepared the tables. EF provided economic expertise. All authors contributed to preparation of the manuscript and approved the final version.
Funding EF is funded by the University of Glasgow. CM and HT were funded by the Chief Scientist Office at the Scottish Government Health Directorate as part of the Evaluating Social Interventions programme at the Medical Research Council and Chief Scientist Office Social & Public Health Sciences Unit (U.130059812).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement This is not an original research article. However full details of the data extracted for this review are available from the authors.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.