Health impact assessment (HIA) assesses the positive and negative effects of a project, programme, or policy on health,^1–4 and on health inequalities through the distribution of those effects.^5,6 The health status of a person is determined primarily by a range of policies outside the jurisdiction of health services or health ministries.^7,8 “Health in other policy areas” is therefore a major focus for HIA. Many UK government departments have issued policy documents that indicate the importance of HIA.^9–15 Consideration of the health impacts of policies has also been encouraged across Europe.^16–18 While evaluation examines the extent to which objectives were achieved,¹⁹ HIA aims to identify all potential health impacts, both intended and unintended. Currently, most HIA takes the form of rapid prospective appraisal of likely outcomes of proposed changes rather than examining policies and interventions while in progress or after completion.

HIA has been described as “the use of the best available evidence to assess the likely effect of a specific policy in a specific situation”,^20,21 leading to comparisons with evidence based medicine. Systematic reviews, the main tool of evidence based medicine, can be very useful inputs into decision making, objectively summarising relevant information to identify possible effects of interventions and gaps in the evidence. Both policy makers and consumers can use them to aid their own decisions.²²

However, the current HIA methodology has been criticised for lack of rigour in collecting and analysing evidence^23,24: the quality of the evidence base currently available^2,25 may limit the soundness and completeness of the conclusions,²⁴ and consequently the capacity to achieve health gain. Thus, the policy drive to encourage HIA may have unintended consequences that are counterproductive: if it comes to be perceived as ineffective, and therefore wasteful of resources, it will gain a poor reputation and will no longer command support.²⁶ Three types of knowledge are combined in HIA: that provided by stakeholders, local data, and evidence from past studies (fig 1). This paper considers the last of these.

Prospective impact assessment has a number of distinctive features including:

• the focus, often on relatively complex policies or interventions with a diverse range of effects on determinants of health;

• the need for evidence concerning the reversibility of adverse factors damaging to health;

• a diverse evidence base comprising studies

  • from different disciplines,

  • using a range of designs,

  • involving a range of evidence relating to socioeconomic determinants of health;

• a range of stakeholders from different backgrounds and with varying priorities, concerns, prior beliefs, and values;

• the need to make recommendations to decision makers regardless of the quality of the evidence;

• tight timescales as the norm.

This paper outlines the importance of these features and considers the implications for commissioning or conducting reviews of the research literature for use in HIA. For brevity, we use the term “intervention” for a project, programme, or policy whose potential impacts are being assessed.

FOCUS ON POLICY AND COMPLEX INTERVENTIONS

Complex policies and interventions are designed to achieve a variety of outcomes. Although many act on various aspects of wellbeing, health improvement may not be a primary objective. The relevant policies and interventions typically involve the complex processes surrounding socioeconomic determinants of health. The Acheson Inquiry on Inequalities in Health recommended that policies should routinely be assessed for their health impacts, giving impetus to HIA in the UK,²⁷ building upon a longstanding awareness of the need for “healthy public policy”, that is also reflected in EU legislation.²⁶

Epidemiology tends to focus on the relations between risk factors (or exposures or determinants) and health outcomes. Health impact assessment considers how these determinants are affected by proposed changes resulting from a policy or intervention. It therefore needs to incorporate a variety of types of information from a broad range of domains to examine the effects of policies on health determinants, factors called social antecedents to individual behaviours²⁸ or determinants of the determinants.²⁶ It tends to give more priority to population level risk because of the greater potential for prevention.²⁹ Unlike in environmental impact assessment from which HIA sprang,³⁰ a focus on this “social model of health”³¹ is the norm. However, causal pathways by which social determinants affect health are not as clearly defined in social compared with biological interventions.

A useful model for HIA is the policy/risk assessment model (PRAM).²⁶ The PRAM model differs from quantitative risk assessment in two simple but fundamental aspects. Firstly, the different options and their effects on determinants of health are the core of the assessment, rather than being just the context in which determinants are studied. Secondly, the focus is not on the total health effects attributable to the baseline or final exposure but on the change in health status consequent on a proposal.

EVIDENCE FOR REVERSIBILITY AND GENERALISABILITY

While HIA is typically oriented towards the future, and thus towards prevention, it is also frequently applied to interventions (for example, urban regeneration schemes), which attempt to counteract existing socioeconomic disadvantage that has already damaged health. Here the issue is also whether an intervention will effectively reverse or ameliorate such disadvantage in ways that will lead to improved health for deprived groups. It is often assumed that this will happen but, for socioeconomic interventions, the research evidence on this point is limited and equivocal.²⁵

For project level HIA, the best evidence is the outcome of similar projects in similar settings.³² There is also evidence that is very sensitive to context, for which it may be particularly difficult to identify generalisable effects that are confidently applicable in different settings. It is important to establish how far evidence is context specific, and to focus on explanations in terms of causal pathways, for reported associations between health determinants and health outcomes. To be useful for HIA, reviews need to consider both the distribution of impacts (especially non-medical health determinants) as well as average health. This requires special attention to causal pathways that may explain associations between specific factors and health inequalities.

Evidence on the effects of policy or other interventions needs to be prioritised and the effects of interventions need to be monitored in order to produce future evidence.³³ It is also important to know the context within which evidence was collected.

DIVERSITY OF THE EVIDENCE

Lack of specificity of the question to be answered

In HIA there is usually a broad question (or series of broad questions) to answer, such as “In what ways will a change in transport policy, or this local housing project, affect health?”, rather than the very specific question recommended by most guidance on systematic reviews.³⁴ Two approaches are possible. One is to compile a long list of more specific questions. The other, often used for policy review¹³ and sometimes in strategic environmental assessment (SEA),^30,35,36 is to consider only “high level” evidence: more general global and regional impacts rather than the detailed local assessment.

Types of evidence

The philosophical question—and the legal, in contrast with scientific, definition—of what is evidence needs consideration.³⁷ While HIA does not try to uncover absolute and incontrovertible truths, it does aim to produce recommendations that could withstand challenge in a court of law, because they are based on a balanced and reasonable interpretation of research evidence. Our ideal is to use information that meets scientific standards, so that the proposals are as well founded as possible, but when this cannot be done, it needs to be acknowledged.

The Cochrane Library is often not helpful in providing evidence to support HIAs: Cochrane reviews tend not to cover the policy interventions in which HIA is most interested because it is mostly health care oriented. For example there is little on transport and currently nothing on transport policy. Cochrane reviews, such as of smoking cessation policies and other public health relevant reviews, include case-control studies and interrupted time series analyses; about 400 reviews include non-randomised studies. Some Campbell collaboration reviews of the effectiveness of social and behavioural interventions in social welfare, education, and crime and justice will be useful in impact assessment.³⁸ These systematic reviews will focus primarily on randomised controlled field trials (RFTs) but will also include good quality studies with quasi-experimental or qualitative design with controlled evaluation.^39–41 The UK Centre for Evidence Based Policy and Practice (funded by the Economic and Social Research Council) has a node on evidence based public health policy.⁴² Policy interventions in education are reviewed by the EPPI-Centre (Evidence for Policy and Practice Information and Co-ordinating Centre).⁴³

In contrast with most reviews of interventions, reviews for HIA usually require synthesis of evidence from epidemiological, toxicological, and sociological studies using a wide range of methodologies, as well as studies from a wide range of disciplines and topic areas, using both quantitative and qualitative research. A recent systematic review of housing interventions included quantitative evaluations, of any design, of the effects of housing interventions⁴⁴ but for use in HIA, good quality observational aetiological studies were added to the review.⁴⁵

Quality of the evidence

The quality of the evidence, and thus the likely validity of results and conclusions, depends first on appropriateness of the study design and secondly on how well it was carried out. The range of types of relevant studies means that many sets of quality criteria are required when reviewing literature for HIA.

Range of fields of study

Because HIAs are most commonly conducted to assess the effects of proposals outside health care, it is the norm rather than the exception for information on potential effects on health and its determinants to be required from fields of study other than health and medicine, but there are difficulties with identifying the studies in cross disciplinary reviews.⁴⁶

Searching is a crucial step in conducting a literature review.⁴⁶ This takes time and expertise to do properly, as well as familiarity with what literature is available, even within a single field. Searching other specialist domains is much more difficult:

• The search terms may be unfamiliar, and studies may be less clearly or differently indexed and key worded than in healthcare databases.

• Search engines or databases may be unavailable; or unfamiliar to health researchers.

• The primary sources of such information, for example academic journals or professional newsletters, may be less accessible.

• Searches of databases may be a less useful source of studies—contact with experts, and using books and bibliographies are essential for a full review.

It is therefore important to work with information experts to develop sensitive and specific search strategies.

Range of sources

Primary sources of information may be “grey literature”, internal reports in a range of disparate institutions with variable archiving skills. Although the English Health Development Agency’s HIA web site now has more than 100 items,⁴⁷ there have been very few methodological papers⁴⁸ or reports of actual HIAs in peer reviewed journals.²⁵ A mechanism is needed for gathering grey literature, both to access the information and to reduce the risk of publication bias.

Combining the evidence

Much has been written about systematic reviews^34,49 and meta-analyses,^50,51 commonly done for intervention studies⁵² but also used in epidemiology^53,54 and psychology.⁵⁵ Combining the information gleaned from both a variety of study designs and a range of disciplines to be useful for HIA is more complex. This is an area that is currently underdeveloped in terms of methodologies. Non-statistical synthesis is often limited to narrative. Increasingly popular approaches for synthesising evidence from qualitative research are meta-ethnography⁵⁶ and more recently Bayesian synthesis, integrating quantitative and qualitative evidence.⁵⁷ These techniques need to be tried in evidence reviews for HIA.

BROAD RANGE OF STAKEHOLDERS

The evidence needs to be evaluated scientifically, but also by professionals in non-scientific, managerial roles and by lay members of society through public participation processes. Involving stakeholders has been recommended as essential for building interest in a project and promoting the potential use of the results. It is important to involve stakeholders from an early stage in the project to promote ownership.⁵⁸ This is a time and resource intensive process for both the stakeholders and researchers. For example, there is a need to develop teaching in critical appraisal skills for a wide range of stakeholders.⁵⁹

An important issue for HIA is the need to resolve tensions between the technocratic and the participative view^60,61; not only do we have to involve stakeholders in understanding the evidence, we also need to give consideration to their perspectives. This would not usually be part of synthesising the generalisable evidence, but is more appropriate as part of each individual HIA, in which both eliciting the opinions of the broad range of stakeholders and linking their views of and with the evidence are major elements (fig 1). It does mean, however, that the evidence and the criteria used to weight it need to be accessible for a non-specialist audience, in ways that enable them to judge for themselves.

ADVICE TO DECISION MAKERS

When there is uncertainty because the evidence is conflicting, of poor quality or absent/insufficient, many reviews either leave out that topic or state simply that there is no [good quality] evidence. However, those involved with HIA are expected to develop conclusions and recommendations for decision makers, regardless of the shortcomings of the evidence base. The evidence is often sufficient for the precautionary principle⁶² to apply. It was enshrined in the Rio Declaration, whose principle 15 states: “Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation”.⁶³ It has been generally used with regard to possible environmental damage,^63,64 but is readily extendable to threats to health. Even in the absence of firm evidence of a causal relation, the other steps for a quantified HIA may be needed if a causal relation cannot be discounted.⁶² This is analogous to the call by Green and Tones for using a [civil] “judicial principle” for assessment of evidence about health promotion, and judging it on the “balance of probabilities” rather than “beyond all reasonable doubt”.⁶⁵

Being explicit about assumptions

When the evidence base is suboptimal, one may still need to summarise the “best available evidence”. This requires a pragmatic but appropriately systematic way of using studies that meet a threshold quality standard. Any approximations and assumptions made in a HIA, and its consequent limitations, must be made explicit. Stakeholders can play an active part in communicating the strengths and limitations of the HIA.

TIMELINESS

The methodologies for HIA share many features.⁶⁶ All aim to present valid evidence to decision makers but are hampered by the short time frame between publication of proposals and decisions thereon. Ideally, impacts on health should be considered at the conceptual stage when opportunities for change are greatest. In practice, potential health impacts cannot be assessed until proposals are firm enough to be examined, but there may be a very short interval before decisions are made.

Systematic reviews have many advantages over traditional narrative reviews²³ but it can be very time consuming to conduct one of high quality. The organisation and ordering of complex literature for the purposes of HIA is difficult. Even when a HIA is begun early enough, tension inevitably exists between the desire to do a good assessment and the need to complete it quickly. In practice, most local and health authorities and government departments are doing rapid appraisal HIA,^67,68 which requires fewer resources and is done within a short time frame of days or weeks, so usually depends on rapid reviews of published evidence, usually based on secondary sources. Poor quality reviews can mislead²⁴; reviews of this kind⁶⁹ have received considerable criticism.^70,71

There is therefore potential to develop and apply systematic review techniques to meet the needs of HIA. Part of this process is about understanding current practice. For example, the Centre for Reviews and Dissemination uses the term “rapid review” to mean a systematic review that is completed more quickly than usual without compromising quality standards (A Sowden, CRD, personal communication (AB)), whereas in HIA “rapid review” means a review of secondary sources completed much more quickly—usually within days or occasionally a few weeks. Rapid reviews are unreliable unless they are systematic, consider bias and confounding, and explicitly examine causality. To meet these conditions, it is unlikely that a satisfactory review could be as rapid as in current HIA practice, except for a topic with very limited evidence that is readily located.⁷²

There are many reasons for non-systematic review but the reason for choosing this approach should always be stated overtly. It is usually assumed that reviews are conducted non-systematically because of severe time constraints or because of lack of training or resources. However, those with vested interests, including businesses and political organisations, may also be interested in non-systematic reviews.

MAKING EVIDENCE AVAILABLE FOR USERS OUTSIDE THE RESEARCH COMMUNITY

An important aspect of HIA is that it involves interpretation of the research evidence by users who are not themselves professional scientific researchers and cannot be expected to quickly assimilate all the knowledge and competencies upon which researchers draw when using evidence. There is a need for something like the Cochrane collaboration, which provides expertly reviewed evidence on key topics to the medical profession and others concerned with healthcare provision.

Pragmatically, rapid reviews will continue to be performed until the full range of up to date systematic reviews is readily available. It is important that these are of sufficient quality to ensure that valid recommendations can be based on them: minimum quality standards are needed to reduce the risk of poor quality, inappropriate and weak recommendations being given in HIA. An initial standard might be to include explicit statements of the expertise and method(s) used.

However, in the longer term, rather than waiting until a systematic review is required for a specific HIA, there are calls for systematic reviews to be conducted proactively^26,48,73 and made available via the internet⁴⁷ to expedite robust local HIA. Such reviews of evidence are useful not only for HIA but also for other studies of policies, determinants of health and health outcomes. Most topics considered within an HIA are not specific to that particular assessment, but are similar to the topics of many other HIAs carried out elsewhere, providing the opportunity to develop an “off the shelf” evidence base,²⁵ starting with areas that are frequent topics for HIA, such as regeneration,⁷⁴ housing,⁴⁵ and transport.⁷⁵ Such reviews are likely to be carried out by a variety of reviewers for a range of purposes. There is a need to set standards and provide guidance to assure a quality product, analogous with the Cochrane collaboration. Recent regional and national discussions⁷⁶ have shown widespread enthusiasm for such work: improving the evidence base used within HIA should lead to greater effectiveness of HIA as a process.⁷⁷

We have shown the particular difficulties encountered when reviewing published evidence for use in HIA. Existing scientific standards for reviews cover only certain aspects of the guidance required. As part of the current programme of research on systematic reviews, the English Department of Health is funding a project designed to develop such guidance. The definition of those standards and guidance is being undertaken collaboratively. The project aims to improve access to robust evidence for HIA and a main outcome from the project will be the publication of demonstration quality assured reviews on the web.