Article Text

PDF

Enhancing the evidence base for health impact assessment
  1. J Mindell1,
  2. A Boaz2,
  3. M Joffe3,
  4. S Curtis4,
  5. M Birley5
  1. 1London Health Observatory, UK
  2. 2ESRC Centre for Evidence-based Policy and Practice, Queen Mary, University of London, UK
  3. 3Department of Epidemiology and Public Health, Imperial College London, UK
  4. 4Department of Geography, Queen Mary, University of London
  5. 5Liverpool University, UK
  1. Correspondence to:
 Dr J Mindell
 London Health Observatory, 11-13 Cavendish Square, London W1G 0AN, UK; jenny.mindelllho.org.uk

Abstract

Health impact assessment differs from other purposes for which evidence is collated in a number of ways, including:

  • the focus on complex interventions or policy and their diverse effects on determinants of health;

  • the need for evidence on the reversibility of adverse factors damaging to health;

  • the diversity of the evidence in terms of relevant disciplines, study designs, quality criteria and sources of information;

  • the broad range of stakeholders involved;

  • the short timescale and limited resources generally available;

  • the pragmatic need to inform decision makers regardless of the quality of the evidence.

These have implications for commissioning and conducting reviews. Methods must be developed to: facilitate comprehensive searching across a broad range of disciplines and information sources; collate appropriate quality criteria to assess a range of study designs; synthesise different kinds of evidence; and facilitate timely stakeholder involvement. Good practice standards for reviews are needed to reduce the risk of poor quality recommendations. Advice to decision makers must make explicit limitations resulting from absent, conflicting, or poor quality evidence.

  • guidelines
  • health impact assessment
  • systematic review
  • methodology

Statistics from Altmetric.com

Health impact assessment (HIA) assesses the positive and negative effects of a project, programme, or policy on health,1–4 and on health inequalities through the distribution of those effects.5,6 The health status of a person is determined primarily by a range of policies outside the jurisdiction of health services or health ministries.7,8 “Health in other policy areas” is therefore a major focus for HIA. Many UK government departments have issued policy documents that indicate the importance of HIA.9–15 Consideration of the health impacts of policies has also been encouraged across Europe.16–18 While evaluation examines the extent to which objectives were achieved,19 HIA aims to identify all potential health impacts, both intended and unintended. Currently, most HIA takes the form of rapid prospective appraisal of likely outcomes of proposed changes rather than examining policies and interventions while in progress or after completion.

HIA has been described as “the use of the best available evidence to assess the likely effect of a specific policy in a specific situation”,20,21 leading to comparisons with evidence based medicine. Systematic reviews, the main tool of evidence based medicine, can be very useful inputs into decision making, objectively summarising relevant information to identify possible effects of interventions and gaps in the evidence. Both policy makers and consumers can use them to aid their own decisions.22

However, the current HIA methodology has been criticised for lack of rigour in collecting and analysing evidence23,24: the quality of the evidence base currently available2,25 may limit the soundness and completeness of the conclusions,24 and consequently the capacity to achieve health gain. Thus, the policy drive to encourage HIA may have unintended consequences that are counterproductive: if it comes to be perceived as ineffective, and therefore wasteful of resources, it will gain a poor reputation and will no longer command support.26 Three types of knowledge are combined in HIA: that provided by stakeholders, local data, and evidence from past studies (fig 1). This paper considers the last of these.

Figure 1

 An outline scheme for health impact assessment, showing the place of “evidence” in the process.

Prospective impact assessment has a number of distinctive features including:

  • the focus, often on relatively complex policies or interventions with a diverse range of effects on determinants of health;

  • the need for evidence concerning the reversibility of adverse factors damaging to health;

  • a diverse evidence base comprising studies

    • from different disciplines,

    • using a range of designs,

    • involving a range of evidence relating to socioeconomic determinants of health;

  • a range of stakeholders from different backgrounds and with varying priorities, concerns, prior beliefs, and values;

  • the need to make recommendations to decision makers regardless of the quality of the evidence;

  • tight timescales as the norm.

This paper outlines the importance of these features and considers the implications for commissioning or conducting reviews of the research literature for use in HIA. For brevity, we use the term “intervention” for a project, programme, or policy whose potential impacts are being assessed.

FOCUS ON POLICY AND COMPLEX INTERVENTIONS

Complex policies and interventions are designed to achieve a variety of outcomes. Although many act on various aspects of wellbeing, health improvement may not be a primary objective. The relevant policies and interventions typically involve the complex processes surrounding socioeconomic determinants of health. The Acheson Inquiry on Inequalities in Health recommended that policies should routinely be assessed for their health impacts, giving impetus to HIA in the UK,27 building upon a longstanding awareness of the need for “healthy public policy”, that is also reflected in EU legislation.26

Epidemiology tends to focus on the relations between risk factors (or exposures or determinants) and health outcomes. Health impact assessment considers how these determinants are affected by proposed changes resulting from a policy or intervention. It therefore needs to incorporate a variety of types of information from a broad range of domains to examine the effects of policies on health determinants, factors called social antecedents to individual behaviours28 or determinants of the determinants.26 It tends to give more priority to population level risk because of the greater potential for prevention.29 Unlike in environmental impact assessment from which HIA sprang,30 a focus on this “social model of health”31 is the norm. However, causal pathways by which social determinants affect health are not as clearly defined in social compared with biological interventions.

A useful model for HIA is the policy/risk assessment model (PRAM).26 The PRAM model differs from quantitative risk assessment in two simple but fundamental aspects. Firstly, the different options and their effects on determinants of health are the core of the assessment, rather than being just the context in which determinants are studied. Secondly, the focus is not on the total health effects attributable to the baseline or final exposure but on the change in health status consequent on a proposal.

EVIDENCE FOR REVERSIBILITY AND GENERALISABILITY

While HIA is typically oriented towards the future, and thus towards prevention, it is also frequently applied to interventions (for example, urban regeneration schemes), which attempt to counteract existing socioeconomic disadvantage that has already damaged health. Here the issue is also whether an intervention will effectively reverse or ameliorate such disadvantage in ways that will lead to improved health for deprived groups. It is often assumed that this will happen but, for socioeconomic interventions, the research evidence on this point is limited and equivocal.25

For project level HIA, the best evidence is the outcome of similar projects in similar settings.32 There is also evidence that is very sensitive to context, for which it may be particularly difficult to identify generalisable effects that are confidently applicable in different settings. It is important to establish how far evidence is context specific, and to focus on explanations in terms of causal pathways, for reported associations between health determinants and health outcomes. To be useful for HIA, reviews need to consider both the distribution of impacts (especially non-medical health determinants) as well as average health. This requires special attention to causal pathways that may explain associations between specific factors and health inequalities.

Evidence on the effects of policy or other interventions needs to be prioritised and the effects of interventions need to be monitored in order to produce future evidence.33 It is also important to know the context within which evidence was collected.

DIVERSITY OF THE EVIDENCE

Lack of specificity of the question to be answered

In HIA there is usually a broad question (or series of broad questions) to answer, such as “In what ways will a change in transport policy, or this local housing project, affect health?”, rather than the very specific question recommended by most guidance on systematic reviews.34 Two approaches are possible. One is to compile a long list of more specific questions. The other, often used for policy review13 and sometimes in strategic environmental assessment (SEA),30,35,36 is to consider only “high level” evidence: more general global and regional impacts rather than the detailed local assessment.

Types of evidence

The philosophical question—and the legal, in contrast with scientific, definition—of what is evidence needs consideration.37 While HIA does not try to uncover absolute and incontrovertible truths, it does aim to produce recommendations that could withstand challenge in a court of law, because they are based on a balanced and reasonable interpretation of research evidence. Our ideal is to use information that meets scientific standards, so that the proposals are as well founded as possible, but when this cannot be done, it needs to be acknowledged.

The Cochrane Library is often not helpful in providing evidence to support HIAs: Cochrane reviews tend not to cover the policy interventions in which HIA is most interested because it is mostly health care oriented. For example there is little on transport and currently nothing on transport policy. Cochrane reviews, such as of smoking cessation policies and other public health relevant reviews, include case-control studies and interrupted time series analyses; about 400 reviews include non-randomised studies. Some Campbell collaboration reviews of the effectiveness of social and behavioural interventions in social welfare, education, and crime and justice will be useful in impact assessment.38 These systematic reviews will focus primarily on randomised controlled field trials (RFTs) but will also include good quality studies with quasi-experimental or qualitative design with controlled evaluation.39–41 The UK Centre for Evidence Based Policy and Practice (funded by the Economic and Social Research Council) has a node on evidence based public health policy.42 Policy interventions in education are reviewed by the EPPI-Centre (Evidence for Policy and Practice Information and Co-ordinating Centre).43

In contrast with most reviews of interventions, reviews for HIA usually require synthesis of evidence from epidemiological, toxicological, and sociological studies using a wide range of methodologies, as well as studies from a wide range of disciplines and topic areas, using both quantitative and qualitative research. A recent systematic review of housing interventions included quantitative evaluations, of any design, of the effects of housing interventions44 but for use in HIA, good quality observational aetiological studies were added to the review.45

Quality of the evidence

The quality of the evidence, and thus the likely validity of results and conclusions, depends first on appropriateness of the study design and secondly on how well it was carried out. The range of types of relevant studies means that many sets of quality criteria are required when reviewing literature for HIA.

Range of fields of study

Because HIAs are most commonly conducted to assess the effects of proposals outside health care, it is the norm rather than the exception for information on potential effects on health and its determinants to be required from fields of study other than health and medicine, but there are difficulties with identifying the studies in cross disciplinary reviews.46

Searching is a crucial step in conducting a literature review.46 This takes time and expertise to do properly, as well as familiarity with what literature is available, even within a single field. Searching other specialist domains is much more difficult:

  • The search terms may be unfamiliar, and studies may be less clearly or differently indexed and key worded than in healthcare databases.

  • Search engines or databases may be unavailable; or unfamiliar to health researchers.

  • The primary sources of such information, for example academic journals or professional newsletters, may be less accessible.

  • Searches of databases may be a less useful source of studies—contact with experts, and using books and bibliographies are essential for a full review.

It is therefore important to work with information experts to develop sensitive and specific search strategies.

Range of sources

Primary sources of information may be “grey literature”, internal reports in a range of disparate institutions with variable archiving skills. Although the English Health Development Agency’s HIA web site now has more than 100 items,47 there have been very few methodological papers48 or reports of actual HIAs in peer reviewed journals.25 A mechanism is needed for gathering grey literature, both to access the information and to reduce the risk of publication bias.

Combining the evidence

Much has been written about systematic reviews34,49 and meta-analyses,50,51 commonly done for intervention studies52 but also used in epidemiology53,54 and psychology.55 Combining the information gleaned from both a variety of study designs and a range of disciplines to be useful for HIA is more complex. This is an area that is currently underdeveloped in terms of methodologies. Non-statistical synthesis is often limited to narrative. Increasingly popular approaches for synthesising evidence from qualitative research are meta-ethnography56 and more recently Bayesian synthesis, integrating quantitative and qualitative evidence.57 These techniques need to be tried in evidence reviews for HIA.

BROAD RANGE OF STAKEHOLDERS

The evidence needs to be evaluated scientifically, but also by professionals in non-scientific, managerial roles and by lay members of society through public participation processes. Involving stakeholders has been recommended as essential for building interest in a project and promoting the potential use of the results. It is important to involve stakeholders from an early stage in the project to promote ownership.58 This is a time and resource intensive process for both the stakeholders and researchers. For example, there is a need to develop teaching in critical appraisal skills for a wide range of stakeholders.59

An important issue for HIA is the need to resolve tensions between the technocratic and the participative view60,61; not only do we have to involve stakeholders in understanding the evidence, we also need to give consideration to their perspectives. This would not usually be part of synthesising the generalisable evidence, but is more appropriate as part of each individual HIA, in which both eliciting the opinions of the broad range of stakeholders and linking their views of and with the evidence are major elements (fig 1). It does mean, however, that the evidence and the criteria used to weight it need to be accessible for a non-specialist audience, in ways that enable them to judge for themselves.

ADVICE TO DECISION MAKERS

When there is uncertainty because the evidence is conflicting, of poor quality or absent/insufficient, many reviews either leave out that topic or state simply that there is no [good quality] evidence. However, those involved with HIA are expected to develop conclusions and recommendations for decision makers, regardless of the shortcomings of the evidence base. The evidence is often sufficient for the precautionary principle62 to apply. It was enshrined in the Rio Declaration, whose principle 15 states: “Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation”.63 It has been generally used with regard to possible environmental damage,63,64 but is readily extendable to threats to health. Even in the absence of firm evidence of a causal relation, the other steps for a quantified HIA may be needed if a causal relation cannot be discounted.62 This is analogous to the call by Green and Tones for using a [civil] “judicial principle” for assessment of evidence about health promotion, and judging it on the “balance of probabilities” rather than “beyond all reasonable doubt”.65

Key points

  • Health impact assessment (HIA) aims to combine peer reviewed evidence with local knowledge to provide decision makers with justified recommendations for project and policy changes that safeguard and enhance health.

  • Systematic reviews of the evidence associated with those determinants are required to support the impact assessment process but there have been criticisms of lack of rigour.

  • Distinctive features of HIA include the focus on complex influences on determinants of health; the range of disciplines and study designs contributing evidence; the involvement of a range of stakeholders; and the short timescales usually available.

  • Guidance for synthesising evidence for use in HIA needs to address all these issues.

Being explicit about assumptions

When the evidence base is suboptimal, one may still need to summarise the “best available evidence”. This requires a pragmatic but appropriately systematic way of using studies that meet a threshold quality standard. Any approximations and assumptions made in a HIA, and its consequent limitations, must be made explicit. Stakeholders can play an active part in communicating the strengths and limitations of the HIA.

TIMELINESS

The methodologies for HIA share many features.66 All aim to present valid evidence to decision makers but are hampered by the short time frame between publication of proposals and decisions thereon. Ideally, impacts on health should be considered at the conceptual stage when opportunities for change are greatest. In practice, potential health impacts cannot be assessed until proposals are firm enough to be examined, but there may be a very short interval before decisions are made.

Systematic reviews have many advantages over traditional narrative reviews23 but it can be very time consuming to conduct one of high quality. The organisation and ordering of complex literature for the purposes of HIA is difficult. Even when a HIA is begun early enough, tension inevitably exists between the desire to do a good assessment and the need to complete it quickly. In practice, most local and health authorities and government departments are doing rapid appraisal HIA,67,68 which requires fewer resources and is done within a short time frame of days or weeks, so usually depends on rapid reviews of published evidence, usually based on secondary sources. Poor quality reviews can mislead24; reviews of this kind69 have received considerable criticism.70,71

There is therefore potential to develop and apply systematic review techniques to meet the needs of HIA. Part of this process is about understanding current practice. For example, the Centre for Reviews and Dissemination uses the term “rapid review” to mean a systematic review that is completed more quickly than usual without compromising quality standards (A Sowden, CRD, personal communication (AB)), whereas in HIA “rapid review” means a review of secondary sources completed much more quickly—usually within days or occasionally a few weeks. Rapid reviews are unreliable unless they are systematic, consider bias and confounding, and explicitly examine causality. To meet these conditions, it is unlikely that a satisfactory review could be as rapid as in current HIA practice, except for a topic with very limited evidence that is readily located.72

There are many reasons for non-systematic review but the reason for choosing this approach should always be stated overtly. It is usually assumed that reviews are conducted non-systematically because of severe time constraints or because of lack of training or resources. However, those with vested interests, including businesses and political organisations, may also be interested in non-systematic reviews.

MAKING EVIDENCE AVAILABLE FOR USERS OUTSIDE THE RESEARCH COMMUNITY

An important aspect of HIA is that it involves interpretation of the research evidence by users who are not themselves professional scientific researchers and cannot be expected to quickly assimilate all the knowledge and competencies upon which researchers draw when using evidence. There is a need for something like the Cochrane collaboration, which provides expertly reviewed evidence on key topics to the medical profession and others concerned with healthcare provision.

Policy implications

  • Proactive production of “off the shelf” systematic literature reviews covering common policy areas are required for health impact assessment (HIA) of interventions outside the health service sector. These could contribute to better quality HIA by improving access to robust evidence and allowing local resources to focus on community participation and collection of local information. Guidelines for both systematic and “rapid” reviews of evidence for use in HIA are required to ensure adequate quality.

Pragmatically, rapid reviews will continue to be performed until the full range of up to date systematic reviews is readily available. It is important that these are of sufficient quality to ensure that valid recommendations can be based on them: minimum quality standards are needed to reduce the risk of poor quality, inappropriate and weak recommendations being given in HIA. An initial standard might be to include explicit statements of the expertise and method(s) used.

However, in the longer term, rather than waiting until a systematic review is required for a specific HIA, there are calls for systematic reviews to be conducted proactively26,48,73 and made available via the internet47 to expedite robust local HIA. Such reviews of evidence are useful not only for HIA but also for other studies of policies, determinants of health and health outcomes. Most topics considered within an HIA are not specific to that particular assessment, but are similar to the topics of many other HIAs carried out elsewhere, providing the opportunity to develop an “off the shelf” evidence base,25 starting with areas that are frequent topics for HIA, such as regeneration,74 housing,45 and transport.75 Such reviews are likely to be carried out by a variety of reviewers for a range of purposes. There is a need to set standards and provide guidance to assure a quality product, analogous with the Cochrane collaboration. Recent regional and national discussions76 have shown widespread enthusiasm for such work: improving the evidence base used within HIA should lead to greater effectiveness of HIA as a process.77

We have shown the particular difficulties encountered when reviewing published evidence for use in HIA. Existing scientific standards for reviews cover only certain aspects of the guidance required. As part of the current programme of research on systematic reviews, the English Department of Health is funding a project designed to develop such guidance. The definition of those standards and guidance is being undertaken collaboratively. The project aims to improve access to robust evidence for HIA and a main outcome from the project will be the publication of demonstration quality assured reviews on the web.

Acknowledgments

We thank Helen Lewis of the London Health Observatory for administrative assistance, Mark Petticrew for extensive and constructive help with earlier drafts of this paper, and all those who participated in discussions of an earlier draft of this paper, at the workshop or by email.

PARTICIPANTS IN THE WORKSHOP DISCUSSION HELD BY THE LONDON HEALTH OBSERVATORY ON 23 SEPTEMBER 2002
 Helen Atkinson, Public health specialist, Lambeth PCT; Martin Birley, Consultant in HIA, University of Liverpool; Veronica Birley, Training and consultancy manager; Annette Boaz, Senior Research Fellow, ESRC Centre for Evidence Based Policy and Practice; Dr Zaid Chalabi, Lecturer in health impact analysis and modelling, LSHTM; Professor Sarah Curtis, Professor of Geography, Queen Mary, University of London; Mandy Guest, Medical informatics library manager, Islington PCT; Dr Bobbie Jacobson, Director, London Health Observatory; Dr Mike Joffe, Reader, Imperial College London; Professor Mike Kelly, Director of Research and Information, Health Development Agency; Dr Karen Lock, MRC Research Fellow, LSHTM; Dr Jennifer Mindell, Deputy Director, London Health Observatory; Dr Lesley Mountford, Specialist Registrar, Greater London Authority; Dr Bhash Naidoo, Research specialist, Health Development Agency; Dr Mark Petticrew, Associate Director, MRC Social and Public Health Sciences Unit, Glasgow; Sarah Price, North East London Strategic Health Authority; Dr Fahri Seljmani, Data analyst, London Health Observatory.

OTHER CONTRIBUTORS TO DRAFTS
 Dr Jane Biddulph, Lecturer in Research Methods and Statistics, Department of Primary Care and Population Sciences, UCL (University College London); Paul Tomlinson, Head Environmental Assessment and Policy, TRL; Caron Bowen, HIA Facilitation Manager, London Health Observatory.

REFERENCES

View Abstract

Footnotes

  • Funding: none.

  • Conflicts of interest: none declared.

  • An earlier draft of this paper was discussed at a workshop held by the London Health Observatory in London on 23 September 2002. Participants are listed above.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles