How policy informs the evidence

“Evidence based” thinking can lead to debased policy making

Education and debate p 222

Who would not want health policy to be based on evidence? “Evidence based medicine” and “evidence based policy” have such reassuring and self evidently desirable qualities that it may seem contrary to question their legitimacy in relation to reducing health inequalities. However, these terms are now so familiar that it is easy to forget the important question about what sort of data provide appropriate evidence for particular types of decisions. The sort of evidence gathered on the benefits of interventions aimed at individuals may not help in guiding policies directed towards reducing health inequalities.

In this week's BMJ readers have the opportunity to assess part of the process leading to the recommendations of the Independent Inquiry into Health Inequalities (the Acheson inquiry),1 established in 1997 to help the government formulate policy to reduce health inequalities. The inquiry established an evaluation group to report on the quality of the evidence it used to reach its conclusions and support its recommendations.2 This group critiqued submissions to the inquiry, and a list of its own remedies for health inequalities—their “10 steps to health equality”—was released before the Acheson inquiry had itself reported (see box on bmj.com).3

The evaluation group appears to have applied evidence based principles to its consideration of ways to reduce inequalities in health. Essentially it wanted evidence from controlled intervention studies, and its main evaluation consisted of checking each recommendation against three earlier reviews (two conducted within an explicit evidence based framework) and the Cochrane Library.

The task of the Acheson inquiry was to make recommendations that would reduce inequalities in health, not merely have a positive overall health benefit. For most of the evaluation group's suggested interventions there are no high quality controlled studies showing that they would reduce health inequalities—for example, the evidence that fluoridation of drinking water would reduce inequalities in dental health is scanty.4 Indeed, some of these interventions could increase inequalities. Smoking cessation may be more successful in advantaged groups. Drugs education in schools may have less impact on those most at risk, because they are more likely to be truants and thus less exposed to it.

On the general question of what sort of evidence is useful to set policy in the public health domain, it is helpful to think back to earlier eras. In the first half of the 19th century there were no “evaluation groups” to point out the lack of evidence from controlled intervention studies showing the health benefits of, for example, stopping children under 9 from working in cotton mills, fencing off dangerous machinery, or reducing the number of hours children could work to only 10 a day. With an evaluation group, implementation of the Factory Acts could have been resisted. The factory owners were certainly keen on “evidence”: the claim that working class children aged 5-10 had lower death rates than middle class children was used to suggest that factory labour was good for the under 10s.5

Clearly the situation is now different, but health inequalities are still large and have increased over the past two decades.6 Premature death rates are over three and a half times higher in Glasgow Shettleston than in Wokingham,6 and a remarkable three quarters of premature deaths in Glasgow Shettleston would not occur if it had the mortality rates of Wokingham. It is no surprise that in Glasgow Shettleston child poverty rates are over six times, and unemployment rates over five times, higher than in Wokingham. Clearly the need is for substantial reductions in socioeconomic inequality, which can follow only from the concerted implementation of policies of progressive taxation and substantial income redistribution.

The evaluation group states that randomised trials of income support have been carried out and could, in principle, have examined health outcomes.7 However, the effects of income redistribution would not be to give a few people a little more money while they remain living in a highly unequal society, but to change the nature of the society. Health is influenced by micro and macro social environments,8 and societies with high levels of income inequality are characterised by a wide range of social-structural attributes that have a detrimental impact on health.9

As Schwartz and Carpenter have pointed out, inappropriately focusing on individual level determinants of health while ignoring more important macrolevel determinants is tantamount to obtaining the right answer to the wrong question.10 Consider the situation of examining risk factors for unemployment. Conventional individual-level studies would probably find that low education, not dressing smartly for interviews, being short, being over 50, or being a member of a minority ethnic group predict being unemployed. Indeed these “risk factors” would probably explain a high percentage of the variance in unemployment. A controlled study finding that counselling on how to dress and behave at job interviews increases success in getting a job could be added to the Cochrane Library. The same risk factors may explain a high percentage of the intra-individual variance in unemployment, both when unemployment is 1% and when it is 14%.

The big difference for the population—and thus for the individual risk of unemployment—is, however, the 14-fold difference in overall levels of unemployment at times when different fiscal policies are being implemented. High variance apparently “explained” by individual-level risk indicators (or markers manipulable in a discrete way within populations) does not mean that they are important determinants of the population level of any outcome.11 These are, however, precisely the factors that evidence based research focuses on. Despite occasional rhetorical interest in wider determinants of health, evidence based assessments are largely restricted to individualised interventions. The Cochrane Library is unlikely ever to contain systematic reviews or trials of the effects of redistributive national fiscal policies, or of economic investment leading to reductions in unemployment, on health.

The insidious nature of this mismatch between evidence and policy is highlighted by the fact that the evaluation group is, as one would expect of such informed commentators, aware of the problem, while implicitly ignoring it. One of the evaluation group stated when launching the “10 steps to health equity”, “Our recommendations are quite medical because those are the sort that tend to have evidence behind them.”3 Health differentials between social groups, or between poor and rich countries, are not primarily generated by medical causes and require solutions at a different level.

One source of the scientific innovation that was institutionalised within the Cochrane Collaboration was a powerful critique of a complacent and uncritical form of health care delivery.12 The establishment of the evidence based medicine movement is a remarkable achievement with an unquestionably favourable influence on the probability that individuals will receive health care that benefits them and be protected from interventions that harm them. It would be ironic, and inconsistent with Cochrane's radical instincts, if the inappropriate applications of those ideas were to provide a complacent barrier to implementing those measures necessary to redress health inequalities.