Summary of study design and main results

We defined ‘amenable mortality’ as causes of death for which mortality rates are likely to reflect variations in the effectiveness of healthcare, with healthcare being limited to primary care, hospital care and personalised public health services (eg, immunisation and screening).

We first selected 54 causes of death for which a considerable decline in mortality has occurred in recent years and for which there was still a sufficient number of deaths in 2000 to allow a meaningful analysis of between-country variations in mortality. These 54 causes were selected by screening all three-digit cause-of-death codes from England and Wales, as an example of one of the larger European Union countries and one where coding specificities were well understood. We took the period 1979–2000, when ICD-9 was in use, to avoid problems arising from a change of ICD version. We selected causes of death with for which mortality declined more than 30% between 1979 and 2000 and for which the number of deaths in 2000 exceeded 100.

For these conditions, we conducted systematic reviews of the literature in order to identify healthcare interventions, which were introduced after 1970 and which, according to evidence from patient- or population-level studies, have effectively reduced mortality from these conditions. The strength of the evidence was highly variable, and our literature reviews revealed a number of important weaknesses in the evidence base. Nevertheless, for 14 conditions, we were able to identify interventions that more or less fulfilled this criterion (HIV/AIDS, colon cancer, cervical cancer, testicular cancer, Hodgkin's disease, rheumatic heart disease, hypertension, ischaemic heart disease, heart failure, cerebrovascular disease, peptic ulcer, renal failure, congenital heart disease and perinatal conditions).

For these interventions, we then identified the timing of introduction in seven European countries (Estonia, France, (West) Germany, the Netherlands, Spain, Sweden, United Kingdom). The selection of countries was based on several considerations: sufficient size to generate robust findings (mortality by cause is very unstable in small countries such as Iceland, Luxembourg or Malta), access to data (there are significant delays in production of mortality data from some countries such as Belgium and Denmark), geographical distribution (and thus economic conditions, with countries from North, South, East and West Europe) and finally, the existence of willing partners with sufficient expertise. The research was undertaken by partners in these countries, for example, by hand searching formularies for the first date of introduction of a drug, tracking sales data, identifying early trials, reviewing documents of professional societies and interviewing knowledgeable informants. Published sources of information are given in the online appendix. There were wide variations between countries in time of introduction of new interventions: for half of the innovations, the difference between the earliest and the latest adopter was more than 10 years.

For each of the 14 conditions, we also obtained mortality data for the seven countries in the study, covering the period 1970–2005. For this purpose, Estonian mortality data from the years before 1990 had to be reclassified from the abridged Soviet version of ICD-9 to the international version of ICD-9. Possible coding changes that might influence mortality trends were identified by an automated jump detection method.30 Correction factors were derived, and these were applied to create a consolidated database.

For each of the 14 conditions, and for men and women separately, mortality trends were analysed using Poisson regression analysis. Age (in 5-year age groups) was controlled for, and in the analyses reported here, no age limits were imposed. In a second series of mortality analyses, we applied an upper age limit of 75 years, and this analysis produced essentially the same results. We applied linear spline regression to identify points in time (‘knots’) at which the mortality trend changed significantly (eg, a mortality decline started or accelerated). We found that although mortality decline was the general pattern, this was far from uniform, both across time and between countries. A model with three knots described all mortality patterns sufficiently well, and we then related these knots to the approximate periods in which a decline in mortality from the introduction of innovations could be expected. We found slightly more matches between the two than could be expected to occur by chance alone. Introduction of innovation and favourable change in mortality trend coincided for only four conditions out of the 14 (ie, HIV/AIDS, colorectal cancer, ischaemic heart disease and cerebrovascular disease).

The results of each of the preceding steps were summarised for each condition in a one-page vignette, and experts were asked to assess the likelihood that variations in current mortality levels from these conditions reflect variations in the effectiveness of healthcare. We involved 23 experts from 16 countries who were either producers or users of evidence on health system performance, seeking participation by both genders and health professionals as well as non-health professionals. We held a Delphi procedure with two rounds31 in which the experts completed an online form for each vignette and were asked to score each cause of death on a scale of 1–9, where 1 was an entirely inappropriate indicator of health system performance and 9 was an entirely appropriate indicator. At the end of the first round, the distribution of scores for each cause was recorded and each participant was fed back both the distribution and their personal score, along with a summary of the comments that had been made. They were then asked to rescore them. In keeping with conventional usage, consensus was deemed to exist when, after excluding the highest and lowest individual scores, the remainder lay within a three-point range. After the second round, the experts had reached consensus on only three out of these 14 conditions (colorectal cancer, cervical cancer and cerebrovascular disease). Their comments suggested that they differed in their expectations and interpretations of indicators and, in particular, the ability to partition the impact of healthcare from other factors.

Four illustrations

We illustrate these results and their interpretation for four candidate conditions, representing both ‘positive’ and ‘negative’ findings: HIV/AIDS, Hodgkin's disease, breast cancer and cerebrovascular disease. Table 1 gives numerical data on these conditions, and figure 1 illustrates mortality trends (for reasons of space, this was restricted to one gender only for each condition). Our reasoning was that for variations in current national mortality levels to be valid indicators of healthcare effectiveness, current mortality levels should primarily reflect the degree of mortality decline induced by the introduction of healthcare interventions. They should not reflect pre-existing mortality levels or the extent of autonomous trends in mortality.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Mortality from four conditions and expected periods of mortality decline based on timing of innovation. (A) Estimated mortality trends from HIV/AIDS (men only) and expected periods of mortality decline based on the introduction of antiretroviral drugs. (B) Estimated mortality trends from Hodgkin's disease (men only) and expected periods of mortality decline based on the introduction of high-dose therapy and peripheral blood stem cell transplantation. (C) Estimated mortality trends from malignant neoplasm of breast (women only) and expected periods of mortality decline based on the introduction of screening by mammography. (D) Estimated mortality trends from cerebrovascular disease (women only) and expected periods of mortality decline based on the introduction of prevention by detection and treatment of hypertension.

In the case of HIV/AIDS, the main innovation occurring in the 1970–2005 period was treatment with antiretroviral drugs, which was introduced in the late 1980s and early 1990s, with some variations between countries32 (table 1). The few randomised clinical trials with mortality as an outcome33–35 cannot easily be generalised to the population now receiving combination therapy. Trials with mortality as an endpoint rapidly gave way to those with surrogate markers, as it was felt that, given the pace of therapeutic developments, results would otherwise be obsolete by the time they were published.36 Despite these problems, however, there is widespread consensus, justified by the combination of evidence from trials and observational studies that these drugs effectively lower mortality among patients,37 even if it is difficult to quantify the impact.38 In most of the countries participating in this study mortality from HIV/AIDS shows steep increases in the late 1980s and early 1990s, related to the spread of the epidemic among homosexual men, injecting drug users and other risk groups,39 followed by a sudden decline after the mid-1990s (figure 1A). The introduction of antiretroviral drugs coincided with a favourable change in mortality trend in most countries (sometimes preceding the peak in the mid-1990s). Countries in which antiretroviral drugs were introduced earlier tended to have earlier changes in mortality trend, as shown by the large number of ‘matches’ between innovation introduction and mortality trend change. The Delphi panel, however, could not reach consensus on the validity of this indicator, mainly because it felt that mortality from HIV/AIDS is also strongly determined by incidence. The final two columns of table 1 show the correlations of the recent mortality level from HIV/AIDS with, respectively, the mortality level before the favourable change in trend and the mortality decline since the favourable change. The first are much more positive than the second, suggesting that current national levels of mortality from HIV/AIDS primarily reflect the peak levels of mortality from this condition and not the extent of mortality decline since these peaks were reached. This casts doubt on the validity of current national mortality levels from HIV/AIDS as indicators of the effectiveness of healthcare for this specific condition.

In the case of Hodgkin's disease, considerable progress had already been made with chemotherapy,40 with a substantial improvement in long-term survival by 1970.41 The main innovation between 1970 and 2005 was high-dose chemotherapy combined with peripheral blood stem cell transplantation for those whose disease was not responding to conventional chemotherapy.42 This was introduced in the late 1980s and 1990s, with considerable variation between countries (table 1). Mortality from Hodgkin's disease, however, followed a more or less uniform decline during this period (figure 1B), and there were almost no ‘matches’ between innovation introduction and mortality trend change. The Delphi panel could not reach consensus on the validity of this indicator.

In the case of breast cancer, we identified two important innovations, screening with mammography and treatment with Tamoxifen. There is considerable evidence that the former is effective in saving lives,43 even if this is disputed,44 ,45 while the latter has been shown to be effective in prolonging survival, especially in women with oestrogen receptor-positive disease.46 Although the benefits of screening are still controversial, it is likely that the observed reductions in population-level mortality reflected the combined effect of both innovations. They were introduced over a protracted period, with considerable variation between countries and with substantial overlap between the periods when both innovations were introduced in each of the countries we studied (table 1). In many countries, mortality from breast cancer increased during the 1970s and 1980s, peaked in the early 1990s and then declined (figure 1C). There were very few ‘matches’ between the introduction of innovations and changes in the mortality trend. The Delphi panel could not reach consensus on the validity of this indicator. As the final columns of table 1 show, the between-country variations in current breast cancer mortality level reflect the mortality level before mortality started to decline, which is likely to reflect disease incidence.

In the case of cerebrovascular disease, we identified two important innovations, detection and treatment of hypertension and intensive management of stroke. Both have been shown to lower mortality.47 ,48 The first was introduced progressively over a protracted period from the 1950s onwards, with enormous variations in the coverage and intensity of treatment between countries. The second was introduced only recently, with less variation (table 1). Mortality from cerebrovascular disease has declined substantially in all countries in this study but with some accelerations and decelerations (figure 1D). We found a large number of ‘matches’ between innovation introduction and mortality trend change, and in this case, the Delphi panel also reached consensus on the validity of this indicator. Nevertheless, as the final columns of table 1 show, the current levels of cerebrovascular disease mortality not only reflect the degree of decline since 1970, but also the mortality level before it started to decline, suggesting that between-country variations in current mortality levels may be driven to a considerable extent by the prevalence of hypertension and associated risk factors and, potentially, patterns of death certification.

The way forward

The 14 conditions that we studied have been selected on the basis of evidence that there has been a substantial reduction in mortality and that effective interventions exist, but despite accumulating much more information than has previously been brought together, in most cases, we have not found clear evidence for a link between the reductions in mortality and the introduction of specific interventions. What does this mean?

It could of course mean that contrary to what we assumed, the interventions have not been effective in lowering mortality. Indeed, there is no doubt that the evidence on the effectiveness of interventions that we found in the literature is in many ways unsatisfactory. Few clinical trials assess mortality, instead focusing on intermediate or surrogate outcomes, even though these can be quite misleading.49–51 Most are undertaken with highly selected subjects, often in potentially atypical centres of excellence, so it is difficult to extrapolate results to the general population.52 Few clinical trials compare a new intervention against no treatment, instead showing incremental improvements on existing treatments. Consequently, while the cumulative effect of progressively better treatments may be considerable, it cannot be captured from trials as reported in the literature. Observational studies on clinical outcomes, on the other hand, often show important improvements in survival, but usually cannot link these to specific interventions. For example, it is apparent from the substantial reductions in mortality from common surgical conditions that non-urgent surgery has become much safer, but this is likely to reflect many different advances in anaesthesia, infection control and surgical technique. Similarly, there have been substantial improvements in long-term survival for many treatable cancers, but again the available studies rarely link this to a specific innovation.53

An alternative interpretation therefore is that innovations in healthcare did lower mortality from many of the conditions that we studied but occurred in a much more diffuse way than we assumed in our study design, which relied on discrete changes in mortality trends. We had considerable difficulty in finding information on when specific innovations were introduced. It was easier for pharmaceutical innovations, but even there we found examples of off-label prescribing prior to licensing. More importantly, the introduction of a particular innovation is only one factor that leads to increasing effectiveness. The determinants of increasing effectiveness can be represented by a three-dimensional diagram (figure 2), with innovation on one axis, coverage (or diffusion) on a second and quality with which it is applied on a third. Take hypertension: although the first drugs became available in the 1950s (innovation54), the threshold for treatment has changed, with progressively lower levels of blood pressure seen as justifying initiation of treatment,55 which has simultaneously been extended to those at older ages (coverage56). The ‘rule of thumb’ that treatment was only required when systolic pressure was ‘100 plus your age’ is now viewed as incorrect and obsolete. Over this time, the quality of care has also improved, with newer classes of drugs having fewer side effects, and thus improved compliance,57 as well as the introduction of combined treatment that maximises the intended effects but minimises the unintended ones,58 and the introduction of practice guidelines which have systematised detection and treatment of hypertension.59 We only studied one of the three factors, with considerable difficulty, and in hind-sight, it is therefore not surprising that we were rarely able to link reductions in mortality to specific healthcare interventions.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Dimensions of healthcare improvements.

Even if, as we believe, improvements in healthcare have contributed importantly to lowering mortality from these conditions, rates of mortality from amenable conditions must be used with caution as indicators of healthcare effectiveness in international comparisons, as our data show (last two columns of table 1). Countries with relatively high mortality rates from these conditions in 2005 often already had relatively high mortality before the innovations were introduced, as shown by the positive correlations between recent mortality levels and the mortality levels before a favourable change in trend. Only in the case of Hodgkin's disease do recent mortality levels seem primarily to reflect the steepness of decline. This implies that variations in current absolute levels of mortality in countries are likely to be determined, to some extent, by variations in the incidence of diseases and their risk factors, which may mask variations in healthcare effectiveness. In other words, at the end of a process of gradual diffusion (and increasing experience in their use along the way) of new interventions, the residual level of mortality probably reflects the influence of other factors that already prevailed before the introduction of the innovation.

It may therefore be useful to consider two other quantitative measures of mortality from conditions amenable to healthcare intervention. The first is the timing of a favourable change in mortality trends that can plausibly be linked to advances in treatment. In some cases, a clear change in trend could be seen, and the timing of these changes has been illustrated for three conditions in table 2. Sweden was clearly ahead of many other countries in bringing down breast cancer mortality (which could be associated with its early adoption of screening in regional pilot studies) and in bringing down cerebrovascular disease mortality (plausibly associated with effective detection and treatment of hypertension). Estonia, on the other hand, was only able to adopt many new interventions after independence in 1991. This indicator may reflect the responsiveness of healthcare systems to innovation.

The second alternative measure is the rate of mortality decline. Mortality decline could be calculated either on an annual basis or cumulatively over a fixed time interval and could be expressed either as a percentage of the original mortality level or in absolute terms. Sometimes large variations in speed of mortality decline are observed, and while these could also be due to variations in the speed with which population exposure to risk factors for amenable conditions changes, they are less likely to be biased by such confounding factors than national mortality levels. This indicator may reflect the speed with which healthcare systems achieve a substantial coverage of those in need. Table 2 shows some illustrative results for the four conditions, using percentage annual decline as an indicator. Spain had a relatively slow mortality decline for Hodgkin's disease and Germany for breast cancer, suggesting low speed of implementation of the respective innovations. This interpretation gains some support from a study of changes in amenable mortality in the different parts of the UK during a period when their relative funding levels were changing.13 It is also consistent with research on the USA showing a failure to achieve gains seen in other industrialised countries.20

In conclusion, in the current state of knowledge, between-country differences in the absolute levels of mortality from amenable conditions cannot be recommended for use in routine surveillance of healthcare performance. These levels are likely to reflect the influence of many different factors and do not support simple judgements. It is more likely that alternative measures of mortality from amenable conditions, such as the timing of a favourable change in mortality or the scale and pace of mortality decline, are better measures of healthcare performance, although both require further investigation. The main lesson from this study is, however, just how scarce is the evidence on, first, the ability of many common healthcare innovations to reduce mortality (which is, after all, one of their main objectives) and, second, on the timing, diffusion and coverage of healthcare interventions in Europe. Given the importance of healthcare to the well-being of European citizens and to economic growth in Europe, it seems remarkable that so little effort has gone into collection of comparative data on the uptake of health innovations and the quality of care among Member States, a situation that contrasts markedly with the large amounts of data collected in other sectors. Until these deficiencies are remedied, it will not be possible to quantify precisely the contribution of healthcare to population health, even though it undeniably exists.