Article Text
PDF
Abstract
Background Amid the calls for ‘more public health evidence’, we also need simple understandable methods of determining when more research really is needed. This paper describes a simple decision aid to help policymakers, researchers and other decision makers assess the potential ‘information value’ of a new public health randomised controlled trial.
Methods The authors developed a flow chart to help make explicit (1) the user's information needs, (2) the intended use of the new information that the study will produce, (3) the added value of the evidence to be derived from the new study and (4) the levels of precision, bias and generalisability required by the user.
Results The flow chart is briefly illustrated, first in generic form and then in a worked example, showing how it may be used in deciding whether a new study should be commissioned to evaluate the health impact of allowing motorcycles to use bus lanes in London.
Conclusions In this paper, the authors have presented a flow chart for enacting an informal ‘Value-of-Information’-like approach to deciding when a new public health evaluation is needed. The authors do not suggest that the flow chart approach is technically the equivalent of Value-of-Information methods. Nonetheless, it represents a valuable perspective and process to adopt, and this structured approach will be more revealing than an unstructured thought experiment as the basis for decisions about a new study. To aid in its development as an effective tool, we invite users from a variety of perspectives and contexts to review it, to use it in practice and to send us their comments.
- Epidemiology ME
- evaluation ME
- policy analysis SI
- public health epidemiology
- randomised trials
Statistics from Altmetric.com
Introduction
There have been many recent calls for more and better public health (PH) evidence. However, the decision to fund or conduct any new research is itself an intervention, and like any intervention, it needs careful consideration of its costs and benefits. To this end, an informal hierarchy of methods for considering the value of a new study—whether in PH or other fields—is currently employed, running from unstructured thought experiments, through deterministic and stochastic simulation of a new study, to formal Value-of-Information (VOI) methods. Systematic reviews have also been advocated for identifying needs for further research; but after collecting and summarising existing evidence, the process of identifying the need for more evidence then usually remains informal. There is need for more formal but easily understood approaches to assessing the need for new PH research, and this paper proposes one such approach, based on a flow chart, which may help consider the value of new PH trials.
Demand versus need for better PH evidence
Many reports in recent years have called for more and better PH evidence. This evidence can take many forms, and although policymakers and practitioners require many types of qualitative and quantitative evidence to inform decision making, there has been a particular focus on the role and suitability of randomised controlled trials (RCTs) in this context.1–5 Demands for rapid development in our knowledge of ‘what works’ are often seen as synonymous with a need for more RCTs, as opposed to other non-experimental study designs. However, in healthcare, there is often a considerable difference between ‘demand’ and ‘need’, and the same applies to research. Demands for new research should be considered carefully, and the costs and benefits carefully assessed. The activities of commissioning, funding and conducting research all have costs and benefits, not least opportunity costs because the budget to fund PH research is not unlimited, and there are constraints in the capacity of the research community. Researchers claiming that more ‘research is needed’, and commissioners of new research need simple and effective ways of assessing the potential gains from new research.
This assessment can be particularly difficult for PH research, where the outcomes of trials may legitimately include costs and benefits in sectors other than health (such as education, transport and other sectors), as well as improvements in knowledge and reductions in uncertainty among decision makers. Although formal methods for assessment of the balance of costs and benefits of a new study (such as VOI approaches) are important,6 they may also be difficult in many current PH contexts. For example, in PH, decision modelling to predict the effects of interventions may be more complex than in clinical therapeutic contexts, covering a wider scope (from incidence to death), and may require significantly more data to populate the models. PH also takes in sectors other than the health sector, and so, the impacts of PH programmes may need to be modelled in terms of housing, employment, crime and other improvements, as well as their health effects. The available data may not be framed in ways that can be easily used; as noted, there are fewer trials, and there is greater emphasis on the use of non-experimental, observational and qualitative evidence. There is also the problem of incommensurability, that is, the difficulty in valuing/weighting different outcomes across different sectors in the same units, and the need to consider equity issues.
As an alternative, or in some cases as a preliminary to formal VOI approaches, we present below an approach based on a flow chart to help PH decision makers at all levels consider whether a new study is needed, and if so, what sort of study is most worthwhile. We have focused here on the need for more RCTs because that is where the PH ‘evidence gap’ is often suggested to lie; there have been many papers and reports about the lack of trials, but few or none about the lack of qualitative research, or observational research. First, however, we consider what influences the perceived usefulness of research.
Would better evidence be used?
One key audience for any new PH RCTs and other studies is policymakers. We therefore need to consider whether new research, if available, would actually be used by this group. Here, the evidence is mixed: robust research methodology is not always an important consideration in whether policymakers use research findings or not. A systematic review examining health policymakers' use of evidence found that the main facilitators of research evidence in policymaking are personal contact, timeliness, clear recommendations, good quality research, confirmatory research and community pressure for research.7 By contrast, research that included effectiveness data (such as RCTs) was mentioned in only 3 studies out of 24.
The perceived value of RCTs also varies between sectors. One UK policy advisor, experienced in using RCTs, and an advocate of their wider use for social policy, has noted: ‘By and large, methodology is a weak influence, in the sense that policy makers don't really tend to weigh up research evidence in terms of the strength of the source, it's much more the signal that they're interested in…The influence of research findings is primarily a function of those findings, rather than the methods’.8
This observation is consistent with a growing literature showing that in PH, as in other fields, the use of evidence is influenced by many factors, few of which are methodological. This leaves researchers and research commissioners with a challenge. There is ample evidence that the PH evidence base is weak, but new research should not simply contribute to what has been termed avoidable waste in the production of research.9
When do we have ‘enough’ evidence?
In some cases, it will be clear that we already have ‘enough’ evidence, even if we have no trials; where the signal is large relative to the background noise, observational data alone may suffice.10 In practical terms, a high signal to noise ratio can be expected when the effect size is large and when there is rapidity of change in subjective and objective outcome measures. An RCT to detect such a clear effect is probably not required, while in other situations, an RCT may be feasible but may be neither necessary or appropriate.11
We also need to consider the strength, as well as the type of evidence that is required. For example, the NICE methods guidance on PH evidence describes the most appropriate evidence to inform different types of PH decision12 (table 1).
Appropriateness of different types of evidence for answering public health questions (reproduced with permission, National Institute for Health and Clinical Excellence (NICE), 2009)
Formal approaches to prioritising the collection of future PH evidence
Although the literature on how policymakers use evidence goes back more than a century,13 it is largely concerned with the use of existing evidence, rather than with how future research priorities may be established. A hierarchy of methods for considering the value of a new study from it already exists.14 15 This runs from an unstructured thought experiment, through deterministic and stochastic simulation of new study, to formal VOI methods.6 Systematic reviews have also often been advocated as an effective means of identifying needs for further research.14 However, although they embody formal methods for summarising existing evidence, the process of identifying the need for more evidence thereafter remains largely undefined.
More formal methods have received less attention in practice. One of the most promising approaches, VOI methods, allows formalised and quantified identification of the contribution of particular pieces of information in decision models. The potential costs of obtaining sample information in a further study and the potential benefits of information so gained may be considered explicitly in terms of improved precision.
Nonetheless, problems arise in applying current VOI approaches in PH. The impacts of PH programmes often need to be modelled in terms of effects in housing, employment, crime and other sectors, as well as their health effects. Problems of incommensurability result: valuing/weighting different outcomes across different sectors in the same units is required. Equity issues also need to be considered,16 though in principle exploration of differences between subgroups is possible.17 In PH, decision modelling to predict the effects of interventions may be more complex than in clinical therapeutic contexts, requiring significantly more data to populate the models. Finally, few researchers and policymakers are likely to have the necessary capacity, skills and resources to conduct some VOI analyses.18
Towards an informal approach to assessing the value of new information
Where a full VOI analysis is not feasible, or not regarded as worthwhile, a structured approach to deciding whether a new study is needed in PH contexts can be helpful. Here, a flow chart may yield some of the benefits of the VOI approach more easily. The example we have developed is targeted principally at PH decision makers in policy and implementation contexts but will also be useful to commissioners and funders of PH research, and researchers themselves, to inform decisions about future research activities.
Use of the flow chart (figure 1) requires making explicit (1) the user's information needs and (2) the intended use of the new information that the study will produce. The added value of the evidence to be derived from the new study and the levels of precision, bias and generalisability required are then considered.
Generic flow chart. RCT, randomised controlled trial. (Continued)
Use of the flow chart is briefly illustrated, first in generic form (figure 1) and then in a worked example, showing how it may be used in deciding whether a new study should be commissioned to evaluate the health impact of allowing motorcycles to use bus lanes in London. (A further two examples will be made available in the web-only materials relating to (1) introducing new environmental standards in residential homes for elderly people and (2) to exercise prescribing by general practitioners.)
The generic framework
The framework (figure 1) begins by asking the user to consider whether the intervention is really ready to be evaluated; since evaluation at too early a stage in their development has been suggested as a reason why PH interventions fail to demonstrate effectiveness.19 A range of approaches have been developed to help evaluators assess the ‘evaluation readiness’ of new interventions.20 After this, the framework encourages the user to think through what they may learn from a new trial, about first the health impacts (see Q1 in figure 1) and then the non-health impacts of the new intervention (Q2 in figure 1). The latter could include impacts on employment or educational opportunities. For example, an intervention that promotes physical activity in people unable to work may improve individual health, but it may also increase an individual's ability to return to work—which may in turn have health impacts.
The user is also asked to consider whether, and how, ambiguous and/or unambiguous research findings from the study would affect future decision making. If their decision would not be affected by an ambiguous research finding, then the value of future information from a new RCT may be low. However, evidence from some other study design may be valuable, and so users are routed to table 2 (adapted from Muir Gray21) to consider what other research options are available to them. If, having progressed through the flow chart, the user still feels that a new RCT will be of value, then other considerations are presented in the (blue) commentary boxes, such as the need to consider healthy equity and adverse effects.
Appropriateness of different research designs to different types of question (adapted from Muir Gray21)
Worked example: health impacts of allowing motorcycles to use bus lanes
Online figure 1 shows an application of the flow chart to a hypothetical transport policy question: ‘Should an RCT be commissioned to evaluate the impact of allowing motorcyclists to use bus lanes in London?’ To start with, the user is asked whether this traffic management intervention is at an appropriate stage for a trial evaluation. The evaluability issues could include the practicality of doing a trial in different parts of London and the resources required. The user is directed through the flow chart if, and only if, they deem that the intervention is appropriate for a trial. Otherwise they are invited to consider other study options such as conducting interviews with stakeholders or carrying out reviews to check whether relevant evidence is available, say, from trials in other European cities (such as Rome or Athens where motorcycles are in common use).
If users regard the intervention as appropriate for evaluation in a trial, they are next asked whether they consider that the trial is required in order to determine the health impacts. If so, they are then invited to consider how to estimate those impacts, say in terms of the rate and severity of traffic-related injuries, and whether to disaggregate these impacts across different road users (cyclists, pedestrians, motorcyclists). If not concerned with the future health impacts, users are directed towards consideration of non-health impacts, such as the costs of implementation, the impact on traffic congestion and the acceptability of the intervention to road users.
Whether interested in the health impacts or the non-health impacts, users are asked in each case to imagine what an unambiguous finding and an ambiguous finding would look like. An unambiguous finding about health impacts might include the reduction of traffic-related injuries for all road users. Non-health impacts might include finding the intervention is cost-effective, reduces traffic congestion and is well received by all road users. On the other hand, an ambiguous finding about health impacts could arise where the intervention reduces traffic-related injuries for one type of road users (eg, motorcyclists) but increases it for another (eg, cyclists); ambiguous non-health impacts could arise if the intervention reduces traffic congestion but is costly and unacceptable to London bus drivers.
The flow chart will guide users in deciding whether the trial would be of value (for evaluating health and/or non-health impacts). If not, they are asked to consider other study options or to re-assess the outcomes, perhaps through assigning relative weights to each of the outcomes following elicitation of the views of stakeholders, say, before going through the flow chart again.
Discussion
In this paper, we have presented a flow chart for enacting an informal VOI-like approach to deciding when a new PH trial is needed. Formulation as a decision problem emphasises the importance of the decision determining what information is needed to make it, rather than allowing the set of information available to determine what decisions are made.22
In most cases, we expect the flow chart will need to be used iteratively; a first pass through the chart will reveal issues—perhaps health outcomes or costs, say—not initially considered and hence needing to be worked through in a second pass, and so on. In most cases, it will be helpful to draw a decision tree summarising the outcomes in the example. As we have already indicated, it is not claimed that the informal approach here will reach the same finishing point as a VOI analysis or indeed a multi-criteria decision analysis approach23; however, in some cases, it may be apparent that one decision option dominates the others since it provides better benefits at lower cost.24 As always in contexts such as these, sensitivity analyses to check assumptions made are essential; it may be that here informal critical threshold analyses can be performed to indicate how extreme cost or benefit estimates would need to be to throw the optimality of a particular decision into doubt.24 Another result of using the approach here may be that a more formal VOI analysis will be seen to be worthwhile.
We are not, of course, suggesting that our flow chart approach is technically the equivalent of VOI methods. Nonetheless, we do regard it as representing a valuable perspective and process to adopt, and more likely to be feasible than VOI per se, at least in PH. In almost all cases, the structured approach will be more revealing than an unstructured thought experiment as the basis for decisions about a new study.
In seeking to keep the flow chart as simple as possible, we will inevitably have omitted or oversimplified some issues, and in focusing on evidence of effectiveness, we have not discussed the importance of context, and the likelihood that policymakers will require not just evidence of whether interventions work, but also whether, and how they work in different contexts.12 To aid in its development as an effective tool, we invite users of it from a variety of perspectives and contexts to review it, to use it in practice and to send us their comments, either directly by email or on a blog set up for this purpose: http://publichealthevidence.blogspot.com/.
What is already known on this subject
There is ongoing debate about the need for more robust public health research, particularly more RCTs.
However, researchers and commissioners of new research also need simple and effective ways of assessing the potential gains from any new research.
Current approaches, such as VOI approaches to estimating the payback from new studies, can be technically challenging and are not transparent to users.
An additional approach is needed.
What this study adds
We propose a novel flow chart-based approach to assessing when new research is needed, and what type of study may be most useful.
It incorporates consideration of health and non-health outcomes and may be of particular use in decisions about the value of public health RCTs.
In almost all cases, the structured approach we propose will be more revealing than an unstructured ‘thought experiment’ as the basis for decisions about a new study.
Acknowledgments
This paper is based in part on work done when DRJ was on sabbatical leave from the University of Leicester in 2009.
References
Footnotes
Competing interests MP and DRJ are members of the National Institute of Health Research Public Health Research Funding Board; DRJ was previously a member of the NICE Public Health Interventions Advisory Board.
Provenance and peer review Not commissioned; externally peer reviewed.