Evaluating the health effects of social interventions

Design 1: randomisation of the intervention

Under a randomised controlled design successful claimants would be randomised immediately after the adjudication decision by the benefits agency. Those in the control group would have their benefit delayed by one year, and those in the intervention group would receive the benefit immediately. This design would ensure that the health status and benefit eligibility of both groups were comparable at baseline. However, the research group considered this design unethical because of the deliberate withholding of an economic benefit, which would also be unacceptable to participants. This design was therefore abandoned.

Design 2: randomising to waiting list

The introduction of a three month waiting list between initial assessment by a nurse and assessment by the welfare rights officer provided an opportunity for random allocation to the control and intervention group. We obtained approval to randomise the clients to a waiting list of a maximum of three months from the Benefits Agency, which provides the welfare rights officer. Thus, elderly clients referred by the nurse to the welfare rights officer could have been randomised to receive the visit either immediately (the intervention group) or after three months (the control group).

This design would have allowed us to compare the groups at the desired time points and provided a directly comparable control group in terms of health status and benefit eligibility. However, it randomises the benefit assessment and not the intervention of interest (receipt of the benefit), and a delay of three months would probably not be long enough to detect important health differences between the two groups. More importantly, it is unlikely to be ethically acceptable to request that study participants, already assessed to be in need of an economic benefit, accept a 50% chance of delaying the application process for three months in the interests of research. We therefore rejected this design.

Design 3: non-randomised controlled trial

A third potential design entailed identifying a non-randomised control group from a nearby area with a similar sociodemographic composition but with no welfare rights officers. In this design, community nurses would have screened potential underclaimants in the control area, who would then have been offered a standard leaflet on how to apply for attendance allowance (a nominal intervention corresponding to “usual care”). This design would have eliminated some of the ethical concerns associated with randomisation and delaying the receipt of benefit, and would have achieved an intermediate level of internal validity by retaining a comparison with a control group. However, recruitment and retention of this control group raises problems.

The success of this design depends on participants in the control group delaying their claim for the duration of the study. Although the effectiveness of the “usual care” intervention, the leaflet, is normally poor, we considered it unlikely that this would be the case after assessment for the study as participants are made aware of their potential eligibility for the benefit. We thought it unacceptable to request that participants delay claiming the additional benefit after drawing attention to their eligibility.

Design 4: uncontrolled study

A before and after study of a group of benefit recipients would be more ethically acceptable, but it would be more difficult to attribute any observed change in health status to the intervention alone. We applied for research funding for a study based on this design, citing the practical and ethical difficulties in designing a randomised controlled trial, but the application was rejected mainly because of the lack of a control group. We presume that the underlying assumption was that such an uncontrolled study would be so biased as to provide no useful information.

Randomisation

Although judgments about equipoise have recently been challenged,9 equipoise around the primary clinical outcome has been the ethical justification for randomising clinical interventions.10 11 Equipoise implies uncertainty around the distribution of costs and benefits between two interventions. Designing a randomised study may be simple in theory, but in cases where the equipoise is around uncertain indirect health impacts, and the primary economic or social impacts seem certain, true equipoise is unlikely and randomisation may be unethical.

Randomising a control group need not always present ethical hurdles. There may be inherent delays in rolling out a new or reformed programme across an area, or an intervention may require rationing or be subject to long waiting lists. These delays may provide ethical and pragmatic opportunities for randomisation12; indeed, randomisation may be the fairest means of rationing an intervention.13 However, delaying access to a tangible benefit for individuals who are assessed as “in need” may not be justifiable on research grounds.

Generating evidence for healthy public policy

An urgent need remains for studies of the indirect health effects of social interventions to improve our understanding of the mechanisms by which health effects can be achieved.14 Attention has already been drawn to the need for careful design of evaluations of complex public health interventions,15^–17 but guidance for evaluating the indirect health impacts of social interventions may require further consideration in light of the issues outlined above. For example, when the direct effects are obvious, randomised controlled trials may be unnecessary and inappropriate.18 In health technology assessment, other study designs have an important role in development15 19 and in helping to detect secondary effects.18 For example, new drugs with established pharmacological mechanisms are investigated at increasing levels of internal and external validity before being tested in a population level randomised controlled trial. Phase I and II studies are often small and uncontrolled, but they help to establish positive and negative effects, clarify the dose-response relations, and provide the background for larger trials.15 19 In addition, once approved, drugs are closely monitored at a population level to detect previously unidentified secondary adverse effects that may outweigh the primary positive effects.18 Our pilot study was similar to a phase II study.

This matching of study designs to the level of development and knowledge of the effects of an intervention could be usefully applied to the study of social interventions. Non-randomised and uncontrolled studies could be used to shed light on the nature and possible size of health effects in practice, to illustrate mechanisms, and to establish plausible outcomes.18 Such studies may serve as a precursor to experimental studies when these are ethically justifiable and appropriate. However, when randomised studies are not possible, we may have to accept data from uncontrolled studies as good enough, given the huge gaps in our knowledge.14 We need to reconsider what sort of evidence is required, how this should be assembled and for what purpose, and the trade offs between bias and utility so that study designs that are acceptable to research participants, users, and funders can be agreed.