What is the ‘added value’ of RCTs to users compared with other study designs?

All interviewees were asked about the extent to which the methodological robustness of RCTs is valued compared with other study designs and other types of information.

One policy advisor in education spoke in terms of a “Sliding scale; at the bottom end an unvalidated advocacy message…might instigate further research, but the scale goes from this to the…RCT…or systematic review or meta-analysis.”

A senior researcher and user of research in public health, experienced in evaluating policy and in liaising with policy makers, spoke of the power of trials to influence policy makers: “I would tend naturally to have more confidence in the results,…assuming that it wasn't just the design but the implementation of it that was satisfactory…the advantage is that even politicians would tend to be influenced by something that was convincingly a controlled trial.”

Another policy advisor in social care concurred, noting the advantages over observational methods: “…it's difficult to get the high quality of analysis other than through an RCT—where possible it should be an RCT.”

A senior manager responsible for policy development in the field of education also claimed that well-controlled experiments: “…do tend to solve arguments…. People respect them.” However, s/he went on to note that this did not always apply to researchers in government “Up until a wee while ago, our research division was very unsympathetic, if not downright antagonistic to, randomised trials.” A related point was made by a social and public policy adviser, to the effect that researchers do not give policy makers clear advice on when to use particular research methods, and experiments were often downplayed: “Policy makers are getting…rather muffled messages about when to do a trial or…when some other method will do…so it's hardly surprising that they are…quite happy to…go on using weaker methods,…they're not getting a clear steer…the choices they're given don't involve the option of running a trial. …There are…quite influential papers about the methodology in evaluating social interventions which give policy makers a lot of rope to hang themselves.”

However, in the experience of a senior education/health researcher, who had designed a number of innovative RCTs of social interventions, it was not trial methodology but instead the ability to attribute costs and benefits to interventions that mattered: “It's still the calculus of policy.” Asked whether the findings of the RCT alone would have had the same weight, s/he continued “some, but the cost-benefit analysis was the big factor…a number of ministers have said it made a big difference to them being able to argue the case.”

Sound methods ≠ useable findings

The senior official in education quoted above was someone who needed to use research but was critical of the sort of research knowledge s/he received: “A lot of research knowledge is not helpful from my perspective.” While this might seem to imply that ‘stronger’ methods would be more useful, this was not necessarily the case. Even among vigorous advocates of trials, it was clear that sound methods did not necessarily lead to useable findings: “As you move up that scale, there [are] more useful ‘findings’ in terms of their scientific validity. Whether they translate into a discourse that you can hold with policy makers is another question.”

A policy advisor, experienced in using and commissioning RCTs in the field of healthcare, and an advocate of their wider use for social policy concurred: “By and large, methodology is a weak influence in the sense that policy makers don't really tend to weigh up research evidence in terms of the strength of the source, it's much more the signal that they're interested in…”

S/he felt that policy makers tended to prefer “very small scale studies, pilots, rather informal evaluation evidence where it supports what they're interested in doing, and [they are]…quite resistant to the much stronger evidence where it doesn't support what they think.”

Another policy advisor, who had worked in several government departments and was a strong advocate of the greater use of social experiments, was similarly less than optimistic about the impact of trials: “Certainly in [my country], the power of a story beats almost anything…If researchers would find a story to tell about their RCT, or personalise it…If you're dealing with…politicians, you have to…appeal with a story.”

One reason for the underuse of RCTs which emerged from the interviews was that paradoxically, the straight answers described by some interviewees as useful in settling questions could be perceived as unhelpful—particularly when they show that favoured interventions do not ‘work’: “…if the results tell you that your intervention isn't working then you're in trouble…to some extent, people would rather have vaguer information about processes, which…carries less risk…I mean, people like the idea of the process of continuous quality improvement with evaluation, contributing something to improve the way you implement your…new policy or your intervention, and I think, to some extent, that's preferred to evidence which…tell(s) you pretty starkly that you ought to stop and that you're wasting public money.”

A government research commissioner in the area of social care observed: “You can't necessarily say it's one type of research over another type, because the type of design depends on the kind of research question you're asking. Different types of research will be valuable in different contexts.”

Specific barriers to and facilitators of the use of RCTs, as opposed to evidence more generally, about which much has already been written12 ,13 were also identified. The perceived lack of flexibility of RCTs, particularly in relation to the adaptability of programmes by practitioners, plus high costs and long timescales were referred to in several cases.

Interviewees suggested that RCTs are underused because users are more interested in evaluation being used strategically to demonstrate policy interest: “There are quite complicated reasons for commissioning evaluations and they're not all about testing how things work. A lot of them are to do with…demonstrating that you are taking the issue seriously.”

This political function of evaluation was highlighted by a former Treasury official who pointed to the importance of presentation and language: “There was ministerial reluctance to appear to be just trying something out, or not to give something to one group who might be equally eligible. We did point out that…we did this kind of thing anyway in practice, but they were happier if we called it a pilot rather than an experiment or a trial.”

One UK policy adviser made an interesting point about the intellectual background of politicians: “versed in the law and advocacy and case study and precedent, rather than science…”

Asked for any examples of unsuccessful attempts to set up policy RCTs, an interviewee drew on the example of an early years intervention involving early education, childcare, health and family support where the evaluation funded was not a RCT: “All the scientists were saying it should be an RCT, but…in this country, the service delivery people don't have the faith that it's important to evaluate things in a very rigorous way, and they felt it was more important to have services which could be adapted to local situations.”

The use of experiments by different sectors/countries

One policy maker who had worked across different sectors noted that: “Health…says it takes—randomised trials much more seriously than other sectors. It certainly takes evidence more seriously…[Problem area] certainly doesn't rely on high quality [evidence], in this sector, we do things, futile things, inappropriate things, [more] than even the health sector or the housing sector or any other, because we're always in a rush. Nobody [in this area] prides themselves on being an expert [in the] evidence-based sector…It's a bizarre system…it's much more dysfunctional than health…the primacy of research is not there.”

A commissioner and user of research suggested, however, that while there was apparently greater use of RCTs in health, this was accompanied by a range of other types of evidence: “Public health professionals in particular are used to working without RCT evidence, for example when there is a disease outbreak, or some other crisis. There, decisions tend to be driven by theoretical constructs tested in related health issues, by basic biological evidence, plus aetiological evidence, and evidence of what's going on in the community, plus evidence on behaviour change—that is, a series of sources of information/evidence.”

Reflecting on the lesser use made of RCTs in sectors other than health, another interviewee said that in [country], RCTs are used as a tool to cut funding, rather than to simply identify ‘what works’:

“The [country] Department of Education…led to a push for more RCTs in education…Now, RCTs and systematic reviews are used as a way of cutting programmes—where there is no good evidence from RCTs, it is used to justify a cut.” S/he expanded this theme of cultural differences in use of experimentation, drawing comparisons between countries: “[It] relates to levels of affluence and the degree of development of the scientific community but in Europe my impression is that Northern Europe has done a lot more in terms of social and health research of an organised type…In the Scandinavian countries you have a tradition of really being…organised and imposing quite high degrees of control over the population in terms of what people can and can't do, and gathering a lot of data on a large scale…whereas in Southern Europe, they have less of a history of social public health trials. The United States and Canada in terms of volume (not necessarily quality) is far ahead of the rest. Australia has done quite a bit given the size of the country as well.”

S/he went on to explain why this may be so; in some countries, evaluation is important for public accountability: “In [our country], we have a very strong tradition of evaluation research and population-based epidemiology. This is linked to the need for accountability for performance—as opposed to seeking harder evidence of effectiveness…”

Another interviewee noted that educational researchers' “methodological tool-kits tend to be in other areas” and that they assume that RCTs are only of value in medicine.

The single trialist we interviewed noted that many countries placed more weight on studies from abroad, and publication in a US journal was often more prestigious. S/he suggested that one argument posed against trials in the education setting was that it is not fair to deprive someone of an intervention which they perceive as being effective.

The US emphasis on trials was also underlined by a former policy maker: “Very strong in the US and possibly in Canada—I have that impression. We're somewhere in the middle. The Europeans are nowhere. There's no interest in continental Europe. It's an Anglo-Saxon disease.”

Training in appropriate skills was identified as a problem in several cases, for example, a UK Research Commissioner said: “I think I find it quite dispiriting that in America, they will invest in these really rigorous studies, and yet in this country, we don't. There's a problem with research capability in this country because people don't develop the skills to do it.”

Finally, interviewees reflected on what additional information is needed beyond RCTs. Suggestions included studies that permit comparisons between countries and studies which describe context: “Useful information includes studies that illuminate the extent and the nature of the problem. [Our country] pays a lot of attention to the PISA study, by the OECD, which includes 40 countries…comparative studies are helpful.”