More than a hundred years ago, the journal Nature called for “a scientific approach to government” in a series of editorials calling for politicians to borrow the methods of science for the common good. Now, advocates of successful randomised controlled trials (RCTs) say the time has come to put aside the political rhetoric and see what actually works by conducting experiments on a societal scale in areas from international development to employment.
“RCTs are the only method that we can hope to generate a bias free answer,” says David Torgerson, Director of the York Trials Unit in the University of York, UK, and co-author of a policy paper championing the benefits of the approach. “There is an almost limitless number of questions that could be answered by using RCTs that presently remain unknown,” he says. The case for countrywide RCTs has been building for some time. In the UK, regions taking up back-to-work employment schemes were compared with others maintaining the status quo. The results revealed a new programme was an expensive flop; it was halted early, saving the taxpayer millions of pounds. Similarly, a study in Switzerland found that randomising girls into girl-only classes versus mixed classes increased maths scores among girls.
In fact, it is not even the result that matters, rather the fact that the trial exists. Economist Zohal Hessami from the University of Konstanz, Germany, who co-conducted the Swiss study, says RCTs offer many advantages over purely observational data. “Most importantly, cause and effect can be cleanly identified from the data,” she says. However, Hessami cautions against policy-makers basing their decisions on RCT results alone. “These studies are often conducted in one specific setting and suffer from small sample sizes.”
In terms of science policy, RCTs wouldn’t be suitable for expensive gala projects such as the Large Hadron Collider that could not be replicated. Grants, too, are a meritocracy and unsuited to the random assignments of trials. RCTs are best for smaller scale projects that involve people, and could be organised by social media. This would apply to academic mentoring at degree or PhD level can make the difference between a lifetime’s inspiration and throwing in the towel early. Academic awards could be scrutinised too by looking at whether they reward and inspire or they are just a way of throwing money at people and projects that are likely to be successful anyway?
Critics of RCTs say they are costly and take too much time to deliver results that are likely to be more ambiguous than a ‘yes or no’ answer obtained by RTCs testing whether a vaccination works. And politicians in particular dislike the random lottery aspect of clinical trials, citing the method creates a divide of ‘haves’ and ‘havenots’; the latter being at a disadvantage in the competitive worlds of scientific education, academia and business. True, but if a policy does not work, do you not have a whole population of havenots?
Then the question of scale arises – within countries or between them? Thibaut Lery, Senior Science Officer in charge of Evaluation Services at the European Science Foundation, agrees that governments frequently introduce drastic changes that affect our lives usually with little evidence underpinning the new policy. But he says RCTs have limitations too. “There is always the possibility of contamination of control groups when estimating the effects of system-wide reform,” he says. “RCTs may not pick up the feedback or macro-effects that would occur if an experimental program were implemented at European level or nation-wide.”
Another objection is that RCTs by nature run against the grain of the political machine. Politicians (and policy makers to some extent) like to think they are right because of their education, ideological position or personal experience; they have a lot to lose by scientifically testing whether their carefully crafted manifesto will actually work.
Nonetheless, there is interest in RCTs at the top levels of the European Commission (EC). “RCTs are certainly a valid approach in terms of support to policy making, and there is some work, it seems, going on in our Joint Research Centre to use behavioural science in support of policy making at the EU level,” says Michael Jennings, EC spokesperson for Research, Innovation and Science based in Brussels, Belgium.
He says they have looked at the performance of entities participating in EC programmes versus those who do not, or participate less. This could be helpful in deciding on future policy focus, but he cites cost, time and that it is not a true RCT as limiting factors. However, he says there may be areas where RCTs might work. “For instance testing new tools for dissemination of results, or other areas on the administrative side of projects, amongst those already receiving a grant.”
The centralised and scientific approach to government that Nature called for a century ago may still face obstacles. But with social media mobilising a new generation of digital natives and continent-wide data collection under way via smartphones, society-wide experiments are probably not too far away.
He also has experience with Nature Publishing Group and Science Photo Library and also works as a Digital Content Producer at BBSRC.
- Top Trumped: what does the US election mean for science and Europe? - 20 December, 2016
- Sweet tooth: countering one of our most lethal addictions - 9 March, 2016
- Policy matters: transparency is rarely a bad thing - 7 October, 2015
EuroScientist journal 