Imagine you are a policymaker and you need to develop a policy on immigration.
Your personal view, and that of your colleagues, is pro-immigration. Your boss is putting pressure on you to come up with a solid, persuasive argument to convince Parliament, and the public, that immigration is good for the country.
Two policy briefings land on your desk, with opposing conclusions. One cites research reflecting favourably on immigrants, with figures indicating the economic benefits they bring. The other references studies showing certain immigrant groups negatively impact the UK economy, receiving more in benefits than they contribute in taxes.
Do you:
- (a) conclude that there is no scientific consensus on the matter, and decide against using either briefing as evidence to inform your policy
- (b) search PubMed for a meta-analysis that will clear things up
- (c) use the briefing that backs up your view as evidence to support a pro-immigration policy, and disregard the evidence in the other briefing?
It may not surprise you that option (c) can be rather appealing. Although this thought experiment may reveal a hint of scepticism regarding how evidence informs policy, it is worth bearing in mind the competing pressures that policymakers face. When devising policies, they have to take into account budgets, party views, public pressure and more. Research evidence has to compete with all of these factors to successfully inform policy.
The need for transparency and reproducibility in research
Academic research generates evidence and insights with great potential to maximise the impact and effectiveness of policymaking – but only under the right circumstances. When confronted with a conflicting or inaccessible research base, alongside competing pressures, policymakers will struggle to use evidence effectively. In contrast, when the research base is transparent and reproducible, this provides more clarity on the evidence base. Policymakers are then better placed to inform policy with evidence.
Transparency ensures that all research findings on a topic can be accessed by researchers, policymakers and the public. This provides a more unbiased, comprehensive picture of the current state of knowledge. Reproducibility acts as a stamp of credibility, ensuring that research findings can be trusted with more certainty.
Key definitions
Research transparency means that research methods, analysis and data are reported and disseminated openly (free of charge), clearly and comprehensively.
Research findings are “reproducible” when independently repeating a study using the same methods and data generates the same results.
Case study: Tamiflu
Governments have spent billions of pounds stockpiling Tamiflu: since 1999, sales of the drug have amounted to over £11 billion. The UK Government spent £424 million on 40 million doses of Tamiflu, while the US stockpiled 65 million doses at a cost of $1.3 billion. By 2009, 96 countries had stockpiled enough Tamiflu to treat 350 million people.
A win for infectious disease preparedness? Unfortunately not. Criticisms of the evidence for Tamiflu’s effectiveness began to surface, including concerns that several trials of the drug had not been published. In other words, much of the evidence base was not transparent, and could not be scrutinised.
Eventually there was a comprehensive review of all of the trials. It concluded that there was no convincing evidence that Tamiflu reduced the risk of flu infections or complications.
If steps had been taken to ensure transparency in all of the trials, it is unlikely that £11 billion would have been spent on Tamiflu. Transparent research methods and data would also have enabled other researchers to repeat the studies, to test their reproducibility – and move closer to a true picture of Tamiflu’s effects.
Why do studies reach such different results?
How come studies on the same topic can reach such differing conclusions? Returning to our earlier example of research on immigration, tweaking assumptions can significantly change the findings. Professor Christian Dustmann, of UCL Economics, and colleagues looked at this in a 2016 paper refreshingly titled, “The Impact of Immigration: Why Do Studies Reach Such Different Results?”.
Dustmann noted that some analyses assume that immigrants and natives who have the same skill level compete with each other for jobs. However, this overlooks evidence – across countries including the UK, US and Germany – that immigrants tend to “downgrade” and instead compete for lower-skill jobs.
As a result, analyses based on this flawed assumption tend to overstate the negative impact of immigration, including the impact on natives’ wages. I need not spell out how such analyses could be cherry picked in the context of political debate.
It is therefore no surprise that, in a 2019 report on reproducibility, the US National Academies advised against making policy decisions based on a single study’s results. Indeed, evidence synthesis has been used for some time to safeguard against excess focus on single studies.
What if there is no scientific consensus on a topic?
In research disciplines where reproducibility is important, ideally all research would be reproducible. In practice, there will always be some variation in findings on a topic, and this does not necessarily mean that findings are invalid or unusable for policy. Sometimes there is no consensus among the academic community. However, transparency exposes the variation in research results, giving policymakers access to a shared, comprehensive set of evidence. While some policy briefings may still cherry pick evidence to support a particular view, transparency enables reflection on the wider evidence base and the debunking of false assertions – as is often done well by fact-checking sites.
Transparency and reproducibility in a post-truth world
Policymakers and the public must trust research for it to inform policy. When numerous studies are not reproducible, or evidence is biased by hidden data, doubt is cast on whether research findings can be trusted. In the context of the public supposedly having “had enough of experts”, it is especially crucial that academic research ups its game.
Transparency and reproducibility are essential to ensure trust in research, and strengthen the voice of evidence in the complex realm of policymaking.
What universities can do to promote transparency and reproducibility
So how can universities foster transparency and reproducibility? Awareness raising, training and incentives are key. These should encourage and reward researchers who: make research outputs open and data FAIR; pre-register their studies; make their statistical methods transparent and more.
As initiatives in support of the transparency and reproducibility agenda gain momentum – notably the UK Reproducibility Network – there are reasons to be optimistic about strengthening trust in research.
