A real-time evaluation of views on REF 2021 shows that the assessment exercise is a driver of researcher behaviour – so why not use that to improve research culture?
Since the late 1980s, the UK has conducted a regular quality exercise assessing research in higher education institutions (HEIs). The results guide the allocation of so-called quality related, or core, funding, to HEIs.
As significant levels of funding rest on the outcomes – typically about £2 billion of public funding per year – these regular assessments, and the institutional and academic interpretation of them, can act as a major driver of behaviour in the UK academic sector, for better or worse.
Measuring the impact
Originally envisaged as a way to drive up quality and excellence in the research system, the exercise has since diversified, with the introduction of the REF in 2014 looking at research impact as well as quality. This move was intended to both reflect the outcomes of UK research and to focus more on the impact of research on society.
REF 2021 is now under way and research submissions were made earlier this year. To understand the impact of the assessment, Research England, on behalf of the four UK funding bodies, commissioned a RAND Europe study to record in real time (i.e. while researchers and HEIs completed their submission) the attitudes of the UK’s research community towards the REF.
This study, building on a pilot from 2017, addressed a significant evidence gap in the research field as there is little systematic and nuanced evidence about how academics across the sector view the REF and how it has shaped their research and wider research in their field.
Through surveys, interviews, and focus groups, the study team was able to gain valuable insight into researcher views across disciplines, career stages, and types of institutions.
Our study shows that on average, academics have negative attitudes towards the REF, however perceptions are also very mixed. The picture very much depends on personal experiences and context.
These high-level negative views are somewhat in line with previous findings. Academics have been vocal about the REF in the past, with some considering it to be a costly and burdensome exercise.
Review recommendations
The last review of the REF, led by Nicholas Stern in 2016, suggested several changes designed to streamline the exercise to help reduce the perceived burden.
We found that the REF influences researchers and the research community across many areas in different ways. For example, most researchers believe that the REF has increased open research, the public relevance of research, and the quantity of research in the research community.
Conversely, many researchers also believe that the REF has decreased the authenticity and novelty of research in the research community.
This view of the impact on the community is interesting to compare with their views of the influence of the REF on their own research, where most researchers stated that, at an individual level, the REF has not influenced their own research. This highlights a disconnect between the perceived impact on others and the real influence individuals perceive on themselves.
Most academics thought that changes to the rules for REF 2021 were positive for them. Academics welcomed the move away from the individual towards a collective and team-based approach to assessment. For example, some felt that certain changes, to the rules around the selection of staff and outputs, may have helped to reduce people “game playing” the assessment process (though it remains a concern).
The study also shows that the wider research environment is an important focus of the REF, and the evolution of the exercise from 2014 to 2021 is broadly seen as positive in this regard.
Creating the right environment
Academics have a range of views on the way the REF could be altered following the 2021 exercise.
When asked about how the REF could be improved, there were divergent views on what the future should look like – with a focus on how to assess, as well as what and who to assess.
With the increasing focus on team science, and the move from the lone academic to a team where members play distinct roles to deliver maximum benefit, we suggest there could be a greater focus on the research process and environment in which research takes place, rather than a retrospective assessment of outputs and impacts.
Rather than functioning as a regulatory instrument, the REF could support behavioural change and ensure a greater focus on a positive working culture.
This could be done, for example, by assessing whether the department offers collegiality, a nurturing environment, mentorship, support to early career researchers, or a mechanism to encourage innovation and creativity.
Researchers, managers, and institutional leaders stressed that a greater focus on equality, diversity, and inclusion could also be important in future cycles of the REF. Single-blind peer review (where reviewers do not know the identity of the author) could be one step towards this.
Institutional responsibility is seen as being important in creating the environment and conditions within which high-quality research and impact can be achieved. This is an important shift for the REF to continue to build on.
In future, the exercise could focus on creating a culture to maximise high-quality research and impact in the 21st century, by taking a collective and team-based approach to assessment and focusing on the wider HEI environment.
We know that the REF can influence the academic community, this may be an opportunity to use that influence to drive a better, more inclusive research culture.
An Interesting report. In the main report there is similar chart that presents data on how REF has influenced an individual’s research. For all the aspects, the dominant response has been “neither negative nor positive”. So it is interesting to see how how micro perceptions is quite different macro perceptions. That would be worth explaining.