What we now know about REF2021

This morning sees the release of the latest iteration of REF2021 panel criteria and submission guidance. Those awaiting a settled REF ruleset will have to wait a little longer – we’ll see a release of both in their final form in early 2019, a slight slip back against the published timeline but not one that is outside expectations set during REF2014.

For those marking off the days on a calendar, that’s just under two years before the submission deadline (27th November 2020), and a year and a half ahead of the all-important census point (31st July 2020). And there will be six months between the final rules being set out and a survey of submission intentions: a (non-binding) survey will support the REF team in allocating assessment resources in the autumn of 2019.

Fun with consultations

So, another consultation. Many of the questions merely cover the clarity and comprehensibility of the draft guidance, but there are a few more pressing issues still requiring input. These include the eligibility of seconded staff, the eligibility of staff based outside the UK, and the way the proposed treatment of staff with specific individual circumstances – for example in reducing the number of required output submissions – might give rise to an overall underrepresentation of staff with protected characteristics in output submission pools.

The ineligibility of former staff members who have been made redundant is still up for debate (remember, in REF2021 the outputs of staff who have moved institutions can be eligible for both their previous and current employer). Points around the treatment of intra-institutional cross-discipline research still need to be finalised – and there are further details of the parallel (though not assessed) survey of research activity costs to come.

With such a complex and ever-evolving system, it can be difficult to keep tabs on all the interlocking issues. Everyone following the changes will have their own key issues that they look out for – and rather than trying to cover everything I thought I’d share mine. I’ve a particular interest on the open access requirements, pedagogic research, and the use of citations in research assessment – so apologies in advance for focusing on those rather than something else.

Who is eligible for REF2021?

Institutionally speaking, any institution in the UK which awards research degrees can make a REF submission. There is even an exceptional route to participation for instances where research capacity is in development and there is a clear aim to award research degrees in future.

But most institutions submit to REF to get access to results-linked research funding, and in England there have been changes to the way this is managed. Any HEI formerly designated for HEFCE funding, or alternative providers with “university title”, or any institution in the OfS “approved” or “approved (fee cap)” categories can submit to REF. But only those in the OfS “approved (fee cap)” category can expect to see any of the linked QR money (“quality-related research funding”) from Research England. In Scotland, Wales and Northern Ireland, eligibility will continue as previously.

The most discussed change for REF2021 has been the end of the opportunity not to include certain staff within a submission. Any member of staff with a “research” or a “research and teaching” contract must be submitted as having significant responsibility for independent research. All units of assessment (UOA) where there are eligible staff must see a submission. There will be cross-checks between submissions and the HESA Staff record to identify anomalies.

In a very small number of cases, an institution may not be required to make a return to a unit of UOA where it is research active. These are usually linked to size (less than 5 FTE researchers) and history (no previous submission, or a significant change in staff profile). In all cases an argument would have to be made for why such researchers are “separate and distinct” from other UOAs.

All this harks back to the Stern review and associated concerns around game playing. The scope for what we may politely call “strategic submission” in the new system is very tightly controlled – there are a few cases where staff would not be eligible (for instance, hourly or daily paid staff contracted to work at least 0.2 FTE per year, research assistants), but institutions are expected to develop a code of practice around the way eligibility is determined, and this will also be audited.

Fans of robust identifiers will be delighted to learn that ORCID for each researcher should be submitted, where held.

Outputs, impact and delinkage

There is a tariff, based on the number of FTE submitted to a UOA, for the number of outputs that must be submitted. Each member of staff submitted will have one output attributed to them, and no single member of staff can be linked to more than five outputs. And former staff employed during the eligible period should also be returned alongside appropriate outputs.

A similar tariff exists for the number of impact case studies required – this shows some gearing, requiring more case studies for a given FTE for smaller departments than larger ones. For instance, a submission involving 20 FTE would require two case studies – one for every 10 FTE – but a submission involving 310 FTE would require 13 case studies, or one for every 23.8 FTE.

And every submission includes an environment statement, covering the “vitality and sustainability of research. There is also, new for REF2021, an institutional environment statement.

The place of citation metrics

REF is a process of expert review. Seeing this up front is a great reassurance that there are no metrics to game and no numbers to suspiciously poke at – though some sub-panels (primarily in medicine and allied subjects, physical sciences, computer sciences, and economics) will look at citation metrics. These are “sometimes, but not always, a reliable indicator” of academic significance, a component of the 60% of the assessment that focuses on “originality, significance, and rigour”. And only a component – at every stage it is human judgement rather than metrics that forms the backbone of the REF.

The enormously hardworking team who run the REF have a particular ire for institutions who rely on citation-related measures to select submissions – expressed in no uncertain terms this time round thus:

“The funding bodies do not sanction or recommend that HEIs rely on citation information to inform the selection of outputs for inclusion in their submissions.”

This includes things like Journal Impact Factor (JIF) or any calculation of a H-index. There is no place for JIF or H-indexes in REF. If anyone tries to tell you otherwise they are dangerously misinformed – though with a growth of institutional awareness of stuff like DORA, and the impact of the Forum for Responsible Metrics, this should hopefully be a less prevalent mistake than in previous rounds.

How open access?

As has made clear all along, open access publication is expected for articles and conference contributions – anything with an ISSN. REF defines Open Access as “”deposited, discoverable, free to read, download, and search within, by anyone with an internet connection”. This is a laudably high standard, but there are exemptions and exceptions:

The requirement only applies to material accepted after 1st April 2016 and published before 31 December 2020.
It explicitly does not apply to monographs, working papers, and commercially (or governmentally) confidential reports.
There are work-arounds for the long embargo periods that some publishers insist on.
There’s a curious hole around submission of work published while at a previous employer. Apparently the new employer doesn’t have to check for OA compliance – mysterious as this is hardly an onerous task and shouldn’t really involve contacting the previous institution.
There are a number of technical and process-based exemptions, covering everything from a catastrophic software failure to publisher-based delays.

But for staff members who have somehow failed to note the open access publication requirement for REF (and for most other major UK funders), failed to note that in most cases REF is not hugely interested where you publish, and failed to take account of the myriad other good reasons to publish in an OA journal, there is a five percent tolerance of non-compliance for eligible materials. This is five percent of the total pool of outputs, and may be a much greater factor governing the selection of submitted materials in non-science subjects.

Treatment of pedagogic research

There are a lot of myths about whether and how research into the practice of learning and teaching in HE can be submitted to REF, but it is really quite simple. For everything other than Main Panel A (medicine and related), pedagogic research can be submitted to any subject panel, which will then take advice from the Education panel.