Thursday will see the publication of the results of a septennial assessment exercise, with the findings set to be used in the calculation of funding allocations for a small proportion of university income linked to research quality.
This data is based on the gold standard of research metrics – the results of considered review by peers with a close interest, and deep understanding of, the subject matter in question. The process has been thoroughly described and clarified via a series of publications that between them answer any conceivable question you may have and several that you would never have thought of. What will emerge on 12 May is the culmination of eight years of hard work (involving six years worth of published research) – through a global pandemic and alongside day-to-day responsibilities – by researchers, research managers and support staff, public servants, and reviewers.
Globally, the process is much envied and often copied or adapted. The data is as robust as it has been possible to make it – recent changes have reduced the value of some of the “game playing” that providers submitting are often accused of. As much as anything can be, it represents a fair and considered peer assessment – suitable for use in funding allocations – of the standing of research in every one of 36 broad subject areas at each university or other research performing organisation that made a submission.
And yet…
It’s the REF. The Research Excellence framework. The source of more anxiety, sleepless nights, weekends of work at the kitchen table, lost opportunities, failed relationships, and personal doubt than anything else in higher education.
It’s a foundational myth underpinning the lived experience of academic life. A source of career validation, and a proxy for every other possible measure of research from social value to economic value to a nebulous sense of personal or strategic fit. Any university will have held countless meetings about REF, and even at meetings about other matters it always exists on the margins.
REF requires an army of institutional appointments and offers many opportunities for consultants. Some readers will have undergone a “mock REF” at department, faculty, or university level (some unlucky souls will have done all three) – nearly every academic will have been subject to some degree of analysis of their degree of responsibility for research or the quality of their outputs with a view to to the design of a REF submission.
It’s not a stretch to say that every single person connected to a higher education provider will see some impact on their own lives from decisions made as a result of the REF results. Priorities will be changed, teams broken up, departments reformed or even closed entirely. Some staff will lose their jobs, others will gain responsibilities or will be set targets that detrimentally affect their health. A good REF performance- to some – will become a valued designation of honour, and its absence a badge of shame.
In a nutshell: how REF works
Around every seven years, the four UK funding agencies (in this case Research England, the Scottish Funding Council, HEFCW, and the Department of the Economy in Northern Ireland) bring together a number of academic panels to take a snapshot of the quality and location of research conducted in UK higher education institutions that award research degrees. The REF 2021 iteration I describe below departs from previous practice in many ways – every iteration of this exercise has been different, making comparisons and timeseries a minefield of caveats and complications.
For each Unit of Assessment (UoA) they employ research active staff in in, institutions must submit a number of research outputs (often journal articles, but can be almost anything produced in the course of research), and information on the impact of research (what happened to it outside academia). Providers also submit details on the research environment (what it is like to do research there).
The numbers of submissions required are calculated (minor exceptions notwithstanding) based on the number of researchers in a UoA in a provider (“researcher” defined as someone who spends more than 20 per cent of their time on research) as follows:
- Outputs: 2.5 per staff FTE, with a minimum of one and a maximum of five submissions per staff member employed as of the census point (31 July 2021)
- Impact case studies: One per submission, plus one per up to 15 staff FTE returned
- Research environment: One per institution.
A unit of assessment panel – composed primarily of UK academics working in the field in question – will make an assessment of each submission, according to one of five levels:
- 4*: Quality that is world leading in originality, significance, and rigour
- 3*: Quality that is internationally excellent in originality, significance, and rigour but which falls short of the highest standards of excellence
- 2*: Quality that is recognised internationally in originality, significance, and rigour.
- 1*: Quality that is recognised nationally in originality, significance, and rigour
- Unclassified: Quality that falls below the standard of nationally recognised work, or a submission that does not meet the published definition of research.
The three assessments are then combined into a single quality profile that represents the panel’s rating of provider activity in that unit of assessment. The combination is weighted – 60 per cent to outputs, 25 per cent to impact, and 15 per cent to the research environment. The UoA judgements are then moderated at “main panel” level (each UoA is part of one of four main panels) to ensure consistency.
Results day will just offer details on the overall quality profile for each UoA at each provider, along with the number of staff submitted. The detail of the panel findings is due in June – at that point providers get detailed (and confidential) feedback on their own submissions, and UoA panel overview reports will be published alongside details of submissions.
Significant investment
It’s nice to have a good quality profile as judged by your peers, but many will be keeping a close eye on the impact of these assessment on research funding income.
However, the institutional funding directly linked to REF results is, in most cases marginal. In 2019-20, Research England allocated around £1.6bn of QR associated funding – not to be sniffed at, at least until you discover that sector income overall was about £36bn for that year.
Here’s a similar calculation (for England only, remember each nation in the UK allocates research funding in its own way) showing the proportion of provider income attributable to QR. To spare blushes, I’ve not done my usual comparison with catering and accommodation income. You’ll note that two groups of providers get a reasonable proportion of their income from QR linked allocations – the small number of providers in receipt of a lot of QR funding, and very small providers with lower overall incomes.
We need to be clear that REF results are not the only component of QR calculations – the volume of current activity, sources of other research income, and the population of postgraduate research students, also play a part. QR also includes funding that contributes to research supervision, and makes a contribution towards covering the full cost of research supported by charities and industry.
What makes this income unique among research funding streams is that it is none of it is attached to any particular expectation of outcome. QR is – nominally at least – the aspect of research funding that allows for experimentation, for blue-skies approaches, and for fundamental research. Prosaically, it does a lot to maintain research capacity between funded projects.
A good REF performance can also be a useful reputational marker to aid in securing other funding. There’s no direct means of measuring this, but we can look at the proportion of income attributable to research as a proxy. In general, providers that get more as a proportion of total income from Research England QR-linked allocations also get more as a proportion of total income from research grants and contracts.
Research England currently allocates funding based on the volume of activity rated 4* or 3* – this is geared towards supporting 4* (the ratio is 4:1). It’s fair to assume that there will be no changes to this model in the near future – there’s no consultation out (which would be required in the event of a change) and no rumours of one to come.
Unlike with project (and teaching) funding there’s never been any attempt to skew QR resources towards favoured subject areas – any improvement in, say, arts research quality would be reflected in the overall mix of allocations. With the funding pot staying the same size, keeping an eye on the proportion of overall funding allocated in each subject will be interesting.
Beyond the money
The lack of a direct link to specific projects gives QR – and the REF that powers it – a totemic feel. It explains the way in which academic staff see REF as a core source of support for the work they choose to do – and the way they perceive their performance is validated by the REF is hugely important for self-worth and self-actualisation.
What the REF actually does is much more subtle and limited. Assessments are of subject areas (UoAs map closely to HESA cost codes) in providers. Before REF2021, the decision to submit a person (or not) to the REF fell to the provider – but the 2016 Stern review proposed that all research active staff should be returned to the REF.
Following numerous consultations, the rules now are that each UoA in each provider should return an average of 2.5 outputs per FTE – and at least one per member of staff employed on 31 July 2020 (the census date). Everyone who performs research – in other words – is REFable (not that this word ever really meant anything).
Though providers will – eventually – understand which particular pieces of work were accorded which grade, the central purpose of the exercise is at unit of assessment level. The Stern review was clear that the purpose of REF was also to say something about the quality of research more generally, for use in understanding national and local priorities. Certainly some providers will use REF results (as one data resource among many) to make strategic decisions about the shape and scope of research activity.
What happened last time?
I’m keen not to lean into the “research metrics as sports” mindset – to me it is clear that although both REF2014 and REF2021 are measures of research quality the methodological difference is such that you can’t realistically identify a direction of travel as by comparing the two.
But for the record and for ease of reference these are the REF2014 results by UoA:
And by provider:
The 2021 iteration will, to my mind, offer a clearer view of the full range of research activity at universities in the UK. For the first time, early career researchers will have their own work assessed – the impact of “research superstars” (whatever that is supposed to imply) will be limited, and the opportunities to game the system with selective submissions and strategic hiring are greatly diminished.
What to watch for?
With the big change this time round being the increase of impact weighting and the requirement to submit more staff to more UoAs there’s a few changes I will be keeping an eye out for:
- Will a larger pool of submissions mean we are better at identifying previously unrecognised islands of excellence? Where will these be, and in what subjects? How will this support or undermine other areas of government policy?
- Will the volume changes and the requirement to submit where staff are employed to research skew the sector’s overall profile excellence away from STEM and towards arts, humanities, and social sciences? Will there be a policy response to this?
- How will the number of staff submitted relate to the number of staff with responsibility for research in a subject area at a provider – how have Stern’s changes worked in that respect?
- How far will assessments of impact drive our understanding of what constitutes research quality? What impact will an enhanced interest in the research environment (and more broadly, research culture) have?
