REF just got real: mist now clearing on the road to 2021

For some of us, summer ended rather abruptly at midday on 1 September with the release by HEFCE of its initial decisions on the design of REF 2021, the latest instalment in the thirty-year saga of UK research assessment.

There’s nothing like an eleven-page missive to the vice-chancellor, buttressed by twenty-five pages of technical guidance, to make one’s memories of the beach fade fast. Having sprouted a scrappy beard over August, by 1 pm, I was at my desk, clean-shaven, drafting a note for the faculty PVC. Lord Adonis, our new patron saint of unseasonal productivity, would have been proud.

As Wonkhe’s David Kernohan wrote on Friday, it can feel like we’re trapped in a perpetual REF consultation, with an end point that always lies just beyond reach. HEFCE’s drip-drip approach to confirming details, across multiple platforms, hasn’t made life easier: a webinar and blogposts on staff selection and portability of outputs in July; initial guidelines last week; now a further micro-consultation, running until the end of September. An official @REF_2021 twitter feed has gone live, and rumour has it that the subpanel chairs will be unveiled just before Christmas on Snapchat.

So it can be a challenge keeping tabs on it all, but to the credit of David Sweeney, Steven Hill and the REF team at HEFCE, the process has also been highly transparent and consultative. And they are treading a delicate line: trying to stay faithful to the spirit of the Stern Review while tempering his recommendations with the shifting priorities of ministers, the (often divergent) views of the research community, and their own practical experience of what will work.

We’re now just past the midpoint between REF 2014 and REF 2021. More than anything, universities want certainty so they can prepare with confidence for the next exercise. It may seem that there’s still a fair bit up in the air, but from the totality of what’s now in front of us, it’s pretty clear where the pieces will land. And senior figures at HEFCE underline that they expect the last details to be resolved by November.

 I won’t rehearse all that’s in the latest guidance – it’s an easy enough read, and there are helpful summaries by others here, here and here. Let me instead highlight a few headlines for planners, assessors, and the hordes of REF-assessed. Two of these are solid certs; two are likely bets.

Deep impact

Of the new certainties, the big surprise is the shift in the weighting for impact from 20% to 25% of the exercise. This was always an option – as HEFCE’s cover letter reminds us, the original plan in 2014 was for impact at 25%, which was then watered down. However, Lord Stern suggested that the weightings should remain stable, at 65% outputs, 20% impact and 15% environment. So what’s changed?

The answers lie in the political environment. Fourteen months ago, when Lord Stern published, there was no industrial strategy, Brexit debates were just starting, and the rationale for UKRI was being hotly debated through the Higher Education and Research Bill.

Now across the research system, the £4.7 billion Industrial Strategy Challenges Fund (ISCF) looms large. The dislocation between universities and some in our communities that we saw in the EU referendum – plus mounting criticisms over student fees, VC salaries and accountability – means that anything which incentivizes the sector to engage proactively with society is seen as a good idea. And Shadow-UKRI is flexing its strategic muscles, ahead of its formal launch in April, heightening pressure on Research England to show how the REF will be properly aligned.

So a strong nod to the industrial strategy makes complete sense. At the same time, HEFCE sources emphasize that there was no explicit pressure from ministers to increase the impact weighting. Nor does the reference to industrial strategy signal any retreat from the broad sweep of impacts rewarded through the last REF, or a push to privilege economic and commercial impacts over those on society, culture, public policy, health or quality of life.

Lord Stern’s call for impact to be “deepened and broadened” is helpful in this respect because it shifts focus away from hierarchies of “impact types” and instead encourages the sector to tease out with greater precision the varied pathways from underpinning research to impact. More rigour is needed here – and the top-rated 4* case studies of REF 2021 are likely to be those that describe, evidence and analyse such pathways convincingly.

An impact gold rush?

For universities, these changes raise the stakes around impact by more than the 5% uplift would at first imply. Because 2014’s impact template will now move into the environment section, the significance of the impact case studies will in effect rise by more than half, from 16% to 25% of the exercise.

This poses particular challenges for smaller submissions, which may require only two case studies, as each of these now determines up to 12.5% of the total result. As Chris Hewson, impact manager at the University of York, noted on twitter, “a poor performing case study could now seriously derail a department’s ranking.”

The shift in weighting, combined with a focus on deeper and broader pathways to impact, places a premium on proactive approaches to planning and enabling impact – both by researchers themselves and by impact managers, whose ranks will surely swell. It also sends an unequivocal signal that the impact agenda is here to stay.

Some will inevitably object to this. But bringing impact into the REF is now widely regarded – across government and the HE community – as a successful experiment that has generated a 7000-strong body of case studies, and encouraged institutions and individuals to look outwards and use public investment to make a positive difference. Could the weighting rise still further after 2021? Views differ, even among HEFCE’s top team, but smart HEIs will start preparing for this eventuality now.

Complex environments

The other new certainty is the heightened significance of the environment section. This leg of the exercise typically gets less discussion than outputs or impact, but while its weighting remains stable at 15%, HEFCE’s guidelines confirm the increased sophistication of what it will now address. This includes the overall strategy and template for impact; structures to support interdisciplinary research; collaborations with “organisations beyond HE”; and a section on “open research”, spanning open access and open data.

The environment section will also include more quantitative data – as recommended by my Metric Tide review. Precisely which indicators, and how to use them, is now the focus of ongoing work by the Forum for Responsible Research Metrics, chaired by David Price, Vice-Provost for Research at UCL (and of which I’m a member).

As with impact, university and UoA leads need to look seriously at what will be required to evidence and perform well against an expanded, more strategic and robust set of criteria. After 2021, environment may well increase in weighting as a result: one can imagine REF 2027 with outputs at 50%, impacts at 30%, and environment at 20%

Short, sharp questions

So those are the solid certainties. My two likely bets can be read between the lines of the additional consultation that launched on Friday. This has a tight timeframe – just four weeks – and an even tighter focus. It is clear that HEFCE has preferred outcomes – and now needs to check one more time if enough of the community agrees.

On staff submission and the process for identifying those with “significant responsibility” for research, there are ostensibly two options, as David Sweeney set out in July on Wonkhe. But the first of these – a “no burden” option of 100% return – is less a question than a gauntlet being lobbed at research-intensive universities, which they would be foolish not to pick up.

As Sweeney wryly notes, this option “will be available, and attractive to many institutions”. It is clear that HEFCE expects research-intensive HEIs to take the 100% option, and automatically submit all their eligible staff, to avoid the bureaucracy and complexity of alternatives. If the Russell Group goes for this as a block, it doesn’t matter how many others take the more complex option of determining who is in and who is out against the criteria in the latest consultation.

If the Russell Group rejects the no burden option, it will appear that its members aren’t serious about burden reduction, and want to thwart Stern’s proposals by nitpicking over selection for trivial gains. Not a good look. And if they accept, then 100% submission instantly becomes the gold standard. The question then is where universities like Bath or Surrey go? With a lower output minimum of one per person, even those HEIs who were hyper-selective in 2014 should be able to cope perfectly well with this shift.

The second part of the latest consultation, on output portability, is even more of a no-brainer. HEIs are being offered two bridges over the gap from portability to non-portability. One (“the simplified model”) is broad and inviting, with a welcome sign flashing in neon, and free cocktails being dished out at the entrance. The other (“the hybrid approach”) is shaky and unnecessary – a bridge of the Boris Johnson and Joanna Lumley kind. I don’t expect lengthy responses to this question. We need to get the rules right, but we mustn’t allow the perfect to become the enemy of the good.

The road from here

So if we assume the “no burden” option for staff selection – at least among research intensives – and a “simplified” model for portability, and throw them into the mix alongside our new certainties over impact and environment, then we have a clear working model for the next REF. The new consultation runs until 29 September, and I would expect HEFCE to announce its final decisions quickly – certainly before the end of November.

One genuine question remains – whether alternative HE providers should be eligible for funding through the REF. This is caught up in the forthcoming consultation on the wider regulatory framework. But for the rest of us, there’s more than enough to go on. REF 2021 just got real.

Beyond 2021, we can already discern in last week’s announcements some of the likely features of the 2027 exercise. Institutional impact case studies – thankfully dropped from 2021 in favour of a longer-term pilot – could form part of this. Further shifts in the weighting of impact and environment also seem likely.

Ideally, under Research England, the process of further developing the REF can be less disruptive than it has been since 2014 – when all options were at some point in play. It would benefit from becoming a more gradual process, informed by evidence from the UK and elsewhere, and part of the “collective intelligence” that UKRI will – in theory – bring to the strategic management of the research system.

One final thought: the inclusion of Lord Stern’s six purposes of REF in the new guidelines is a welcome play by HEFCE to alter the terms of the debate over its costs, benefits and value. As I argued at the outset of the Stern Review, if the exercise is understood to be purely about QR allocation, or allocation + accountability + benchmarking (2014’s three purposes), then the cost-benefit equation looks very different from a more rounded account of the purposes it actually serves in most HEIs, and for the UK research system as a whole. I hope the six purposes are now formally adopted as the basis of REF 2021. Being clearer about what the REF is for might encourage some to pause before weighing in based on a more partial view.

Then again, REF has become a process that so many of us love to hate. Whatever else changes, I imagine arguments are likely to remain its one permanent feature.