It would be possible to run a low bureaucracy REF that only measures research outputs.
Research England and its counterparts could put all the research outputs in the UK into a big ranking using a mixture of bibliometric and altmetric data (things like total tweets about research) then assign funding based on which universities the research came from.
If this was deemed too narrow even for the most pure outcome driven research fans it would be possible to quite easily add in some data on impact. This could be data on funding, collaborations, total doctoral students, and so on.
The metric tide revisited again
A purely metric driven exercise would be significantly cheaper to the tune of several hundred million pounds. And, while Research England would correctly argue the administration of REF represents a fraction of total research spending, it is money that could be spent on actual research, not the administration of research. Although some research would suggest there is low salience between peer-review results and metric results, this could as easily be a criticism of peer-review as it is the cold face of metric driven performance review.
This is before considering the enormous administrative burden within universities on preparing for the REF. There are teams, departments, and cumulatively years and years of activity dedicated to preparing for REF.
If REF was only about measuring quality of research the argument that it should be solely about the measure of outcomes in the least burdensome way would hold water. As former government advisor Iain Mansfielfd argued on twitter once you begin to add in weightings for activities like culture and environment “Now we’ll have people judging if the ‘culture’ is ‘right’ for producing great research, rather than actually looking at the outcomes.”
It is entirely reasonable to want to look mostly, or even entirely, at outcomes but it misses a point that the REF is about incentives not just results. It is also possible to argue that if research is great it is the product of effective research cultures by virtue of having produced great research. And it is absolutely the case that the UK is great at research.
Turn the tide
The proposal to increase the weight of people, culture, and environment, to 25 per cent of the overall score has moved this debate into the spotlight. However, it is entirely congruent to want both great outcomes and great research cultures, and believe the REF can measure both of these things without diluting the quality of research overall.
For a start, it may be great research is being produced in spite of enormous issues in research cultures across the country. As of 2022 universities were celebrating the employment of the 41st black woman professor not in their institution but in the entire country. It is difficult to look at this statistic and then also argue that incentivising more equal institutions is a net negative.
Similarly, even taking the argument that effectively the quasi-market decides what quality looks like by measuring outcomes ignores that there is a massive market inefficiency in the loss of talent in universities.
The REF does not measure the cumulative talent of researchers, or even the UK’s total research power, it is a snapshot of a point in time of the researchers that have managed to stick in an institution long enough to become “REFable”. There is an incredible loss of talent of people who leave academia, not because they want to do something else, but because it’s too expensive, too precarious, or they can turn their substantial gifts to a better quality of worklife.
When REF 2028 reports there should of course be careful interrogation of research quality. Research is one of the UK’s leading global exports. However, measuring people, culture, and environment, is not a soggy undefinable thing of nice feelings and good intentions. It is about the serious enquiry, in a way which is more metric based than impact, about whether the ways in which research is organised, funded, supported, shared, and managed, is conducive to a globally competitive and sustainable research environment that enables talented staff, wherever they may be and whoever they may be, to succeed.
This is woke nonsense
Far from being woke nonsense, what’s being proposed for REF is a sensible and welcome step forward. It represents an acknowledgment that outcomes are important, but that contrary to current dogmas in higher education assessment/regulation process matters as well. As universities we operate in hugely complex environments, so that we have much more limited influence/impact on the outcomes we achieve than we like to imagine (see for example the work at https://www.humanlearning.systems/ ). So *solely* focusing on outcomes is a deeply flawed approach – organisational intent and process do matter in their own right, and need to be recognised. To me the REF changes are a welcome acknowledgment of this, while acknowledging that outcomes are also crucially important (so 50% of the weighting being on outputs). A recognition in other areas of HE assessment/regulation of the need to balance better the assessment of intent, process and outcome would be welcome.
The idea that all research can ever be captured using just a mixture of bibliometric and altmetric data is too biased towards the physcial sciences, as well as being based on simplistic and unrealistic hopes. To make the claim as if it is a self-evident fact is disingenuous.
Much robust and socially useful research generates no direct bibliometric footprint – an issue the UK national funder’s reporting platform is currently reviewing in a genuine attempt to genuinely support all of academia. In those disciplines that currently overwhelmingly rely on bibliometric data, the emphasis on publication alone has been creating multiple perverse incentives for decades, as the EU has recently recognised in its attempts to reduce the stranglehold of publication-dominated academic status for grant awards. The opposition to this change has (uunsurprisingly) come from those individuals whose current elevated status is threatened by a shift to more real-world impact.
As for the hope altmetrics will save the day: the use of Twittter as the benchmark for this approach is telling. Putting the UK’s academic funding into the hands of Elon Musk’s latest whim regarding maintaining twitter’s audiences and permitted posts appears far beyond negligent.
The underlying premise of these changes (assuming that the ultimate aim is still to measure research quality) is that the quality of research output is a function of research culture/environment. Two possible conclusions follow if the premise is accepted: either i), since outputs are merely the symptoms or effects of the environment, the weighting should be 100% environment and 0% outputs; or ii), since the quality of the outputs is the index of the health of the environment, the weighting should be 100% outputs and 0% environment. One argument for favouring the second of these two indifferent conclusions might be that new intellectual discoveries are inherently more interesting than the summed FTE of wellbeing advisers that a given institution employs.
as long as the Altmetric disappears into darkness. The mere idea that interactions on privately owned platforms, which are often a grotesque display of egotism, can be used as an indicator is uniquely abhorrent