The Hidden REF celebrates the whole research ecosystem

The REF comes in for a lot of criticism: it’s big, it’s expensive and it soaks up time that many argue would be better spent on conducting research.

There’s truth to these criticisms of course, but this does not mean that research assessment is unnecessary. In fact, a certain level of accountability is important, especially in receipt of some £11.5 billion in research and development conducted in higher education and government-run labs.

The REF is the perfect system for assessing research yesterday, but research assessment must remain fit for more modern challenges, including the need to align research and society values more closely, and the evolution of how research is practised and knowledge disseminated.

Enter the Hidden REF, the initiative that celebrates all research outputs and the people who make them possible. Using crowdsourced categories, any member of the research community is welcome to nominate a colleague or their own research outputs, with submissions accepted up to Friday 14 May. And unlike the REF submission, you can submit to the Hidden REF in only 300 words.

Putting up with narrow outputs

Currently research assessment is restricted to a small number of outputs, concentrating on the publication with laser-like precision, but research produces a dizzying range of outputs. In the last REF, 97 per cent of outputs were publications. Though publications are vital for assessment of research dissemination, researchers are becoming increasingly savvy with how and where they disseminate their findings.

A 2019 RAND report on the future of research and research assessment highlighted that not only do researchers produce a diversity of output forms above and beyond the traditional publication, they envision embracing a wider range of outputs for their research findings in the coming five to ten years.

Indeed, a survey of over 3600 UK researchers showed that, aside from every academic expressing their intention to write a book in the next five to ten years (pre-pandemic estimations and ambitions), that academics were more open to disseminating their research as website content, blogs, digital and visual media and research reports for external stakeholders. The problem remains that as outputs of assessment in the REF, such outputs are not part of the mainstream assessment process.

Reasons for overlooking these new outputs in the REF assessment are understandable. Their inclusion alongside the traditional publication is perceived as difficult, because an assessment of their quality would require different approaches, the knowledge of which may be scarce amongst traditional REF review panellists.

Arguably, the REF instills confidence and trust in research in an age of alternative facts and science denialism masquerading as trustworthy science presented as research through blogs, digital and visual media and even working papers. There is a need for REF to assess the information presented in these all outputs as reliable and trustworthy, above and beyond whether the information is restricted to an elitist peer reviewed journal article.

Recognising the people who drive research

There is also a pressing need to acknowledge that modern research is increasingly reliant on non-traditional roles, such as research software engineers, research managers and data stewards, and that it has always been heavily reliant on roles that have long been overlooked, such as finance, librarians and technicians.

Though assessment of publications is vital, it recognises – with almost complete exclusivity – only traditional academics. So how do we recognise the achievements of people who disseminate high quality ideas outside of the traditional structure of authorship in publications?

It is in this space that the Hidden REF can play an important complementary role in testing the evaluation process and assessing a wider range of research outputs. This knowledge – what works, what doesn’t, and how it can work – can be fed into future mainstream REF processes.

Thus, the Hidden REF can be used to ensure not only that the REF remains in line with the changing values and methods of dissemination used by research, but is also sensitive to the ways in which research can be communicated via alternative channels which, in the absence of quality assessment, might be more vulnerable to misinterpretation or channels for misuse.

Already the Hidden REF is attracting submissions to a wide variety of categories of research dissemination not covered by the mainstream REF such as Citizen Science, Training materials, and even Grimpact.

The Hidden REF is positioned to experiment with, not only the inclusion of a wider range of outputs, but also the processes needed to evaluate them. Indeed, the Hidden REF will experiment with a process of open evaluation, a process that is gaining popularity within research culture as part of its growing support for increased accountability and transparency, which is currently difficult to support as part of the mainstream REF.

As a research community, we cannot continue to apply the same rules about what type of research is valued, without change. The assessment processes governing research must evolve as research evolves, but the REF’s long cycle time and heavyweight assessment is not well suited to experimentation.

What is needed is something that can evolve faster, can assess more quickly and can experiment more freely – that can then feed into the REF if successful. The REF as it currently stands may be imperfect, but it’s a necessary endeavour, and the Hidden REF exists in part to aid the continual improvement of the REF by enabling the experimentation of ideas for assessment that are costly for the mainstream REF to get wrong. We are here to achieve a better understanding of how to understand the impact of research.