Implementing REF2021 needs the input of the whole research community

After the Independent Review of the Research Excellence Framework which BEIS commissioned from Lord Stern, the Minister wrote to us and asked us to work with the other HE funding bodies to develop detailed proposals which take forward the recommendations from the Review, including any further testing and piloting.

Today HEFCE has today published a consultation on the second Research Excellence Framework – REF2021.

We have produced proposals for operationalising all of the Stern recommendations, but we realise that in developing the detailed proposals, there are some options about particular paths to pursue and we are inviting comments about the various options. We also welcome any proposals for alternative arrangements which may better capture the principles which Lord Stern highlighted, recognising that a coherent set of proposals (as the Review identified) may introduce unexpected consequences when the operational detail is considered.

Reducing burden

Lord Stern and his colleagues were set the challenge of identifying ways in which burden could be reduced. They responded by proposing that selectivity of staff should be abolished, thus also eliminating the requirement for consideration of special circumstances. Staff selectivity and the process for consideration of special circumstances were identified as major institutional and personal costs (in terms of academic time).

Of course, it is acknowledged that the equality and diversity provisions in REF2014 brought important benefits. It will be essential to have appropriate means for ensuring that the next exercise meets the highest standards in terms of supporting equality and diversity.

Greater institutional focus

Lord Stern drew attention to a longstanding principle that the exercise was actually about the work of submitted units, not individuals. He suggested a greater focus on the body of work carried out in the institution rather than on particular researchers. Accordingly, we worked from a principle of less personalisation and a greater institutional focus, implementing Lord Stern’s suggestion of a variable number of outputs from individuals.

Recognising interdisciplinarity

We will also call for some institutional level case studies and the environment element will have an institutional component. Lord Stern identified that an exercise which has individual disciplines at its heart should do more to recognise interdisciplinarity, and the institutional level case studies will indeed invite the submission of even more interdisciplinary work, complementing the strong set of interdisciplinary case studies submitted to REF2014.

A more rounded submission

In any complex assessment process, there is a risk that attempts may be made to exploit the rules to produce an outcome which does not appropriately represent the objectives of the exercise. So following a principle of less game-playing and also a more rounded submission, we are implementing his proposal that the exercise should recognise the work of all research-active academics.

Acknowledging investment

In a past exercise, we consulted the sector over whether staff who move during the assessment period should submit outputs from their former institution in their new institution’s submission. Providing due recognition for investment, we are proposing that the outputs should remain with the former institution.

We also want to support the development of early-career staff so the consultation explores how we might mitigate against non-portability having negative consequences for certain groups.

Broadening impact

We are also proposing a broadening of impact. Here, the aim is to encourage universities to identify a more diverse set of impacts in their case studies, as Lord Stern proposed. And, depending heavily on the consultation responses, we are suggesting that panels identify where research does a particularly strong job in supporting teaching.

Even a casual glance shows how some of the new proposals support several of these principles. Not only is the burden on institutions decreased by eliminating staff selectivity, but there is less scope for game-playing.

Some thoughts on the detail….

We are particularly keen to hear views on the following issues:

‘Submitting all research-active staff’ requires us to identify who is in the target pool in a way which is transparent and auditable. We’ve previously used data collected by HESA for this, and we propose to do so again, with an additional test, analogous to the old ‘independent researcher’ test. We can see no better option but our conversations in developing proposals have identified some issues around the use of HESA data for this purpose.

Similarly, the use of HESA costs centres to allocate staff to units of assessment (avoiding selection being used tactically at UoA level) has given rise to some comment from those contributing to the development of the consultation. Although we are very interested in critiques of any of our proposals, we are also looking for alternatives which can be implemented. We do not think any perfect system exists, but we do look for a most reasonable compromise which delivers the objectives of the exercise.

Allowing a variable number of outputs per person has many advantages, as Lord Stern set out. However, he asked us to consider which variable numbers were appropriate. In order to contain the burden for panels we now expect an average of only two outputs per submitted academic. Allowing a range of 0-6 outputs would mean the body of work of the submitted unit could potentially represent the work of only a third of all staff within a UOA. Is that a more rounded assessment? Would a variable number between 1 and 4 give a better-rounded assessment?

A further consequence of the proposed changes to the number of outputs per submitted researcher is that research units will be able to submit only their very best outputs. For many of our very best research units, this may mean that a large number of the outputs submitted from them will be assessed as world-leading.

There are many positive aspects of restricting outputs to the university where the work was carried out, most notably to restrict incentives for artificially high levels of academic staff mobility prior to the REF census date. However, there are considerable challenges in identifying at which institution a researcher carried out a specific piece of research. This may prove difficult for journal articles but could be even more so for long-form research outputs. The processes of research leading to a scholarly monograph could extend over long periods, well after an initial proposal might be accepted by a publisher. We would welcome suggestions from respondents on this issue.

Only by harnessing the collective wisdom of the community can we best implement the Stern principles or, if necessary, favour some principles over others in operationalising the proposals. The funding bodies will be holding a range of meetings, and our senior staff will be visiting universities and other groups to take part in discussions about the proposals.

Every ten years or so we review our research assessment process and we have just done so, both with Lord Stern’s report and the influential review of metrics which Professor James Wilsdon led. No doubt we will undertake another review in a few years’ time. But for now, we look forward to working with the community on finalising the details of this exercise.