The Government’s recent Green Paper made no bones about the fact that it thinks that REF is too expensive. Fuelled by academic discontent with REF, the paper clearly seeks to co-opt academic protest in order to shape future exercises towards metrics-based approaches.
However, there’s a snag for the government. The recent influential report on research metrics by James Wilsdon clearly stated that there is no measure at the moment that can adequately capture the research landscape. This is obvious to anyone who works in a university. The current regime of peer review-based national research assessment, however flawed and expensive, both carries the confidence of universities and provokes a number of positive management behaviours (also alluded to in the Green Paper, though selectively, and only in a negative sense). As such, while it is straightforward to imagine ways to translate flawed evaluation procedures into mechanistic funding allocation systems, it can’t be done by flicking a switch.
The devastating damage to the UK HE research sector would be widespread and intolerable.
Given the above, it’s interesting that BIS/HEFCE recently advertised the public sector tender contract: UK SBS BLOJEU CR150082HEFCE Provision Of Citation Indicators.
Advertised for just 10 days starting on October 29th (one day before the Green Paper was released) and with a fulfilment deadline of 10 days after the contract was awarded, this contract asks for “robust citation data relating to institutions’ research outputs in the period 2008-2014” containing the following data:
For each journal article in the period 2008-2014 with at least one author from a UK HEI, we require:
- Article DOI
- Field-weighted citation impact (FWCI)
- An indicator of the citation impact of the journal. This should not be the journal impact factor, but an indicator which seeks to address some of the issues identified with journal impact factors, such as the Source Normalized Impact per Paper (SNIP).
- Citation percentile
- Name(s) and UK nation of the UK HEI(s) included in the address field
- Journal subject classification
- Year of publication
This data is to be provided so that HEFCE can “undertake further analysis on the data” and so that it “can inform internal policy discussions and development”
In each case, “journal subject classification should be mapped onto REF 2014 units of assessment”.
The contract is worth £16,667, which means that the provider is not likely to do the work within a ten-day period, but is going to export an existing dataset and give it to the government in a form that they can play with. There are only two providers that I know of that will have this data to hand and ready to go within ten days: Elsevier and Thomson Reuters. One of them is likely to get the contract. We also know that BIS and its ministers meet frequently with these organisations.
Also, for information, the specifically requested Field-Weighted Citation Impact “is the ratio of the total citations actually received by the denominator’s output, and the total citations that would be expected based on the average of the subject field” – see page 55 of the Snowball Metrics report.
I can’t be sure, but I suspect that Elsevier is the only provider with the FWCI data, as they provided this report to BIS previously.
So, what’s going on? It’s not worth being overly conspiratorial about it; it might be nothing and one would sincerely hope that a £16.5k tender could not be given equal weight alongside a hefty and legitimated independent review of metrics. Process must count. But the Metric Tide already did a comprehensive citation correlation analysis to previous REF results, backed by Elsevier’s own sanction that the metrics simply aren’t good enough to replicate the peer-review panels. So it’s not likely to be a correlation exercise.
The only thing I can think that it might be is modelling of a simpler non-selective (or random sampling) exercise to allocate funding. This is, after all, a bigger dataset than the correlation analysis in the Metric Tide. It comprises all outputs with UK HEI authors, not just those submitted to the last REF.
Indeed, eliminating selection from REF would yield some of the savings that BIS would like to make. Given wider political agendas around teaching, scientific selectivity, and research concentration, one might also infer that the devastation caused to the research base might be deemed acceptable if it left Cambridge, Oxford, Imperial and/or UCL unaffected. If Manchester can do a bit better, that’s probably OK as well.
If policymakers can find a way to use citation metrics to produce that kind of result (probably through some kind of skewed banding procedure that maintains the grip of elite research-intensive universities on the bulk of research funding), while shaking up some of the other institutions, that could be made to work. But it would be awful for actual research quality. I also think that it would be extremely politically damaging to pursue such a route and would probably be contentious at various levels within the Conservative party itself.
What happens next will depend upon how confident the government is that it can allocate funding in a way that does not need to carry the confidence of the HE sector, when all evidence suggests that universities are only likely to intensify their gaming and strategising behaviours if more metrics are introduced. It is also clear that a metrics-based approach would not drive down costs at HEIs; we’ll all just end up paying Elsevier and Thomson Reuters for the data and analytics.
On the other hand, the government have already signalled through the Green Paper that REF is too expensive for their liking. So it’s likely they are planning to try and change something in this space. Hence, I suspect, the need to ‘run the numbers’ quickly now while everyone is distracted by the TEF and the fallout from the Green Paper.
Update: HEFCE have provided us with a comment on the above story:
“The recent higher education green paper has indicated that the Government wishes to consider options such as making greater use of metrics and other measures to “refresh” the REF results. In line with HEFCE’s ongoing policy development and analysis activities, we have been asked by BIS to investigate this further.
We welcome Professor James Wilsdon’s reconvening of the metrics review group, and will be happy to receive further advice from them.”
I don’t know enough about the politics in this area, but it does smell rather like “We convened a huge working group to give us the Metrics Tide report, but that told us truths that we don’t like; so we’re going to pay Elsevier to tell us lies that we do like.”