Rank hypocrisy – how universities betray their promises on responsible research assessment

Scientific integrity and ethical conduct are prerequisites for ensuring society’s faith in institutions entrusted with the pursuit of knowledge. As trust in science and scientists is under scrutiny, it is imperative that universities work together to strengthen trust in higher education.

It is therefore welcome that, across the globe, universities are collectively taking steps to stamp out questionable practices that undermine their trustworthiness. For example, the sector is making rapid progress in developing better ways of assessing the quality of research. These changes were sparked by a long-established body of evidence about the significant flaws in metrics such as journal impact factors. Now over 24,000 individuals and organisations from 166 countries are signatories of the Declaration on Research Assessment (DORA), in explicit recognition of the pernicious impacts of the irresponsible use of research metrics.

Even so, universities continue to be complicit in the pervasive and reckless use of much more questionable metrics in the form of commercial university rankings. These increasingly shape not only how universities market themselves but also how they operate: some institutions appear to spend more time thinking about how best to “game” rankings than about improving how they fulfil their core functions. Many use institutional and subject rankings as key performance indicators and exhort departments and academics to be more “competitive”.

Unnecessary evil

Commercial university rankings are often positioned as a necessary evil in the life of universities. This is despite a substantial body of international literature demonstrating unequivocally their flawed nature, which is as least as strong as the evidence undermining journal impact factors. Most institutional leaders react with an embarrassed shrug; after all they must play the hand they are dealt.

Under the explanation that rankings are not going to go away, and often pushed hard by lay governors ignorant of the meaninglessness of rankings as measures of institutional quality, they do their best to maximise their institution’s performance. They even dedicate senior posts solely to this purpose. They then cover their websites and their buildings in loud proclamations about their “world leading” performance in these rankings. This is all at the expense of the long-term health of the sector and higher education’s reputation for scientific integrity.

There is something soul destroying about institutions, whose role is dedicated to the pursuit and sharing of knowledge, appearing to take seriously measures that involve combining incomparable measures into aggregated scores and the use of rank ordering, which disproportionately exaggerates very small differences in the scores of institutions.

Very few, if any, of the measures used are valid or reliable indicators of the quality of education or research but instead simply mirror the wealth and prestige of universities. Even worse, a primary purpose served by these rankings is – perversely – for those who produce them to sell advertising and consultancy services to the universities they are ranking.

Despite their misleading nature being widely known and understood, the performance of universities in these rankings is still used to recruit students, and governments around the world use them to determine funding for students and initiatives. All are being deceived. Any form of university education that claims its quality is demonstrated through commercial university rankings has been mis-sold.

There are signs of change. The University of Utrecht in the Netherlands has recently announced it will no longer provide data for commercial rankings, following the example of others, including Rhodes University in South Africa which has refused to do so for many years. Universities who have signed up to More than Our Rank also emphasise other ways of measuring their quality, although in this case, there is more than a slight sense that these universities want to exploit their ranking whilst keeping their integrity. This is simply not possible.

Cognitive dissonance

It is time for this nonsense to end. We are currently in the crazy position where, as part of their DORA commitments, ancient universities make strong promises not to use any metric without being explicit about its limitations on one part of their website, while on another, they unreservedly boast about their performance in commercial rankings to prospective students. This rank hypocrisy must stop if universities are not to undermine their position as institutions dedicated to the pursuit and sharing of trustworthy knowledge in society.

This may feel like a forlorn hope given the severe financial pressures that so many universities are under. However, these pressures make it even more timely for universities to stop dedicating resources to rankings whether this is through providing data to commercial rankings or paying for the “services” of commercial ranking companies, and committing institutional effort, to promote their position in rankings.

It is important to remember that DORA developed into a global phenomenon from an annual meeting of the American Society for Cell Biology. With the institutions who have withdrawn from commercial rankings and the organisations already signed up to More than Our Rank, there are the makings of a significant movement against commercial rankings. However, this movement needs to be focused on promoting “quality Not rankings”, making it clear that the latter provides no meaningful measure of the former.

To strengthen this growing movement academics need to stop completing hollow reputation surveys. University leadership teams and governing bodies need to urgently reflect on the grave harm that continuing to play the zero-sum rankings game is doing – both to themselves and the long-term credibility of the sector.

Once the spell of commercial rankings is broken, we will wonder why universities ever participated so greedily in this deceitful practice that misleads prospective students, funding bodies, governments, and employers. Higher education institutions face enough challenges from an increasingly sceptical society without engaging in divisive and meaningless competition, which undermines their integrity and trustworthiness, and is solely for the benefit of those who produce commercial university rankings.