Conflicts of interest and compromise – the subject-level TEF consultation

While I doubt it will take off as a leisure pursuit, I propose a new game for us wonks: guess the institution’s Teaching Excellence Framework rating from its response to the subject-level TEF consultation. I’m not offering any prizes, though, as I think the game will be too easy.

All institutions will have considered the subject-level TEF proposals in the light of achieving outcomes that best represent them. Of course we will, that is our job. I predict, therefore, that the responses soon to be sifted through at the Department for Education will be characterised by evidence of a new sector dividing line, one determined by how close an institution might consider itself to climbing – or slipping – between the TEF’s Gold, Silver and Bronze ratings.

Model A vs Model B

That new dividing line will commonly be found in institutions’ preference for Model A or Model B. Model A – for those who might have managed to avoid this so far – is the “by exception: approach which assesses only those subject areas where metrics are deemed to differ significantly from the provider metrics, allocating the provider rating to all other subjects. Meanwhile, Model B is the “bottom up” approach that assesses all subject areas and then uses the subject ratings to inform the provider rating. Let’s be clear, both models are complicated, so lacking a certain easy transparency, and hindered by challenges around data robustness and queries about how meaningful the results will be for students.

It is highly likely that in the consultation response some institutions won’t have been able to bring themselves to opt for one over the other (look out for a Model C). Of the two presented, however, Model B best meets the primary intention of Subject-Level TEF – that being to provide greater information to students – since it allows for greater variation between outcomes for subjects. However, highlighting variation in provision will only be attractive to institutions where that differentiation is a better rating than the current provider-level rating. If you want to hide weaker performance, then opt for Model A.

The main argument in favour of Model A is that it will reduce the burden of submission and assessment. That will be attractive to institutions which, having been through the exercise once and established their credentials, perceive the requirements of TEF as an unnecessary additional imposition that will deliver minimal return. Solid Golds and Silvers are likely to prefer Model A for this reason. Those at the borders of the ratings, with an eye on how close they are to moving between them, are more likely to see value in the greater effort required by Model B.

Proportionality and value for money are of course important considerations when designing these exercises. But let’s not forget that this is meant to be an attempt to give students more information. Some aspects of the proposals feel as if the primary purpose of the whole thing is to minimise the burden for institutions and assessors.

Duration of award

The dividing line between institutions will also be seen when it comes to the proposals around lengthening the duration of TEF awards to 5 or 6 years.

As with Model A vs Model B, institutional responses will depend on the perceived value to them of undertaking the TEF submission. Those which are unlikely to see their rating change, or indeed which might see their metrics moving in the wrong direction and worry about a lesser rating, will naturally support longer duration awards. Those hoping to gain a shinier medal as a result of improving performance will see value in more regular submissions.

There is a (rather insulting) concern expressed in the consultation documents that annual assessment exercises may tempt institutions to focus on “gaming” the exercise rather than achieving genuine improvement. This might be countered with an (equally insulting) argument that Gold awards for a duration of 5 or 6 years may lead to complacency and hide dips in performance rather than incentivising sustained excellence.

One can’t help but feel a distinct “kicking of the whole thing into the long grass” in the proposal to extend ratings to well beyond the lifecycle of any undergraduate degree. Indeed, many might feel that’s where the current TEF methodology belongs, and there must be a strong argument for pausing until the statutory review is undertaken. But an exercise designed to appease institutional reluctance to engage too frequently can only undermine its own objectives of driving change and sustained excellence.

A point of agreement

There are, however, bound to be areas of common ground on the consultation proposals. Every institution I have spoken to has identified a problem with the subject classifications, highlighting why combining disciplines X and Y makes no sense in their institution. However, in each case the disciplines cited are different because the issues stem primarily from institutional structures.

At Goldsmiths, the “creative arts and design” subject area covers four very different departments teaching entirely distinct subjects in their own specialist environments. Aside from the challenge that would pose for writing a single coherent submission, a single TEF rating for these departments would be of no more meaning to students than an institutional rating. In fact, in this case I would argue that is isn’t institutional structures that are the problem – how art, design, music, drama, dance, cinematics and photography can be contained within a single meaningful assessment at any institution is difficult to imagine.

In Goldsmiths’ consultation response we have suggested that institutions should be able to make more than one submission against a subject category, as is permitted with units of assessment in the Research Excellence Framework. This would require reportable metrics but, where that requirement is met, institutions are best-placed to determine if a separate submission is justified in terms of distinctiveness of subjects and meaningfulness for students. That, perhaps, is one proposal we can all rally around.

In the meantime, I do not envy the task of those at the DfE who are opening the consultation mailbox this morning. Negotiating a fair and robust system which truly serves the student interest rather than simply being a cobbled-together compromise between the sector’s conflicted interests will be no mean feat.