A first look at what the 2018 TEF3 data tells us

The messages that the Office for Students (OfS) want to get out are that higher education is a high performing sector and that there is excellence across a diverse range of higher education providers.

It also wants to emphasise that TEF 2018 (TEF3) is “broadly comparable” with last year (TEF2) and that the data is “very solid and very valid”.

Here’s what we think…

1) A summary of the overall results

What are the TEF3 results? 86 institutions got a (non-provisional) award in TEF3 – 26 entering for the first time, 60 TEF2 holders applied again. That means a total of 269 institutions are now current (non-provisional) TEF award holders, with 34.6% applying this year. 73 (27%) providers hold gold, 134 (50%) have silver, and 62 (23%) have bronze.

Only providers from England and Wales entered for 2018. None entered from Scotland and it is the second year with no entries from providers in Northern Ireland – TEF still feels very much like an English exercise.

Most providers who entered both TEF2 and TEF3 improved on their performance between the years: 43 of 60 reapplying institutions received a higher award in 2018, with 15 holding the same award. The biggest upward movement was seen from providers which previously held bronze awards and now hold silver. In only two instances did providers receive a lower award than in the previous year – South Thames Colleges Group and South & City College Birmingham hold a bronze award after having received silver in TEF2.

2) What are provisional awards?

Of the 113 awards given in 2018, 27 are provisional. These are given to providers that meet threshold quality requirements but do not have sufficient data to be assessed for a gold, silver or bronze award.

3) Is TEF3 really comparable with TEF2?

At the press briefing, it was stated that it is “broadly comparable”, with some “slight changes of emphasis”. It was highlighted that the same criteria, judgements and underlying metrics were used for the two years. When asked if this gave fair and accurate information to students, it was emphasised the broad (but not public) ranges of performance within the three broad award categories, and the “cliff edges” between them. It was pointed out that institutions may choose to re-enter TEF if they think their data is good enough to move them up a band, and that they may have still improved even if they stay in the same band or don’t choose to reenter. Of course, in the latter two cases, they might not have improved too.  

Frankly, we’re not convinced by this. The initial hypothesis is calculated using a different weighting of metrics than last year. Even though the panel and other, more “holistic”, elements of the process have a lot of sway over the final award, the initial baseline is generated differently. TEF3 is a different game to TEF2 – there are different rules. And really “official statistics” should reflect this difference in presenting the data.

4) Why do some awards last less than three years?

Normally TEF awards last three years, but where there aren’t enough years of historical data some providers have one- or two-year awards.

5) Are institutions just getting better at TEF, is there real improvement?

It was asserted that the panel had judged the evidence fairly at the two different points of time, for TEF2 and TEF3. We don’t actually know (and have no way of knowing) if a rise in TEF award reflects underlying quality improvements or better gaming. We do know that National Student Survey (NSS) input is halved between TEF2 and TEF3.

6) Why do institutions choose voluntarily to enter TEF?

It’s a lot of effort, the outcome is uncertain, there’s no evidence of impact on admissions, and since the Prime Minister’s conference speech, the potential reward on the table is the ability to charge an extra £250 for tuition fees if you get bronze or better. However, fees may go up again, TEF is an opportunity for providers to evaluate their data, and – as was stated at the briefing – a chance to “valorise teaching”. Institutions may choose to stick (rather than twist) with their existing award or fancy their chances of going up an award if their data looks good. It’ll be interesting to see if any institutions change several award bands rapidly, and why. And, of course, from the 2018/19 academic year, all institutions in England with over 500 students will have to hold a valid TEF award in order to be registered with OfS.

7) Will the proportions of the three awards stay the same?

The intention with TEF is not that everybody slowly moves up into gold, but rather that a classic bell curve distribution is represented across the three awards. This is deliberate, with TEF designed to be “always self-balancing”. Although with TEF3 there are hints of grade inflation – with 43 institutions going up to a better award, 15 staying the same, and only two going down – it’s worth noting the overall proportion of gold awards is slightly smaller than with TEF2 (from 25% down to 23%). It was stated that this didn’t indicate “deteriorating” teaching quality across the sector, but rather that it’s a different cohort.

8) Is TEF biased in favour of Russell Group institutions?

It was pointed out that although four Russell Group members have improved, 18 non-members have too. It was also said that the methodology has been applied transparently and consistently, without any preconceptions. This doesn’t explain where the methodology gives certain types of institutions particular advantages or disadvantages.

9) Did written submissions “bump up” institutions’ awards?

“No!” was the message, no awards are given to anybody until final judgements are complete. There are three stages to this; an initial algorithmic review of core metrics, a review of the supplementary evidence, then an examination of the open text submission. Finally, a holistic judgement is made by taking account of all the evidence. It was also stressed that there are “different routes” to gold.

10) Will TEF continue to evolve?

In response to this question, it was said that whether it’s REF, the Ofsted inspection framework, or the Care Quality Commission – such things always evolve. It’s worth noting however that all of those typically take place more years apart, six years in the case of REF, whereas TEF is annual (though awards can last up to three years). It’s also worth asking if people make such direct comparisons between different REF exercises, given the changes to the approach.  

11) Is TEF data really “official” statistics?

Despite DfE and OfS inserting the O word wherever possible, it’s not been a great first stab at this. It will also be published via UCAS and Unistats. A “technical glitch” delayed the publication of the underlying data, panel judgements, and institutional submissions by a couple of hours. The UK Statistics Authority Feb 2018 Statistics Code of Practice states:

• T3.6 Statistics should be released to all users at 9.30am on a weekday.
• V2.4 Statistics, data and metadata, including those available through data services, should be released at the greatest level of detail that is practicable to meet user needs. They should be consistent with common data standards and protocols wherever possible.

OfS will likely be designated a producer of official statistics via the passage of the Official Statistics Order 2018 statutory instrument by resolution of each House of Parliament.

12) And finally … biblical news

We were initially surprised to see the Assemblies of God achieve only a Bronze award, but on reflection, this was foretold in Revelations Chapter 1, Verse 15: “His feet like fine bronze fired in a furnace, and His voice like the sound of cascading waters.”