LIVE: TEF results day on Wonkhe

That concludes today’s Live Blog. There’s still plenty more analysis and crunching of data to be done in the coming days and weeks, but we hope you’ve found Wonkhe’s coverage today informative, analytical, and a bit entertaining.

Remember to look at our #TEF tag as a one-stop shop for all our analysis, as well as comment and opinion from voices across the sector and outside it.

Mark, Ant, Nona, David and David 🙂

Editor Mark Leach sat down with Team Wonkhe’s David Kernohan to talk about TEF in the context of teaching excellence initiatives. Was there any correlation between TEF and CETLs for example?

Read more about history of attempts to improve university teaching.

London’s non-specialist universities have struggled in TEF, and could have done even worse were it not for the panel’s discretion. Nona Buckley-Irvine asks why students seem to have a much poorer experience in the capital.

Does TEF suggest that universities need to narrow their strategic focus on teaching or research in order to succeed?

Perhaps. We’ve had a look at how today’s results compare with the 2014 REF. Here are the results (hover over each marker to identify the university):

David Morris has written a full analysis of this graph here.

HEFCE is still (just) a funding council, and as such is a major investor in the English Higher Education market. But how well does HEFCE pick winners when it comes to TEF?

On teaching funding, the average amount (in £) of HEFCE funding by institution is slightly larger for gold than it is for silver, but there is a big drop between silver and bronze. As HEFCE funding is concentrated in lab sciences, medicine and some smaller subjects, this offers us some hints about the subject mix that appears to do better in TEF.

For research funding the picture is quite startling.

(and no, those are the correct way round). Substantially more research funding goes into TEF gold awardees, with significant gaps between gold, silver and bronze. For all the talk of refocusing attention on teaching, HEFCE research funding is also focused on the gold group, although of course the presence of Oxford and Cambridge in the group skews the picture significantly.

In January, we looked at the patterns in UCAS data for undergraduate admissions and we’ve gone back to the lists to look at the top and bottom ten institutions arranged by TEF outcome. While none of the bottom ten received Gold, we expect those in Silver to feel more optimistic about the potential for a turnaround in their fortunes while the Bronze institutions in the top growers might be more concerned.

Highest % growth in 18 year old undergraduate admissions year-on-year to 2016

TEF Gold

• N21 Newcastle University +19%
• S85 University of Surrey +18%
• S36 University of St Andrews +18%
• L39 University of Lincoln +15%
• F33 Falmouth University +14%

TEF Silver

• L34 University of Leicester +22%
• C60 City University London +21%
• M80 Middlesex University +17%
• S93 Swansea University +16%
• R12 University of Reading +14%
• Y50 University of York +12%

TEF Bronze

• G56 Goldsmiths University of London +18%
• S49 St Georges University of London +17%

Highest % decline in 18 year old undergraduate admissions year-on-year to 2016

TEF Silver

• S84 University of Sunderland -26%
• W05 The University of West London -8%
• S72 Staffordshire University -6%
• B22 University of Bedfordshire -5%

TEF Bronze

• S30 Southampton Solent University -18%
• L68 London Metropolitan University -14%
• C99 University of Cumbria -13%
• W75 University of Wolverhampton -12%
• K84 Kingston University -9%

Did not enter

• U40 University of the West of Scotland -7%

Is there any better marketing opportunity than being awarded TEF Gold?

Marketing teams across the country appear to have been playing Spandau Ballet on repeat, whilst donning golden balloons, dodgy photoshops, and covering their campuses in golden tin foil. From the ‘Golden Six’ to some more political messaging, we’ve found some of the best (and most cringeworthy) attempts to seize the day and reign victorious.

Wonkhe believes that these efforts should not go unnoticed, and after-all, noone likes a sore winner. So without further ado, we introduce the Wonkhe Outstanding Excellence Framework for blowing your golden trumpet. No questions about the metrics please.

Right, so we’ve been able to work out the answer to a question that many people have been asking: how many times did the TEF panels change an institution’s outcome from the initial hypothesis shown by the metrics?

The answer is – 64 times, and 36 times for higher education institutions and alternative providers.

Three institutions had their final assessments downgraded from their initial hypothesis: BPP University, Bucks New University, and the British School of Osteopathy.

Thirty-three institutions had their final assessments upgraded from their initial hypothesis, including eight in the Russell Group, and twelve in London.

Seventeen institutions were upgraded from a Bronze to Silver, including University College London, King’s College London, and the University of Bristol.

Fifteen institutions were upgraded from a Silver to a Gold, including Imperial College London, the University of Nottingham, and the University of Birmingham.

Remarkably, the Royal Veterinary College was upgraded from a Bronze to a Gold!

The TEF panel will no doubt have many reasons for this, and their statements give an indication as to whether provider submissions were deemed to address negative flags to justify an upgrade. But let’s remind ourselves of what the original guidance said:

“The likelihood of the initial hypotheses being maintained after the additional evidence in the provider submission is considered will increase commensurately with the number of positive or negative flags on core metrics. That is, the more clear-cut performance is against the core metrics, the less likely it is that the initial hypothesis will change in either direction in light of the further evidence.”

Here are those upgraded higher education institutions in graph, together with the ‘flag score’ we used earlier. The labels denote the ‘initial hypothesis’ as determined by the metrics, and the colour’s the final outcome.

The TEF outcomes have been primarily based upon a ‘flagging system’, which identified whether institutions did significantly better or worse on their core metric scores. Where they did, universities were issued a positive or negative flag, or even a ‘double positive’ or ‘double negative’ flag for outstanding results either way.

We’ve created a scoring system for the flags in order to list institutions. A ‘double positive’ earns a score of +2, a ‘single positive of +1’, a ‘no flag’ 0, and so on. This gives us another way of listing the outcomes and comparing them to the final results.

We’ve put this in a graph here and below (scroll across to see all institutions, or click to view in full screen).

Editor Mark Leach spoke to Wonkhe’s Ant Bagshaw about the provider submissions – now we’ve got the full crop, what do they say and which were influential?

While there’s been metaphorical dancing in the street, and literal circulation of (gold) chocolate coins in at least one institution, there are also some people particularly unhappy with their TEF results. To recap, the TEF guidance (see p61) allows only very limited groups for appeal:

After the publication of the TEF outcomes, a provider can appeal its outcome on the basis of a significant procedural irregularity in the consideration of its TEF application. This might be on the basis that the published process was not followed when reaching a decision. A significant factual inaccuracy in the statement of findings may be taken by the provider to indicate a potential procedural irregularity. To have grounds for appeal, the procedural irregularity needs to be significant, meaning that it materially affected one of the following decisions:

• whether to accept a data amendment request
• whether the provider is eligible for a TEF Year Two award
• the panel’s judgement on the rating awarded to the provider.

Providers would not be able to appeal if:

1. They were challenging the underpinning principles of the TEF or the criteria or process set out in the TEF specification or this additional guidance.
2. They were challenging the accuracy of the data underlying the TEF metrics.
3. They were challenging the academic judgement of the panel.
4. New information had come to light that was not included in the submission.

Now it may be that threats of judicial review come come into play, but an appeal in that way would need to meet a high bar of ‘irrationality’ (at this point, it’s time to point out that you’d better take proper legal advice).

The news reports are focusing on Southampton VC Chris Snowden’s unhappiness with his result; the university has form for appealing given a win over a poor QAA review result in 2013. It’s likely that there will be some appeals this year, though it’s hard to know at this stage how likely any are to succeed. Watch this podium…

Think this is for you?

The appeals timeline is as follows: 

• Tuesday 27 June, 12 noon – Deadline to submit a ‘Notice of intention to appeal’
• Tuesday 4 July – Providers informed if their appeal is admissible and if so, they will receive any documentation as appropriate to the appeal
• Tuesday 18 July, 12 noon – Deadline to submit an appeal
• Wednesday 26 July – TEF Appeals Panel meets
• By 11 August – Providers informed of the outcome of their appeal and any decision that was reconsidered
• By 15 August – Any outcomes amended as a result of appeals published on the HEFCE website, the HEFCE Register and Unistats

We can now present our full list of the TEF core metrics results. This list of ‘z scores’ shows how institutions did on each of the six core TEF metrics compared to their benchmarks. A positive number indicates being ‘above benchmark’, and a negative number indicates being ‘below benchmark’. A sum of these numbers gives us an overall ranking, which roughly corroborates with TEF’s Gold, Silver and Bronze outcomes.

You can see our full ‘Tableau’ of the results here. 

Props to the planning team at the University of Huddersfield, who have made a nice visual representation of the distribution of flags and their relationship to the medal awards.

Full #TEF TriPlot showing the probability distribution of flags vs awards. Lots to take from this. pic.twitter.com/Gp1KwMjdnp

— UoH Planning Team (@HUDPlanningTeam) June 22, 2017

You might be wondering how to pronounce TEF in Welsh (spelt FfRhA).

A TEF Silver (Arian) award

We are too, which is an embarrassment in particular for our one Welsh member of staff… So, in true Wonkhe fashion we’re launch our “Welsh TEF Pronunciation Contest!” Email us at team@wonkhe.com with your pronunciations and we’ll judge the worthy winners later today.

Twitter is alive and kicking, with #TEF and #TEFresults full on activity, comment, criticism, and celebration.

‘Kudos’, says Jo Johnson, TEF architect and designer.

Kudos to all 295 institutions that volunteered for the first Teaching Excellence Framework assessment – results here https://t.co/kSISAAtl2K

— Jo Johnson (@JoJohnsonUK) June 22, 2017

Aaron Porter speculates on the scenes in some VC offices this morning

Secretly somewhere, a Vice Chancellor proudly sits behind their desk with a whopping great Gold medal round their neck #TEF

— Aaron Porter (@AaronPorter) 22 June 2017

NUS Vice President Sorana Vieru will be choosy in making alliances with disgruntled vice chancellors

Only cookies will be given to VCs who get gold and denounce the TEF as a load of s**t

— Sorana Vieru (@SoranaBanana) 21 June 2017

Stephen McKay points out a the ‘unlikely’ relationship between TEF panel members and their institutions’ results.

#TEF panel, like #REF panels, clearly selected from the upper end of quality. pic.twitter.com/YOmXU0Bgh7

— Stephen McKay (@SocialPolicy) June 22, 2017

Ben Britton makes a good point about the media coverage.

Curious that many news outlets focus on losers in #TEF – cheap headlines ? – could have a progressive dialogue ?https://t.co/ER8zLAJexD

— Dr Ben Britton (@BMatB) 22 June 2017

Jim Dickinson poops on Loughborough’s party.

They still won’t let you into RG though. That’s not how the gang works. https://t.co/JjPaYcGZXu

— Jim Dickinson (@jim_dickinson) 22 June 2017

A wry comment about university marketing departments’ hard work from Wonkhe contributor Alex Prestage

Some of these #TEF reveal videos are as cringe as baby gender reveals.

— Alex Prestage (@alexoprestage) June 22, 2017

This might best sum up Twitter…

“Come on. It’s too simplistic to say those struggling in #TEF will hate it, and those doing well will love it.”

*checks twitter* pic.twitter.com/JVLChVo034

— Gareth Millward (@MillieQED) June 22, 2017

But when it comes down to it, we hope that Julie Vincent speaks for all Wonkhe readers.

I have a feeling I’m going to end spending a lot the day reading articles about #TEF and watching Twitter. https://t.co/IfuWT9Fdn1

— Julie Vincent (@VncntCnsltng) 22 June 2017

It’s not like the mainstream media don’t have enough to cover today, with recriminations from yesterday’s Queens Speech ongoing. It is a mark to TEF’s significance that it has been covered extensively by almost all major outlets.

• The Guardian: “Many top UK universities miss out on top award in controversial new test”
• The BBC: “Leading universities rated ‘bronze’ under new ranking system”
• The Telegraph: “Top universities slip down rankings under new student-focused system”
• The Times: “Elite universities including LSE, Southampton and Liverpool exposed as second-rate”
• “Scottish universities win TEF UK ranking system ‘gold’”
• FT: “Russell Group universities fall short of teaching quality benchmark”
• ITV Meridian: “Four universities celebrate government’s “Gold” rating – while two others don’t make the grade“
• TES: “Going for gold: How did colleges fare in the TEF?”
• The Independent: “Elite UK universities found to be second-rate in new Government rankings”
• The Today programme had a discussion between Sorana Vieru and Jo Johnson. You can listen from 7:40 here. There is another feature on TEF at 08:50.

Good morning all keen TEF results followers. We’re just settling down for our second run of analysis of the TEF results, after a day’s reading under embargo yesterday. You can read the results of our deliberations below.

Suffice to say, the reaction to TEF results across the traditional media and social media has been lively. We’ll be bringing the best (and the worst) of the coverage.

And with the underlying data for TEF released at midnight last night, it’ll be graphs galore on Wonkhe today as we try to visualise the numbers.

Watch this space. Updates will be coming all day.

The TEF results were made public at 00:01am. We have already published a range of articles based on our early analysis.

• David Morris gave us eight lessons we took from our first look.
• Nona Buckley-Irving waded through the first outflow from the sector TEF spin machine.
• Ant Bagshaw analyses the HEFCE TEF press release and is far from impressed.
• Ant also wades into panel judgements and the metrics – why were some institutions marked up and not others?
• Editor Mark Leach’s leading article asks whether the prizes still glitter?

You can find our fully searchable table of TEF results here. HEFCE’s extensive data release can be found here.

There is, of course, a lot more to come – for today you can find it here on the #TEF live blog.