Live: The Incredible Machine – What next for TEF?

It’s been a lively and “wonky” event. In closing Ant noted that accentuate the posiTEF saw a real commitment for parity of esteem for teaching, which is to be welcomed. He highlighted the work of the many contributors to Wonkhe in providing the detailed coverage that has informed debate across the sector.

This concludes the live blog – hope you have found it useful and apologies that we couldn’t capture everything, or invite remote readers to our garden party.

The first question concerned the potential PG TEF – David Morris saw this as a rolling forward of the performance measure culture that has engulfed most other public sector service delivery – and in feeding in to value-for-money calculations in years to come. Jones noted the existence of PTES and agreed that the decision between encouraging market behaviours and supporting teaching enhancement will determine the nature of such an exercise. Marginson noted the differing value placed on PGT and undergraduate courses internationally – for example in the US a first degree is an expected baseline, a second degree allows graduates to use their knowledge. Rigby – who chairs a group looking at the feasibility of developing a PGT NSS – noted that PGT is not a priority for many institutions and that we need a tool to address this.

Turning to rewarding individual teachers and teaching promotions, Jones felt there should be systems of recognition for those involved in teaching – and more recognition of collaborative awards. Marginson agreed that collaborative teaching needs to be recognised – seeing a parallel with with research where individual excellence is privileged above wider community support. As the UK as a world-leading capability in HE teaching and widening participation research you would think that social inclusion metrics could be added to TEF.

Future of narrative submissions – Jones felt that there should be an open transparent weighting between the narrative and metrics. It is also possible that narrative submissions will steer future metrics decisions for TEF. Rigby felt that narrative submissions need to stay as the current metrics are not good enough.

There are potential issues with the way data are presented and this can effect inclusivity. Rigby feels that split metrics are useful here but we need to move from benchmark to absolute aspirations, making it awkward for HE!

Morris recognised that performance management done badly can be hugely damaging, but feels that this is not a reason to resist all performance managements. Jones added that the sector is not good at having difficult conversations with undermining staff. Rigby noted that she has brought in chairs of learning and teaching in two institutions which has had a huge impact on culture, but this needs to be managed alongside promotion expectations for existing staff.

We’ve just seen confirmation that LEO will be released at 9:30am on 13th June.

Dr Mark Jones (Chief Operating Officer, Higher Education Academy) felt that the “elephant in the room” was that the TEF doesn’t measure teaching (much less teaching excellence). But the TEF has prompted serious and intense consideration of teaching amongst providers, which is to be welcomed. The HEA believe (from the HEPI-HEA survey) that teaching quality has an impact on student perceptions of value for money. TEF could have been a much more supportive exercise, there is a risk that on Monday that many institutions will see TEF as a punishment. Jones argued that the sector should “take back control” of the idea of teaching excellence – promoting the idea of teaching excellence as a journey, and as institution (and subject) specific value,

Jones feels that the sector should play a part in developing teaching metrics rather than just arguing against them. Significant research (for example Gibbs in 2010, and more recent research on subject area teaching improvement) scaffolds the opportunity to develop high-quality measures of teaching. And we should not forget to look at international initiatives in this area.

Professor Simon Marginson (Professor of International Higher Education, UCL Institute of Education) underlines the need for performance management to be part of our lives, and that better data collection (and hopefully better data quality) gives us a great potential resource if we can learn the best way to use it. Teaching and learning needs to be seen as equal to research, but it cannot be where research is performance managed and teaching is not. He argues that performance measures – so far in the REF – have driven quality improvement, but this may not happen in the TEF as it does not directly measure teaching quality. Factors other than institution teaching quality are at play in many of the metrics that underpin the TEF.

The REF has had a global impact in HE policymaking. The TEF will not – it does not create a plausible relationship to teaching that would support quality enhancement. Bad measures (such as some league tables) can become ingrained – the TEF could survive with government support but we should take a different road, possibly via specific discipline based measures drawing on international work.

Professor Sue Rigby, (Deputy Vice Chancellor – Student Development, University of Lincoln) had an “uncomfortable epiphany” whilst writing her TEF submission… there is something fundamentally odd about the idea that learning gain is not linked to teaching excellence. The raw “learning experience” have been marginalised. Skelton (2014) suggest that teaching excellence was a “contested concept which is historically and situationally contingent“.

An informal survey of teaching experts (self-selected group, no attempt to balance the sample) around the definitions of teaching excellence saw teaching as being orthogonal to student learning, with the subject discipline being used to develop analytical capacities: “the ability to engage with the unknown in a creative, analytical, contingent manner”. Experts saw the learning journey as unique to each student, and involving an element of risk: intellectual spaces should develop to encourage this. “Getting students feeling comfortable is a prerequisite to intellectual risk, but is not the endpoint.

Teacher authenticity derives from subject experience and research – not from teaching skills. And there was consensus on the need for learning to be adaptive and iterative. So the long term future of teaching excellence needs to be intimately connected to student experience and learning gain, and should explore how the impact of the “teacher” can be measured.

David Morris was a panel interloper, who responded to questions.

Wonkhe’s Nona Buckley-Irvine chairs the final panel of the day; featuring Dr Mark Jones (Chief Operating Officer, Higher Education Academy), Professor Simon Marginson (Professor of International Higher Education, UCL Institute of Education) and Professor Sue Rigby, (Deputy Vice Chancellor – Student Development, University of Lincoln). Here we’ll be focusing on the future – considering both future iterations of the TEF and the future of teaching quality enhancement more widely. It’s been a sub-theme of the day but an understanding of the history of this field is a hugely important thing to have when considering both the present and the future: we plugged our overview at lunchtime but I really do insist you take some time this election day and enjoy the long read.

Sue Rigby, DVC at Lincoln (and involved in the HEFCE Learning Gain work), spoke on a personal basis about her experience with Learning Gain. Drawing a parallel with sports statistics, she cautioned against the use of a stream of numbers where a few simple words would do. There is no blueprint for teaching excellence, and there are frequently headwinds as ministers make remarks about poor quality teaching.

In developing degree programme descriptions, skills and attributes are often implied rather than stated. She cited the Wilsdon report on the intelligent use of metrics, noting that a response to “bad metrics” is often to look for a better one. The overall emphasis was on the benefits from thinking harder and better about learning, learning gain will not provide a single proxy that is more reliable than institutional data.

Learning gain may be a better metric, but it will not be perfect and will not answer all of the issues that current metrics have. The RAND Europe report underpinned the development of an investigative programme – defining learning gain as “the improvement in knowledge, skills, work-readiness and personal development made by students during their time in higher education“. Thirteen longitudinal projects, a national mixed methodology pilot and a data driven analysis (HELGA) that will attempt to combine and interpret available data on every aspect of teaching.

The process may offer students the opportunity for reflection on learning, and may offer a better understanding of focused intervention effectiveness. And some metrics of subject area of institutional metrics may emerge. Although many may ask for an overall metric, we should be careful what we wish for.

Wonkhe Deputy Editor David Morris introduced the Longitudinal Education Outcomes (LEO) data. LEO links tax (HMRC), benefits (DWP) and student loans (SLC) data, showing employment and salary outcomes of UK graduates at 1,3,5 and 10 year intervals. If it is presented as previously it will be broken down by input variables: gender, ethnicity, region, prior attainment… subdivided by JACS1 subject codes. It covers “leavers” rather than graduates.

It is important to note that the data does not take into account the changes in the labour market over the 10 year period, and is not benchmarked. LEO draws on a background of DLHE and the UK Labour Force Survey, and an IFS/Nuffield study, Human Capital. Our best guide to the way that the data will be represented is the experimental release last year – which also introduced us to the key findings that UCAS points are a good predictor of future earnings, and that there are still (in 2014-15!) gender gaps in salaries everywhere.

LEO, again, is far from perfect. For example it does not control for regional pay differences, it only covers English HE and is not comprehensive. But the implications are commercial and political dynamite, even though it is mostly good news for the usual winners. It may eventually end up in TEF, and it may creep into the promised “major review” of tertiary education promised by the next Conservative government.

There was concern from the audience around the value judgements inherent within LEO, which was largely agreed to by the panel. Njoroge notes it will be difficult to move away from this, as legislation has been passed (via a little-noticed addition to the Digital Economy Act) to make possible the data sharing that underpins LEO possible. Heily-Rayner was a little more positive, reflecting that some not all students will find this information valuable. Elliot-Bowman notes that it may be of lesser value without an understanding of what students are doing.

Is TEF on the radar for prospective students and parents?

Elliot-Bowman repeats that the independent HE sector are keen to make use of TEF ratings when they are available, and that subject area TEF will be valuable in comparing independent institutions with individual departments or courses (as in KIS, which independent HE will enter from this year) in traditional universities. She feels that some students may note the presence of absence of TEF awards when they are close to making an application decision.

Njoroge agrees that it is difficult to know at this stage. She was surprised that there was no reflection on the lessons learned from KIS before TEF was launched ostensibly to do the same job in informing student choice. But if institutions look after the student experience the data will look after itself.

Thoughtful discussion ensued concerning the aspiration for improvement (all institutions should aim for Gold) acting against the functional enabling of the market (all institutions cannot be Gold). The uncertainty of oncoming TEF changes are another factor in this landscape.

Advice for a new minister or DofE staff? Heily-Rayner encouraged that the subject level TEF should not be abandoned, and called for further development of reliable and defensible value-added measures. Njoroge advocate the design of future exercises based on the data needs of particular questions, rather than just on what data is available. Elliot-Bowman simply urges that we “slow down” – too many things are happening at once.

In closing, Ant Bagshaw noted that the need to understand data in order to rationally use it has come up throughout the day.

Matt Heily-Rayner at Intelligent Metrix outlines some of the decisions made by the Guardian in producing their HE “league tables”. Distinguishing league tables from *benchmarked” performance indicators, the TEF can be considered as attempting to straddle both aspects, with the panel judgement being a new data source alongside the already well-used metrics.

Will the Guardian League Tables use the TEF results? Considering data resolution – the single institutional focus of next week’s release (and the low number of subject grouped rumoured for future years), and the small range of potential outcomes both present potential problems. The Guardian uses eight metrics, to compare with the six (drawn from three data sources) currently in TEF.  The TEF also weights NSS scores way beyond what is already criticised in league tables. Finally, the TEF uses three years of data in comparison to the most recent available year used in the Guardian – using old data is a concern.

As things stand, the Guardian will not integrate the TEF into the league tables they publish.

Jackie Njoroge at Salford comes from a background in university planning, and recommends that the rest of us read the HESA technical guidance on benchmarking and table 6 of the HEFCE TEF guidance. TEF, after all, is benchmark rather than absolute metrics based.

One surprise to her was the omission of POLAR from the TEF – regional factors are likely to influence results, as are student aspirations.

Many of the TEF metrics in use are very old, though the advent of data futures will lead to more real-time data. She cautions that students tend not to look at subject level data, and is surprised by the emphasis on subject level data in policy. She’d rather see learning gain as an evidence base for decision making.

We’re straight back in after lunch with one of the day’s more technical sessions – “The devil is in the data: LEO, league tables and the dreaded metrics”. Wonkhe’s own Ant Bagshaw is chairing a panel that features Joy Elliott-Bowman (Policy and Public Affairs Manager, Independent Higher Education), Matt Hiely-Rayner (Director of Intelligent Metrix and Head of Planning, Kingston University) and Jackie Njoroge (Director of Strategy, University of Salford) – it’s a line up to delight the data-orientated demographic within this very engaged audience.

Joy Elliot-Bowman reports that the independent HE sector, who have mostly been eligible only for “provisional” awards (a level separate from the Bronze, Silver, Gold that permits a fee loan cap uplift), are as yet undecided on the impact of TEF. For new providers, and “micro-providers” who may offer a degree-level course only to a few students within a much wider body, the data is hard to come by. These providers are still learning the processes and practices that generate data, so even when it does appear it is “new” data – with all of the caveats that this entails.

Students in independent HE are more likely to be “atypical” (demographically, delivery type…) than those at traditional providers, and neatly fall into those groups that are hard to reach with surveys.  Elliot-Bowman described LEO as having the potential to be “an exercise in benchmarks gone mad”, with students likely to see unrealistic or misleading data on likely salaries. Institutions need to be proactive in explaining and caveating LEO to prospective students.

Whilst delegates break for lunch, we recommend that readers take a look at our history of efforts to improve university teaching that have come and gone since 1945. TEF is only the latest instalment in a series of government concerns about the quality of teaching in universities, and is best understood in that context.

In the history of the western university, the idea of teaching as a discrete activity is a recent one, and consensus as to what teaching is and how this should be done has still not entirely been reached.

This, in an age of accountability and measurement, is a profoundly unfashionable idea. But it is a thread that runs from Cardinal Newman through innumerable discourses on the links between research and teaching up to the present day and principled opposition to the teaching excellence framework on the grounds that none of the three words are clearly defined or understood.

The story of teaching quality enhancement can be seen as the story of the replacement – to burlesque Stefan Collini – of the idea of a community with the idea of commerce. This has happened slowly and insidiously, and with culpability on all sides. It is surely right that institutions are accountable for public funds, and it is surely proper that funding is allocated with a purpose in mind. And the parallel has constantly been drawn with research – with a need to “balance” research and teaching paramount.

Read the piece in full here.

We have a question about the involvement of university governing bodies in submissions. Michael argues that they can make a helpful contribution to advising on submissions, but also on making strategic decisions about where the extra funding from increased fees will go. Alastair says that Abertay’s governing body were more informed than consulted.

We also have a question about the helpfulness of the guidance issued by HEFCE on submissions. Alastair says he found the submission a challenge to write, as it was more a retrospective analysis rather than a reflective look ahead that focused on enhancement. Alastair gives a shout out to the HEA for their guidance, which he found very helpful.

That focus on retrospective and evidence perhaps gets in the way of using submissions to articulate an institution’s educational philosophy. But as Jayne says, the whole process has rather provoked consideration or revising institutions’ educational philosophies, and the guidance challenged universities to think again about their conventional assumptions about what they do and why.

Ant has taken the opportunity to ask a question himself. It’s been an enormous year of debate on higher education, and Ant wonders if the energy and oxygen spent debating TEF might have been better spent considering other matters: regulation change, quality assurance reforms, Brexit, and more.

Alastair recognises the frustration that can develop in going round and round in dry conversations about the ins and outs of metrics and process. But, he suggests, the speed of the process has perhaps been one reason for that.

Jayne suggests that much of the energy lobbying on process has really been a complaint about the anticipated outcomes. But it hasn’t all be whinging, and the sector has spent years reflecting and considering teaching excellence, and there have been lots of constructive inputs into TEF that have sought to learn from the history of this.

Mark argues that the sector miscalculated the minister’s determination to see TEF through, which is so much time has been devoted to trying to stall, undermine, or criticise it. Some in the sector were in denial for a long time after Jo Johnson announced his intentions, thinking that a strength of argument could force a minister to back down. This simply doesn’t happen in political life.

A fun question now: what advice would each panelist give to a new universities minister about TEF if we get one?

Mark pleas that a new minister “doesn’t believe the hype” about the current system and isn’t rigid about the current structure of the exercise. The upside of a new minister is they won’t have had two years of “universities moaning at them at every meeting” that Jo Johnson has had.

Jayne advises that a new minister look back at the history of efforts to improve teaching excellence, including our recent successes and failures in quality assurance.

Alastair is blunt with his advice: ditch subject-level TEF. “The burden is going to be astronomical, for very little gain”. He argues that there is plenty of information available at a subject level for applicants, and trying to do TEF here will be too difficult.

Michael’s advice is “be clear about the problem you’re trying to solve”. A new minister should be able to show that students actually want TEF and its future iterations.

Our first question for the panel posits whether institutions written submissions might have been more productive and useful if they were not due to be made public documents? Our panellists generally agree that it would not, at least this time around, as there was simply too little time.

It appears there has been huge variation in how the submissions have been put together. Some institutions have focused very much on data, whereas others were more qualitative. Some institutions have had small teams work on their submission, whilst others have been determined by large committees. And some universities brought in outside help from consultants.

Our next question asks about whether TEF has led to greater investment in careers and employability services. Broadly, yes, say Michael and Alastair. Jayne says that TEF has not so much changed funding streams, but begun a conversation about ’employment’ vs ’employability’. BGU will be focusing more on employability skills as much as employment outcomes themselves.

We also have a question about split metrics and whether their inclusion has forced universities to look more closely at equality of outcomes. The panellists agree that this is one of the most important and positive results of TEF. Jayne says “it forces you to shine a light” on problems that everyone knows have been around for a long time. Splits have forced institutions to “accelerate or intensity activity to tackle differential outcomes”. Mark says “it’s very heartening” to know this is taking place.

Finally on our panel we have Dr Michael Wykes, Director of Policy, Planning and Business Intelligence at the University of Exeter. He begins with arguing that there are two parts to TEF: narrative, and numbers. Both intertwine.

Like the speakers before, Michael highlights that TEF has provoked significant internal conversations with academic practitioners about evidence gathering and new ways of looking at teaching effectiveness. This included extensive engagement with Exeter’s two student guilds.

TEF has also required a common understanding of what the metrics are about. Michael has been asked by curious managers and academics, ‘what is a Z score?’. Making sense of this in common language has been a particular challenge, and DfE, HEFCE and others will need to consider how best to ensure the process is transparent and understandable.

And then there is the narrative and the politics. Exeter vice chancellor Steve Smith was vocal (and perhaps alone) in advocating TEF and its link to fees. The question put in one Guardian article is apt: “will TEF do serious damage, or just puncture a few egos?”

Finally, TEF has provoked attempts to evidence the link between teaching and research, which Exeter has set to do.

“There’s no doubt about it” – TEF has provoked greater engagement and concern about teaching quality within the university, particularly in a research-intensive institution. But going forward, there is work to do in better understanding different models and measure of teaching excellence that are less blunt than the current metrics.

And so we move to our first panel, chaired by Wonkhe’s Catherine Boyd.

Our first speaker on the panel is Professor Jayne Mitchell, Deputy Vice Chancellor of Bishop Grosseteste University, who was also a TEF panel assessor.

Looking back to TEF’s origins, Jayne states that her institution welcomed the attention behind TEF because it would give small, teaching-focused institutions a chance to demonstrate where they are best. However, TEF as it has come together – particularly the metrics and benchmarking – have created some very real challenges for small institutions. Outcomes for these institutions can be swayed by very small changes in metric outcomes.

TEF has prompted institutions to reflect on the depth and breadth of their academic portfolio. It will also be a challenge for communications and marketing departments, and impact how the university presents itself to the outside world. And TEF has also influence BGU’s business planning. The process (Jayne’s emphasis) of preparing for TEF has “been a worthwhile exercise, irrespective of what the outcome is next week” – the focus on making the most of teaching and pedagogy, creating a more robust evaluation framework, and evidencing teaching effectiveness better.

Still, there are unintended consequences of TEF. Jayne fears that TEF will separate quality assurance and quality enhancement processes. The nuances of how the data and benchmarking are flagged will also likely be lost on casual observers. Crucially, TEF appears not to recognise the distinction between baseline quality assurance, and celebrating excellence over and above that standard. The big question is: will TEF beg a rephrasing of the famous question: is a Gold at your university vice chancellor, the same as a Gold at another university, vice chancellor?

Next up we have Dr Alastair Robertson, Director of Teaching & Learning Enhancement at Abertay University in Dundee.

Alastair begins with a reflection on the divergent quality enhancement cultures developing in Scotland and the rest of the UK. He suggests that Scotland, in some ways, has been reactive in this area, responding to developments elsewhere with explicitly ‘non market’ approaches to enhancement. However, in Scotland as elsewhere, data driven evaluation and enhancement of university teaching is here to stay.

“We have not”, says Alastair, “seen a clear lead from the Scottish government or funding council on Scottish universities participation in TEF”. Universities have largely been left to make their own decisions, and are able to raise their fees for English students without participating.

Abertay is one of only five Scottish universities to enter TEF – interestingly, all of them are on the east coast. The main incentive for participating is reputational, and those that have entered expect to do well. Aside from the benefit of getting the outcome they hope, participation in TEF has had internal benefits, encouraging the university to ask questions about evidence-based enhancement. Abertay thus feels well set up for the new Enhancement Theme to be announced later today in Scotland.

However, TEF has polarised the student and staff community – participation is opposed by Abertay’s student association.

Next we have Wonkhe’s Ant Bagshaw introducing the first phase of our ‘Accentuate the PosiTEF’ exercise. Ant asks our assembled delegates to give their collective view on their confidence in TEF against its stated aims, and their optimism or pessimism on its future.

Unsurprisingly, there is a general wariness towards the exercise and a cynicism about whether it has achieved its stated aims. Our attempts to quantify this through an online voting system sadly crashed the wifi… But we have boards up in the refreshments area, and we’ll be feeding back on delegates’ views through the day. As Ant puts it – we are getting in ahead of the planned review exercise.

Ant goes through what we at Team Wonkhe will be focusing on next week. Will there be any surprise results? What do the results tell us about the operation of the panels? And how are results being spun?

On that note, the excellent Jess Moody of ECU has shared on Twitter her ‘TEF bingo’ for the day.

‘The Incredible Machine’, for those who were wondering, was a complex computer game from the nineties, hence Wonkhe’s moniker for the TEF.

Mark reflects on the debate and journey that TEF has gone through. In particular, though the Higher Education and Research Act is only peripherally concerned with TEF, the looming exercise dominated the parliamentary debate. The Duke of Wellington’s intervention was just one example of some of the surreal moments during that debate.

Though there are many similarities between TEF mark 1, launched in November 2015’s Green Paper, and the TEF that we now know, the exercise has only grown in complexity.

Wonkhe’s archive of nearly 120 articles on TEF is reflective of that complexity and the attached policy debate.

“The train has left the station” on next week’s results release, though a new minister could chose to do any number of things.

Looking ahead to today, we’ve got a lot of material to cover, with contributions from expert colleagues. We’re hoping to have a mix of contributions – “there will be time for moaning” – but we’re also hoping to encourage constructive engagement with our ‘Accentuate the PosiTEF’ exercise.

And so we begin today’s conference with our Founder and Editor, Mark Leach, welcoming our packed conference hall and reflecting on the origins of TEF.

“It’s been an interesting 18 months”, Mark tells us, and we move to a large slide of Jo Johnson, our esteemed (still) Minister for Universities, Research, Science and Innovation, greeted with a few groans…

In September 2015, the then fresh minister told assembled vice chancellors at UUK’s conference that universities needed to fix “lamentable teaching” and so introduced the Teaching Excellence Framework. Mark recalls the “ashen faces” of some university leaders as they realised what was about to come upon them…

Good morning. We’ve got an excellent day of debate and discussion on TEF lined up for you. Follow us here for live updates, and on #WonkheTEF on Twitter, beginning at 10.30am.

Speakers today include:

And hear from Team Wonkhe’s Mark Leach, Ant Bagshaw, David Morris, Catherine Boyd and Nona Buckley-Irvine