The best laid plans of mice and men might have gone awry due to Brexit, but we now know that the Teaching Excellence Framework will go ahead as scheduled. Millions of pounds have already been poured into the quest for new indicators of teaching excellence, including measures of learning gain, a ‘highly skilled employment’ metric, and the longitudinal learning outcomes dataset (LEO).
At EvaSys we asked a sample of teaching experts from over thirty UK universities for their views on the TEF year two proposals. Their hopes, fears and advice are an important message for Jo Johnson and DfE as they take forward the development of the TEF.
Good intentions
A senior academic and member of a university executive team spoke for several of our respondents when arguing that “a push towards greater recognition for teaching is well overdue; the TEF could result in students being taught by people who have greater understanding of what teaching in HE is about, rather than those who just teach how they were taught; not just doing the job, but doing it really well.” Another respondent suggested that TEF would help students make better decisions based on a “heavyweight” analysis of what they will experience. The three aspects of teaching and learning quality identified by BIS – teaching quality, learning environment, and student outcomes and learning gain – were recognised as the right areas to focus on.
The proposal to develop teaching quality assessments at discipline level from TEF year three was also welcomed. One academic observed that “the NSS will never address the nuances of the total student experience across a full undergraduate programme … institutions need … more sensitive internal processes to get a more sophisticated picture of what’s going on at a local level.” Another even suggested that “further detail is needed to separate out the student satisfaction scores at the module level from the NSS scores, [which] is a post-hoc, programme level metric and has little utility in highlighting day-to-day measures of teaching excellence.” In fact, 71% of our survey participants reported that their institutions already have mechanisms in place to monitor student satisfaction and/or progress throughout their programmes in order to address issues more promptly, and presumably be prepared with the best possible scores when it comes to TEF assessment.
Those HE providers who are using the new EvaSys MBE Module Benchmarking™ service will be a step ahead in this respect. In the first benchmark period for the service (September to December 2015), 13 participating HEIs, ranging from a Russell Group institution to a specialist business school, submitted data from 14,253 module deliveries and 314,705 student responses. Each HEI received a National Benchmark Report, comparing institutional module performance by subject area, and an Institutional Benchmark Report, enabling these providers to benchmark module deliveries and modules internally, and against equivalent modules from the sector. The reports also show results for module instances and module delivery over time, making it possible to see how well a module is delivered at different times or in different locations by different teaching teams. This is a much more precise and responsive source of student feedback than the NSS can ever be.
Numbers game
The BIS TEF2 consultation conspicuously omitted any question about the planned core metrics: NSS, DLHE and retention data. Nonetheless, we asked our informants for their views on these in any case, and while some were satisfied that these metrics would provide appropriate and reliable proxy measures, others were not shy about expressing their concerns. One senior academic warned that in order to work, the TEF must have the respect of every stakeholder, and that even BIS had acknowledging the inadequacies of the NSS and DLHE scores as proxy measures of teaching quality and student outcomes. Furthermore, respondents predicted that an entirely metrics-based approach will lead to ‘gaming’, and will be unlikely to achieve the aims of improving teaching quality and driving excellence.
Advice from the academy
Our respondents were not short of advice on how to improve the TEF framework. They felt strongly that educational value cannot be measured solely by graduate employment and earnings, and stressed that the employment and employability agenda should not be allowed to hijack the concept of true teaching excellence. Respondents argued for the wider social value of higher education, career diversity, and student well-being to be captured in order to prevent higher education drifting towards a narrow focus on ‘professional training’. Many were also disappointed that the TEF appears to lack opportunities to capture and disseminate examples of excellent practice across the sector.
The concerns raised by our respondents do throw doubt on the extent to which TEF assessors will be able to make fair and robust judgments of teaching quality. As one respondent said, “TEF assessors will be able to make a judgement but the issue is the extent to which it is ‘clear’, particularly in relation to the distinction between ‘excellent’ and ‘outstanding’”.
They (and we) hope that Jo Johnson will listen seriously to the voices of those who know the business of teaching and learning in higher education better than anyone, and give this critically important process the time and effort it deserves to develop a truly transformational framework.
TEF will “have to” (as in will be politically steered to) avoid the two most likely outcomes:
1. the results will be so wildly at odds with existing perceptions – e.g. (as in THE mock TEF) Cambridge ranked almost level with Bishop Grosseteste (two fine but very different institutions);
2. the results will just closely replicate existing league tables.
In the first case it will, regardless of how worthy, undermine public (and more importantly political and press) confidence in the TEF.
In the second case the question will be asked “what was the point of that?” with the simple answer “more money into the pockets of bureaucrats”, which is a nice story for the Telegraph.
So if “…BIS TEF2 consultation conspicuously omitted any question about the planned core metrics…” that could be a way of allowing sufficient wriggle room to avoid these two extremes. In which case, TEF is doing political work, not genuine teaching quality work.
I should like to know what the TEF assessors are going to use as final criteria for their reports and who is going to pay the bill for their salaries? Perhaps we should like to see several women to be assessors as well (not because they cost less).