The summary statements for the Teaching Excellence Framework 2023 (TEF) have all now been published, along with the summary statements, overall results, aspect ratings, provider and student submissions.
Indicators for providers over the 2018-2022 time-period are also freely available as part of the Office for Students’ TEF data dashboard.
There is a wealth of data and information for institutions to use to enhance their provision.
Within this context, it is instructive to consider one innovation brought in for TEF’s 2023 iteration. In the student outcomes aspect, three features assessed a provider’s approach to educational gains. The inclusion of educational gains in the TEF was one of the recommendations that arose from the independent Pearce Review, which called for “institutionally determined evidence addressing what our students gain from our educational experience and how we evidence that”.
In searching for lessons based on what providers said about educational gains in their provider submissions we welcome the crowd-sourced work by Debbie McVitty and Wonkhe, and we acknowledge the work underway as part of a QAA Collaborative Enhancement Project – both of which have also sought to examine educational gains.
What’s an educational gain?
Educational gains, previously referred to as learning gain, has a complex history in higher education. Indeed, almost ten years ago, the OfS’ predecessor – the Higher Education Funding Council of England (HEFCE) – supported a £4m initiative including 13 different pilot projects that attempted to define, assess, and scale a national concept of learning gain. The basic premise was to see if learning gain, loosely conceived, could be captured during the course of a degree – and whether value was added which could be quantified and measured in a way that had national applicability.
It proved hard to define, and even harder to scale across institutions. And we shouldn’t have been surprised. Institutions have different educational missions and purposes, consonant with factors like the nature of their applicant and entrant base, their regional location and civic responsibility, and their specific context (colleges and universities, teaching intensive, research intensive).
TEF guidance
Considering the challenges that were identified in the HEFCE projects and discussed in the Pearce Review, the most recent TEF guidance from OfS – Regulatory Advice RA22 – recognised that there were no national measures of educational gain, and gave institutions the scope to define what they individually meant by educational gains: across dimensions of intended and articulated gains, support to achieve gains, and demonstration and evaluation of gains.
The guidance drew very much from a provider defined approach, recognising that this was something new for the sector. It was explicit that if a provider was unable to show evidence that it was evaluating gains it would not be prevented from being awarded higher TEF ratings solely on this basis. Interestingly the guidance goes on to say that this approach is intended to allow providers time to establish their practice in measuring and evidencing educational gains which could then become the focus of assessment in subsequent TEF exercises.
What happened
If TEF 2027 goes ahead, and it looks likely that it will, educational gains from intention to measurement may well be an integral part of the next assessment. We thought that we would look at what we can learn from current submissions and summary statements. To that end, here is a little demographic data. Of 227 published summary statements (at the time of writing):
- Only 18 institutions received an “outstanding” judgement across all three educational gain features.
- A further 20 were given any combination of two outstanding and one very high-quality judgement on these features.
- Overall, most “outstanding” judgements were awarded to SO4 (intended gains), at 82 providers. Next was SO5 (approaches to supporting gains) at 54 providers, followed by SO6 (evaluation and demonstration of gains) at 22 providers..
- The most common pattern is of silver providers with a mixture of very high quality judgements and insufficient evidence to judge.
- Two outstanding judgements and one insufficient rating was rare, being seen for just eight providers.
- And not all institutions judged outstanding across two or three features, or across two, were gold institutions. Some 23 gold institutions did not receive outstanding across all features, or outstanding plus one very high quality rating.
- Indeed, four gold institutions did not have a single outstanding educational gain feature.
- Only one bronze provider received an outstanding judgement for any educational gain feature, in the context generally of insufficient evidence to judge ratings. And no providers rated “requires improvement” received an outstanding judgement for educational gain features.
This pattern suggests that it was hard to receive outstanding judgements across all features of educational gains, recognising the embryonic stage of development that the sector is at in relation to the concept.
It shows that providers may have been comfortable in articulating intended educational gains for their students (SO4), but find it far harder to articulate how it is demonstrated and evaluated (SO6). That there is a relatively loose association between an overall rating of gold ratings and outstanding judgements across educational gains features implies that gold providers who are outstanding for measures of continuation, completion and progression, but less strong on educational gains, are potentially vulnerable in the next assessment to not achieving (or for some maintaining) gold for student outcomes if their ability to define, measure and evaluate gains does not improve.
Bronze providers and those silver providers with a mix of very high quality and insufficient features need to be clearer about what their intention and approaches to educational gains for their students are, and their theory of change to effect and demonstrate them – if they are to enhance their ratings for student outcomes in 2027.
A win by submission
It is also instructive to look at those gold providers with outstanding judgments across all educational gain features and see what they say in their provider submissions. Here we focus on 6 full-service universities, rather than on the further 6 who are either specialist institutions (such as RADA, Chicken Shed, Harper Adams, Spurgeon) who one might argue have more focused mission underpinning their educational gains, and two Further Education Colleges.
Aston has a clear conceptualisation of intended educational gains through a “students as partners” model. Employment and employability is woven through courses: career development in 80 per cent of programmes, career and personal development start in year one, while year two students source their own placements, thus integrating work experience and placements into all programmes. In terms of approaches to supporting educational gains, large scale collaborations with employers, extensive placements, former students are invited to play a role in supporting current students’ employability. Evaluation and demonstration are achieved through student surveys that evaluate educational progress, learning analytics are used to monitor and track student progress, ensuring early intervention when needed, and high completion rates of courses. The student written submission endorses this approach.
Bath has a strong emphasis on gaining the high-level skills needed to be able to apply advanced knowledge and sophisticated understanding that leads to positive/strong graduate outcomes and employment. They have co-created their approach with the Students’ Union and they approach the alignment of their subject mix and real-world application through the embedding of skills, placements/study abroad and developing employability. Their approach to evaluation is grounded in their strategy and a focus on graduate outcomes and graduates progressing to highly skilled employment.
Hull has a coherent and strategic approach to educational gains focusing on the development of the whole student through a competency-based approach to knowledge management, professional and disciplinary experience including development of professional networks and self-awareness. Educational gains are threaded through programmes via the competency framework and transforming programmes initiatives. There is a focus on inclusion and personalisation. Placements and internships feature and evidence of success is through very high quality progression rates. Evaluation and demonstration are via a range of internal surveys and through continual monitoring, evaluation, and enhancement.
Leicester establishes a clear vision of students as citizens of change emphasised well in the student submission too, and clearly demonstrates that its values are reflected in everyday practice. They clearly articulate the aspects of their intended gains (confidence, criticality, social responsibility and careers readiness) with clear descriptions of each and they can demonstrate how these are embedded throughout the curriculum. In addition, an approach to curriculum enrichment and support uses insights from a range of data to create an ‘educational gains evaluation framework’ that demonstrates a clear and robust way to ensure consistency in evaluation and continuous improvement, by mapping core measures to each element of its definition of educational gain.
At Plymouth educational gains are built into the university mission and vision through the Plymouth Compass framework for graduate attributes across all its programmes. The submission describes an evidence based theory of change for ensuring educational gains, including career readiness, knowledge, civic and societal knowledge exchange and increased aspiration and confidence. The provider is able to describe the impact of educational gains on its students, including specific interventions which are built on in partnership with the student union and external organisations. This approach has resulted in a metric that measures how far students have developed.
Solent’s approach to educational gains is built round work readiness and commitment to social mobility, social justice and inclusivity – recognising that not all its students based on their starting positions come ready for the journey. Gains are focused on confidence and personal insight, global and ethical awareness and work readiness and commercial grounding. Approaches to supporting educational gains come from evidence of embedded inclusive learning teaching and assessment, relevant library and careers support, a survey designed to build understanding of career readiness, and evaluation through a checklist linked to four stages of impact: with a clear approach to evidence of impact and that students are achieving educational gains.
Gain stages
So, what are general lessons from these examples and our wider learning from the TEF 2023 exercise that could inform what an “outstanding” conceptual framework of educational gains might look like? Here are a few that occur to us:
- Approaches appear to adopt one of two broad models: a competency/outcomes model focussed on good student outcomes and progression to employment, or an input model focussed on the curriculum and student co-creation.
- A conceptual framework of educational gains should be shared and ideally co-created with students, with a golden thread through courses, curricula, co-curricular and extra-curricular.
- Any approach to educational gains has to be coherent and strategic, supported, evaluated and able to be demonstrated. Some examples of where this worked best was in the local economic focus adopted by the best further education colleges, the occupational articulation of the land-based colleges and where a very small number of the research-intensive multi-faculty institutions were able to move beyond a simplistic “research informed teaching” rhetoric in their articulation.
- A conceptual framework which can link with greater work readiness and employability, often through placements and work experience, and is able to clearly demonstrate how this is deployed in practice and has real meaning across all students and in all subjects.
- A conceptual framework that can link with acquired skills whether they are professional, personal, or social. This is often referred to as graduate attributes, but need to be expressed in a dynamic way that permits growth and development.
- A conceptual framework that can be evaluated and demonstrated via learning analytics, surveys, completion, and progression indicators, student portfolios and reflection, allowing the development specific metrics and evaluation checklists.
- ·A conceptual framework that is widely understood, and communicated across the organisation, students, staff, graduates.
As the clock begins to tick down to the TEF 2027 and the next assessment, providers should be reviewing their commitments and promises from TEF 2023 and looking to ensure their educational gains approach is clearly articulated, delivering in practice and being measured. If, that is, they are to avoid opening another blank Word document with the words “educational gains” staring back at them.
