Legitimacy and Ofqual

With 2021 A level results fast approaching, Roger Taylor – former Ofqual chair – has taken the opportunity to write up his impressions of last year’s “examnishambles” for the Centre for Progressive Policy.

You may already have seen the Guardian’s write up of the intervention – Education Editor Richard Adams picks up that Taylor’s preferred position would have been not to run A levels, and to expand the number of places available at university to compensate. Expanding provision is what ended up happening – but how did Taylor think admissions should have been handled without A levels? Strap in for an absolute rollercoaster of excitement as we read the footnotes of an essay published by a think tank.

Footnotes coming home

We start our tour at footnote 8. Taylor suggests that the aim of the much discussed “mutant algorithm” was to predict what the results of A level exams would have been, but is also clear that the notion of exam results themselves being an imperfect observation of an accurate “true grade”. Indeed, Ofqual analysis has suggested that the “calculated grades” (the ones based on teacher assessment spat out by the algorithm) may more closely match this ideal grade than exams.

But accuracy and legitimacy are not the same thing. Exams have a legitimacy because they have become the accepted proxy for the “true grade” – if someone tells us they got an A in A level economics we understand that as them being in the second-to-the-top slice of students that sat the exam in that year, and thus (comparatively) good at economics. Although we may cherish a school report that says someone is working at an “A” standard in economics, this teacher judgement (based on the teacher’s impression of the performance of a given student compared to similar previous students) is not afforded the same weight – it may be a more “accurate” assessment of a student’s potential but it is less legitimate.

Taylor puts it like this:

it can be rational to prefer a legitimate but relatively inaccurate system over a more accurate but illegitimate alternative

Which is one of those lines that makes you stop and think faintly heretical thoughts.

Too legit to quit

Keep these ideas in mind as we advance to footnote 12. In the main text, Taylor is making the relatively uncontroversial assertion that if an algorithm is a hammer then not every data-related education problem is a nail – the note applies this old saw to the summer’s unpleasantness. The point here is that there was (according to Taylor) no discussion of any solution that wasn’t an algorithm – and in footnote 12 we get such an example.

The postulated alternative was:

• Students and UCAS receive notification of teacher awarded grades (the Centre Assessed Grades – CAGs – that students ended up getting)
• Ofqual issues a “school leaving certificate” to meet their duty to maintain standards.
• All university offers based on the usual mix of predicted grades and contextual information are cancelled.
• University admissions teams then make offers in light of the CAGs

So, Taylor’s alternative model would have seen confusion and anger over grades compounded by all students losing their offers and entering what basically amounts to clearing. Here we can just spoon in the established findings regarding the undesirability of students making decisions at the last minute.

Why would anyone argue for such a thing?

By putting the final decision in the hands of the universities it would quite likely have had greater legitimacy. But it still feels a something of a stretch to think this would have been acceptable.

Do feel free to pause to laugh at that last line, but it’s the first one that concerns me. We’re back to the idea of “legitimacy” here, and to Taylor university admissions processes are especially legitimate for reasons that remain unclear.

Algorithms – they taste awful but they are good for you

One of the most powerful positions Taylor puts forward is that the public were as angry about the algorithm being correct as they were about it being incorrect.

Putting into hard code the fact that the school you attend is a predictor of the grades you are likely to achieve does not seem to square with meritocracy

As I’ve lamented before, an “algorithm” (grading to a curve) pollutes the accuracy of A level results every year, and the impact of socio-economic background, disability, and parental education can be starkly seen in any set of exam results. As the report relates, this is why universities and employers use contextual data to aid there interpretation of grades and use outreach to support disadvantaged learners. And – yes – some of these approaches are also driven by algorithms of various sorts – as are some of the means by which awarding bodies are attempting to improve qualifications to meet these needs (“comparative judgement” is the example cited).

Algorithms, in other words, are the future of grades – the problem lies with the public (vote them out! we need a new public!) refusing to accord legitimacy to non-human centred processes – something seldom dealt with in policy making.

Legitimacy is when people accept the authority of a decision making process: errors, bugs, biases and all. Legitimacy is perhaps less a property of the decision-making process itself, and more something to be found in the attitudes and beliefs of the people affected by it

What you could have won

The “non-qualification” leaving certificates were on the table in the summer. This would have been a perfect outcome for Ofqual as it would have taken the agency off the “legitimacy” pitch and allowed schools and universities to sort out admissions for themselves – with applicants and those who care about them punted into even more chaos.

The board made the depressingly cynical calculation that:

the elected government has more legitimacy in deciding what will command public confidence than the regulator

Yes, legitimacy again accorded to populism rather than expertise. That said, there was no defensible rule that Ofqual felt empowered it to change “obviously wrong” grades and getting through a judicial review.

Though this document is constructed as something as an apologia for Ofqual’s actions, it actually demonstrates the issues that the timidity of the agency – and a lack of willingness to engage meaningful with the idea of legitimacy – caused in bringing the 2020 exam season to such an unsatisfying end.