The problem of graduate jobs and odd SOCs

It’s one of my external speaker go-tos that if you want to follow higher education policy the he-said/she-said of Westminster and Whitehall may be fun to watch, but real policy happens in data definition.

And here we have a perfect illustration of that phenomenon. If you’re not a HESA-literate with responsibility for Graduate Outcomes you might not be aware of the ongoing battle over how SOC codes are applied to graduate survey responses – an issue that rises in importance as it becomes clear that the “graduate job” is one of the few non-laughable indicators of a “low quality” course that DfE and the Conservative Party have in their respective arsenals.

The process goes like this – a graduate is contacted as a part of the survey, and replies with their current job title. She’s a “Digital Learning Developer”, or a “Constituency Assistant”, or a “DevOps Engineer”. The contractor that deals with survey responses – Oblong – then has to match this with the Office for National Statistics (ONS) Standard Occupational Coding (SOC) framework. As there are an infinite number of possible job titles, and only 412 units (which are now underpinned by the sub-unit groups, of course) a certain amount of skill and judgement is required to assign the correct code.

By the time we see these codings in public-facing data they are aggregated to the very highest 9 SOC levels, of which the top three are deemed to be highly skilled. DT072 table 7 is the one that provokes the provider nightmares, but in reality it is not the whole story. Each individual SOC unit group includes a description (“typical entry routes and associated qualification” that pushes the role into one of four categories – a degree is required, a degree is common, a degree may also be a possible route, and a degree is not required.

Here’s my interpretation of how that looks for unit groups (note this is SOC2020, which will be used for all Graduate Outcomes releases going forward).

[Full screen]

It certainly feels like universities might have an interest in ensuring that whatever their graduates put on the form is coded to a graduate job. And that’s the source of a very interesting section (4.1 – common misunderstandings) in a recent HESA report into the Graduate Outcomes SOC coding assurance process.

Here’s a taste:

Salary cannot be solely used to denote a ‘professional role’. There are a wide range of factors that influence salary, including location, age of graduate, employer etc. It is not a reliable predictor of major group. An assessment was carried out on the overall salary distribution by SOC major groups at a national level in the additional quality checks described above and as requested by the sector.

and

A job cannot necessarily be coded differently just because a graduate has a degree or is highly qualified for a role, especially without further evidence. Equally, there is an assumption that if a degree is required an occupation must be in Major Groups 1, 2 or 3, which is not always the case.

and

There are certain employers that HE providers argue should be coded differently regardless of job title and job duties. This is not an approach we recommend.

HESA points, righly, to the process descriptions it has already published. A read through these illustrates both the effort that has been taken to carry out coding in a clear and consistent manner, and the necessarily human nature of the coding process.

It is a common phenomenon when descriptive data is used for blunt and clumsy policy processes – a necessarily interpretive activity starts being treated as if it is the application of a natural law. Even the skills description of the units are focused largely on what qualifications current and past employees hold rather than what might be desirable or useful. Both the HESA report and the independent examination by the ONS conclude that the job is done well and reliably for statistical purposes. But if we want to base funding decisions on Graduate Outcomes (or any other SOC-based metric) there is a problem