The Institute for Fiscal Studies (IFS) has been a key facet of the Department for Education’s (DfE) attempts to drive interest in and acceptance of the Longitudinal Educational Outcomes (LEO) data releases.
Today, we are looking at 10-year salary benefits of particular HE institution and subject area combinations. Earlier this year, we saw the first of what will be four releases born of this collaboration, on relative earnings benefits. The final two releases will look at postgraduate and late career salary premiums.
With this whole series, the aim is simple – the government wants to normalise the idea of thinking about the quality of university teaching via graduate earnings. The headlines that are generated may not bear sustained scrutiny. Indeed, the IFS report itself doesn’t bear sustained scrutiny, but that isn’t the point.
Into the data
The data analysed here looks at the salary at age 29 (those who took their GCSEs between 2002 and 2007) of those who attended a particular institution and studied a particular subject full-time. We’re looking at those in employment (with PAYE contributions) only. Earnings are compared with a representative population of non-HE participants of the same age with five or more GCSEs at grades A*-C – one thing to note is that sector-wide data (not data matching the characteristic of individual institutional intakes) is used for these comparison populations.
Data note: use the filters to select the sex and subject you are interested in. This reflects subject provision about 10 or so years ago, so subjects that are taught in 2018 by an institution will not feature. If you want to filter out subject/institution combinations with a low population size, use the slider provided – there are some very small numbers in there.
If your ears are twitching at the idea of the ten-year cohort, then welcome to my world. To say that students starting work at the outset of the 2008 crash may have had an atypical employment experience is perhaps being kind. On any time series, this group would be an outlier. Even beyond the more general issue of assuming the next 10 years will have anything in common with the last 10, this is of questionable use to those making decisions about where and what to study.
The subject component looks at broad subject areas (CAH2, with some bespoke additions), though the report unhelpfully uses the language of courses. Each subject area will likely contain multiple courses (and parts of multiple courses), many of which may no longer be offered by the institution in question. Course to subject code mapping is not always clean or simple – and this data goes through many such mappings, from the actual course as delivered to JACS and then from JACS to HECOS – before CAH aggregation and partial disaggregation for this report.
The non-completion issue
I was wary of the inclusion of students who dropped out of HE. We already know that non-completers see a salary detriment compared to the non-HE population and we know from other sources that non-completion is highest among non-traditional entrants. So subject areas and institutions that do best at recruiting hard-to-reach entrants will be unfairly penalised.
The early parts of this graph (from HEFCE data) overlap with the studied population. You can see a clear link between non-continuation and POLAR3, non-continuation and living in a parental or owned home, non-continuation and non-standard entry qualification. Institutions enrolling students with these characteristics will see a detriment in their average graduate salary figures.
IFS did control for a similar list of characteristics (free school meals, special educational needs, English as an additional language, deprivation indices, ethnicity and Key Stage 4 educational attainment). But comparing a graph that looks at graduates only suggests that there is a significant difference in many cases. And we know that graduates earn more than non-graduates – kind of the point of this report. All of which prompts the question – why look at all attendees if you know there is already significant negative detriment for non-completers? Happily, there’s graduate-only data – which should really have been the focus of the report.
There’s also no compensation for the salary impact of a person’s domicile age 29. We know that similarly skilled individuals in, say, Stockton-on-Tees and Southwark, working in similar roles at a similar level will earn different amounts. So University of Teesside graduates (again, for example) who may choose to stay and work in the area after graduation will drag down the median salaries in this analysis.
What’s maddening is that both of these would be very easy to control for. Non-continuation data is available from HESA, and I refuse to believe that HMRC does not collect the home postcodes of PAYE taxpayers. Without these controls it is very difficult to make assertions with any competence – and this many be why the report itself claims not to infer causality.
Politics of salary comparisons
But it is Sam Gyimah’s interventionist language that worries me most. If he expects OfS to intervene based on these findings (which was the impression he gave me) then he needs to be clear that an age 29 salary detriment is due mostly or entirely to the quality of HE provision. Without controlling for region or qualification status, and without a proper historical treatment of the data, this assertion can’t safely be made. We are seeing the effects of poor quality salary data in policy already – as institutions like the University of Bolton would perhaps most easily address OfS registration conditions by upping sticks and moving to Bloomsbury.
Statements and press releases have included, at least in passing, the idea that salary might not be the only measure of higher education success. The idea of hard-working nurses and diligent social workers – or the artists and writers that contribute to our idea of a civilisation – is waved at us as a token alternative to a purely salary driven metric. But without the corollary that they should perhaps earn more, and that in many cases it is within the gift of the government to make that happen.
I’ve always held the position that this is interesting research data, but it is not useful for policy making or application decision making. But Sam Gyimah feels it is “better than nothing” for both those use cases. It isn’t – it is actively unhelpful. For all the prestige that the IFS brand offers, this is political data designed to act as a signal in the still fondly hoped for HE market.
Thanks David, this is a really helpful analysis. The lack of regional benchmarking is a fundamental flaw in the whole exercise. We see institutions in regions where salaries are lower being penalised: to be meaningful, the reference points should be the relative salaries of the A level cohorts vs the graduate cohorts within the same regions.
As you point out, and has been said about the use of LEO previously, we should ship all our graduates to the South-East!
Quite apart from the absolutely excellent points you’ve already made as to why this research doesn’t work, even assuming it did work – to use it in the way the Minister is suggesting doesn’t make sense in the government’s own logic.
For example – artists and writers don’t just contribute to our idea of a civilisation, they very practically contribute to the creative industries, which last I looked were growing at twice the rate of the rest of the economy and represent a huge export success for the UK. A significant number of the people driving that notoriously low paid (at the age of 29) industry are graduates of courses and universities that do poorly in this assessment, meaning in the Minister’s conclusion they are poor value for money.
But having no account of the industry they are going into (generally low pay) and your relative position within it, let alone the impact that industry has at a national economic level (creative industries are a supposed key pillar of the government’s industrial strategy) means judging courses this way, even if the data was accurate (and PAYE isn’t used by the self-employed contractors that make up a big portion of film, tv, theatre etc leaving again a big gap in assessing the outcomes of those courses) wouldn’t make sense when looked at holistically with the government’s overall ambitions.
Depressingly short sighted or distressingly wilfully misleading from the Minister. Again.