The Knowledge Exchange Framework: year zero

I’ve got a theory about KEF – if it didn’t have such a flashy name it wouldn’t get half the attention it does.

The Knowledge Exchange Framework is not (like REF and TEF are) an “excellence framework”. It doesn’t make any judgement on the quality of business and community interaction, just on the proportional volume and likely output of a number of activities described in the HE-BCI survey data. Neither is it of use to professional or armchair rankers – it doesn’t offer named awards or simple stepped gradations that demonstrate one thing is unfailingly better than another.

It may eventually be used to support the allocation of the £200m Higher Education Innovation Fund (HEIF), which is currently allocated using similar data. But for the first year (2020-21) it is for entertainment and edification purposes only.

Note: the data presented here is under review, and has been updated several times with the support of many specialists in the sector. Please read the caveats carefully.

KEF under the bonnet

Though we’ll have the results of the first iteration of KEF in the summer, much of the data that underpins them has not yet been published. We’re expecting the next iteration of HE-BCI data from HESA in April – meaning currently we have only two of the three years of necessary data. What’s more, a small amount of data in the form of self-assessments and narrative statements still needs to be collected, and there is a co-authorship of research outputs strand where a suitable source needs to be identified.

Two or more metrics sit under seven perspectives as follows:

The major change from the consultation is the use of a subset of institutional income (funding councils, tuition fees, research contracts) as a way to control for institutional size in place of staff numbers. Income from facilities and equipment are now also included when considering working with business. And a measure of academic time commitment to public and community engagement has been replaced with a provisional self-assessment score based on an instrument co-developed with the National Co-ordinating Centre for Public Engagement (NCCPE).

Metrics are a three-year average, mostly (as can be seen from the table) as ratios, which are converted at perspective level into deciles. This reduces a great deal of data and analysis into what amounts to a set of marks out of 10, which are compared to an average mark from comparable institutions (the infamous clusters) – externalising the strategy office job of identifying sensible comparators via a superbly rigorous report. The composition of these clusters is mostly as in the consultation, there are a few changes to some of the smaller groups that will be made in consultation with the institutions in questions

But what will it look like?

Research England has a grand plan to use spider graphs to show institutional scores alongside cluster averages, with an option to drill down into more detailed data on each metric. I’m not as struck by this as they are – the exercise is designed to support comparisons and spider diagrams are an unwieldy way to do this. I also feel like the individual metrics are still fairly abstract, you have to go quite a long way back down the methodology to get something that the mind can easily take hold of.

But I wanted to give you some sense of what the results would feel like, so – using the last three years of available data – I’ve plotted the top level metrics as two simple bar charts, one each for a provider and their appropriate clusters (using the originally consulted-on groupings). Because I was interested enough to run this imaginary exercise UK wide, there are extra groupings for Wales, Scotland, Northern Ireland, and I’ve also included a number of English HEIs that would not be required to enter the KEF. These additions do have an impact on the (ranking-derived) scores for core English HEIs. Think of this as an expanded KEF year zero, if you will.

[Full screen]

On the main dashboard you select the appropriate comparison cluster by moving your mouse pointer over any of the institutional bars. I’m not convinced that, on average, there is a massive difference in terms of these metrics between the clusters, something you can see a little more clearly via the “cluster comparison” tab.

Update: This presentation has changed several times following feedback and suggestions from the sector. It’s never going to be perfect, but it will become more useful (and more accurate, within the bounds set by the expanded way I’ve run the exercise) as time passes.

Closing thoughts

I’ve hung around a lot of league table compilers to know about the sniff test – a ranking needs to look right (with the right kinds of provider at each end) as well as have a defensible methodology. However we try to deal with our prejudices, all of us carry around an idea of which providers we would expect to find where in any given instrument – which is what makes this first glimpse of KEF all the more fascinating. Though this isn’t a new set of data or a new area of analysis, Research England has succeeded in developing a refreshing and challenging new tool to make sense of what happens in the sector. I’m fascinated to see it run for real in the summer.

Caveat: Though (as with all these things) I’ve run KEF year 0 to the best of my ability, it is entirely possible that I may have got something wrong. If I have the error is mine – so please complain to me and not Research England. I’ll continue to endeavour to fix issues that people spot – and I’m always happy to chat about KEF!