A recent Wonkhe blog from Julian Crockford, which offered a response to sector noise about evaluation of access and participation work, alluded to an “evaluation collective” which Rachel Spacey and I have been leading.
Our humble beginnings late last year were about support for common issues, until evaluation became the new hot topic and a flagship motto for the new Director of Fair Access and Participation.
For those of us working in this space, this increased sense of importance (something we already knew) was both pleasing and unsettling.
We didn’t set out to be disruptors, but a group of like-minded colleagues from various universities have now co-created an Evaluation Manifesto to challenge dominant narratives in this policy space.
Evaluation should be a collective and transformative action
Our Evaluation Collective defines evaluation as an approach which helps to understand and explore what works and doesn’t work in a given context and is of value to stakeholders. The aim of evaluation is actionable evidence-informed learning and continuous improvement of process and impact.
We stand against the performative, task-focused, tick box evaluation machine, and define evaluation as a collective responsibility within organisations, focusing attention and effort on transformative educational outcomes.
Evaluation advocacy can be an empowering tool for social justice
The Evaluation Collective takes the position of advocates of evaluation. The scope of our advocacy spreads throughout access and participation work in higher education. We believe that evaluation can be an influential tool for social justice and should be used to amplify marginalised voices and minimise barriers to educational outcomes.
We believe that the evidence gathered through evaluation can be empowering, especially in ensuring responsibility and accountability for “fit for purpose” activities, systems, processes, policies and practices. Our Manifesto aims to generate support for those seeking to understand the impact of their current and future activities, and develop confidence through our advocacy.
Evaluation will evolve in visible, inclusive, and supportive spaces
As individuals in the Collective, we inhabit a range of educational spaces, roles and contexts, mirroring the inclusive advocacy approach to evaluation that we actively promote. Some of us have knowledge, experience, and expertise as higher education evaluators. However, we believe that anyone can evaluate given the right guidance.
We ask those with formal evaluation training and experience, including ourselves, to support those who don’t so that we can achieve our common goals. We highly value practitioner-led evaluation and advocate for spaces of critical self-reflection as integral to an understanding of impact. We also understand that evaluation work, as a highly flexible and adaptable thinking process, can be hidden within job roles and is an increasing expectation. If this speaks to you, we invite you to join us, and learn with us.
Evaluation can be demystified by rejecting labels and gatekeepers
The Evaluation Collective views evaluation as a safe space for critical reflection and learning, supported by reliable evidence, and modelled by both language and actions. We welcome collaboration and mutual respect in spaces where evaluation takes place.
To this end, we reject divisive and excluding labels of “evaluation nerdiness” which reinforce the positions of nominated gatekeepers and can stifle engagement. Instead, we endeavour to make evaluation language, approaches, and methods accessible and familiar. We seek to demystify and democratise evaluation and we will work hard to reduce any barriers to empowering evidence use.
Building confidence and capacity is key to developing evaluation practices
In order to build confidence with evaluating our own practices and those of others, we advocate for the development of evaluation literacy within and beyond higher education institutions. Evaluation literacy ensures that colleagues delivering activities are aware of and understand some of the key tenets of evaluation and can understand how this relates to and supports their work.
This is sometimes referred to as an “evaluative mindset” and has critical thinking and challenge at the foundation. When used effectively it can empower practitioners and bring valuable tacit knowledge into our professional conversations to the benefit of students and other stakeholders. A particular gap has been identified in the support for evaluators and practitioners attempting to build evaluation capacity and evaluate interventions beyond access, into participation/student success.
Evaluations should be transparently grounded in criticality and context
The Evaluation Collective acknowledges that evaluation takes place in diverse spaces, sometimes dichotomised as self-evaluation and independent evaluation. We firmly believe that these approaches can co-exist and can complement each other, but that this must be in a non-hierarchical and therefore non-judgemental manner. A clear rationale for any approach should be transparent and resourced and aligned to robustness and quality.
Independent evaluation can provide a short term exploration of an issue by adding to gaps in knowledge, experience and resources. These evaluations can create distance between the activity and the evaluators which can be viewed positively or negatively. A lens of proportionality should be applied to independent evaluations, for example, reserved for activities or programmes of high spend or substantial size and scope. We challenge an understanding of “independent” evaluation that solely equates to “external” (outside of the institution) evaluation. Instead, we promote independent evaluation as a professional value, aligning with criticality, additionality, timeliness and materiality.
Self-evaluation encourages critical reflection, is embedded in context over the longer term, and is owned by those responsible for enhancing student outcomes. We believe that active learning within self evaluation is an essential component of effective change. A stakeholder advisory group or nominated critical friend can bring additional rigour to an evaluation design, conclusions and learning.
All types of evidence used in evaluation can have value
The Evaluation Collective believes that all types of evidence used in evaluation can have value. We recommend an exploration of desired impact claims with key stakeholders at the outset of implementing an activity so that appropriate evaluation measures can be aligned and then explored. Relevant standards of evidence should be discussed, applied and critiqued.
Hierarchical assumptions about evaluation methods should be disrupted
The Evaluation Collective believes that a range of methods can be proportionately employed to evaluate within higher education. We advise against method-driven evaluations in favour of theory-driven methods and recommend the use of a co-created Theory of Change to frame and guide all access and participation work.
We will continue to engage in dialogue surrounding the use of experimental and quasi-experimental methods and will draw on them when appropriate; we recognise these can have their place in particular and clearly bounded circumstances. In recognition of the complexity of evaluating social phenomena, we will continually seek clarity about the appropriateness of use.
We actively seek to disrupt the dominance of these methods based on their inherent and often overlooked faults when applied in an educational context. Alternatively, we will champion methods which are practitioner-led, empower those involved, are grounded in a participatory ethos, and foreground ethical practices so that outcomes are not imposed but co-created.
An inclusive approach to evaluation funding should be adopted
We believe evaluation funding should be available for all and not only reserved for those with formal evaluation training and experience. Associated funding for capacity building, when needed, should be ring fenced in project planning.
Higher education requires a cultural shift to ensure evaluations lead to learning
Finally, the Evaluation Collective believe that higher education institutions should be learning organisations which promote thinking cultures and enact meaningful change. An expansive understanding of evaluation such as ours creates a space where this learning culture can flourish.
There is a need to move the sector beyond simply seeking and receiving reported impact. We suggest that colleagues develop inclusive internal communities of practice to support colleagues, and thus institutions, on their evaluation journeys. We, the Evaluation Collective, will then provide the space to connect success and lessons learnt across the sector.
The founding members of the Evaluation Collective are listed here. Affiliation with the Evaluation Collective does not assume institutional endorsement: Liz Austen, Sheffield Hallam University, Rachel Spacey, University of Lincoln, Naomi Clements, Southern Universities Network, University of Southampton, Lucy Clague, Sheffield Hallam University, Julian Crockford, Villiers Park Educational Trust, Billy Wong, University of Reading, Emily Parkin, Northumbria University, Sam Child, Oxford Brookes University, Pickering, Nathaniel, Sheffield Hallam University, Samuel Elkington, Teesside University, Rebecca Sanderson, University of Lincoln, Tünde Varga-Atkins, University of Liverpool, Greg Brown, Alan Donnelly Sheffield Hallam University, Mark O’Brien University of Liverpool, Stella Jones-Devitt, Staffordshire University. To support our manifesto, and join the mailing list please use this Google Form.
At Access all areas – getting in and getting on we’ll assess the current access and participation landscape and consider what will need to change in terms of outreach, information, advice, and guidance, partnerships and pathways between providers, and on-course student support to sustain and grow education opportunity in the years ahead. On Tuesday 10 May at the Mermaid in London: register now.
