At about the halfway point between two REF submission dates, universities are beginning to look more closely at which impact cases are developing strongly in this REF cycle.​​ ​​

We first took a deep dive on this last spring when Vertigo Ventures, along with Digital Science, were commissioned by HEFCE to review best practice guidance in terms of impact evidence from REF 2014.  That work can be seen in the report ‘Collecting Research Impact Evidence: Best Practice Guidance for the Research Community’ and also in the accompanying webinar.

Researchers and research managers refer to this report to develop methods for tracking and capturing impact, but we decided to take a more qualitative look at a set of outstanding case studies to see how they use impact evidence to build outstanding impact cases.

This blog outlines headlines from our findings but for more details please sign up for the webinar on July 14th.

We looked in detail at cases from each main panel where the impact submission scored 100% 4*, 15 from Clinical Medicine, five from Electrical and Electronic Engineering and two from Civil and Construction engineering (all the 100% 4* cases from Panel B), eight from Education (Panel C) and nine from Communications, Cultural and Media Studies (Panel D).

So what might we be looking for when thinking about how these 4* cases apply good practice to using evidence? There are some instructions in the REF 2014 guidance:

The sources should be external to the submitting institution, they should corroborate specific claims, they should not ‘substitute’ for providing clear evidence of impact in section four, and there is an indicative maximum of 10 references.

In keeping with our findings of the Evidence report from June 2016, use of impact evidence in the cases we looked at in detail, varied widely.

In terms of following the guidance, the 4* cases were consistent in providing independent sources.  All the evidence was from non-academic sources, except where an academic paper was used, for example to outline the effects of treatment in Clinical Medicine, or to outline changes in practice using new concepts, in Education.

There are instances when a report generated within a University can be used as evidence and this approach is occasionally used in the cases we looked at. If there is a great deal of material that is better presented in an aggregated report, for example media appearances or mentions, this material can be combined and submitted as one piece of evidence.

Most of the 4* case studies used effective methods to link the impact evidence to specific claims.  All used referencing to do this, providing a number, letter, or combination of them in relevant phrases or paragraphs, and itemising the evidence against them.

In terms of using evidence to support the claims made, rather than substitute for them, we found no examples where evidence in section five was not supported by narrative in section four.  However there were occasions when an aspect of the impact narrative did not have an auditable evidence item in section five, but in these instances the authors usually provided extensive information and even links, within the body of the case.

Beyond the specific guidance from HEFCE, panel members from REF 2014 have also spoken about good practice as they saw it, in using evidence.  This advice includes:

Make sure the case study is self-contained, as panel members are expected to rely only on material within the four page case study for assessment.  So where the piece of evidence contains material that’s vital for your impact narrative, make sure you quote from it in the text.

Say what each piece of evidence is. Some of the cases did this well with a line or two about each piece of evidence, reinforcing what it was claiming and identifying what the piece of evidence was.  This is most important when providing something like a web link, which should have at least a description, not just a web address.

The guidance provides examples of suitable types of evidence: Reports, reviews, web links, confidential reports or documents, individual users/beneficiaries, or factual statements (testimonials) already provided to the HEI by key users/beneficiaries.

These types of impact evidence were used by cases across all the panels. We didn’t see many uses of social media, except one link to Facebook and one use of tweets (evidenced via a screen grab).  More common were wide ranging references to media campaigns, rather than picking out one specific media activity.

Use of reports and websites however, were very specific, with cases frequently explaining what aspects of impact were being corroborated by a specific report.

The variety in application of the guidance, good practice, reinforces what panel members have said about the importance of building a good narrative across the whole case study.  Stephen Holgate, chair of REF Panel A has been quoted as saying “Narrative is highly influential in generating confidence in the impact”. No one element will build your impact into a 4* case, each will play its part and should complement the whole. However, when looking at identifying and selecting evidence for impact case studies these 4* cases show that being able to access the right report, speech or web page from the many sources available, or building a picture of media activity to corroborate your specific impact, is key to good evidence and generating a strong result at assessment.

For more details and discussion on this topic sign up for the webinar ‘What does 4* impact evidence look like?’ 10am July 14th.