Research Impact in Hong Kong and a Closer Look at the RAE 2020 Results

On Monday this week, the Hong Kong University Grants Committee (UGC) announced the results of its recent assessment, the Research Assessment Exercise (RAE) 2020.

Unlike previous editions, this round of the RAE introduced research impact as part of its assessment criteria for the first time. Impact now constitutes 15% of the overall weighting. The remaining weightings are predominately made up by research outputs (70%) and the environment (15%).

So, how does Hong Kong define research impact? According to the RAE 2020 Guidelines, impact can be understood as:

. . . the demonstrable contribution, beneficial effects, valuable changes or advantages that research qualitatively brings to the economy, society, culture, public policy or services, health, the environment or quality of life whether locally, regionally or internationally, and that are beyond academia.

Measuring this requires faculties to look at the benefits of their work, revisit external contacts and seek suitable evidence to understand how their research is effecting positive changes for the economy, environment and society.

Assessing the impact of research is nothing new in other parts of the world. The United Kingdom’s Research Excellence Framework (REF) and Australia’s Engagement and Impact Assessment (EIA) introduced an impact component in 2014 and 2018 respectively. UK universities have recently completed their 2021 submissions.

And in North America, while there are no national assessments, the research impact community is as vibrant as ever. In Canada, universities and faculties are coming together to encourage more collaboration and connect their work to various aspects of the economy, environment and society. A network called Research Impact Canada was founded in 2006 to develop ‘institutional capacities to support creating and assessing impacts of research, scholarship and creative activities by developing and sharing best practices, services and tools’.

In the United States, the National Council of University Research Administration (NCURA) is providing professional development programs and sharing of knowledge and experiences of impact to research administrators; and various grants are requiring researchers to provide use of research evidence and the details, beneficiaries and nature of the impact.

This increased focus is both exciting and frightening to many academics around the world. Many question its importance. An article published in The Conversation concisely notes that:

A focus on the benefits that accrue from university research provides an opportunity for universities to enhance public trust and confidence in the value of their research. An engaged and supportive public may just be the most effective pathway towards creating the political will to adopt coherent, evidence-based policy. . . For researchers, greater impact contributes to a virtuous research life cycle, including more sustainable funding.

Interpreting the RAE 2020 Impact Results

In the RAE, research impact is assessed in terms of reach and significance. ‘Reach’ refers to ‘the extent and/or breadth of beneficiaries of the impact’; and ‘significance’ refers to ‘the degree to which the impact has enabled, enriched, influenced, informed or changed the products, services, performance, practices, policies or understanding of commerce, industry or other organisations, governments, communities or individuals’ (UGC, 2019).

Submissions are ranked in terms of the following categories:

At this moment, the RAE 2020 results only show the percentage of each rating each university received for each subject panel for which it has submitted case studies. We have tabulated all the scoring in the diagram below to highlight the overall percentage of each rating the 8 universities received.

Based on these criteria, the University of Hong Kong is ranked 1st on research impact with nearly 50% of all impact case studies submitted receiving a rating of 4* and 44% receiving a rating of 3*.

Ranked in 2nd is the Hong Kong University of Science & Technology with 43% receiving a rating of 4* and 46% receiving a rating of 3*.

In 3rd place is the Hong Kong Polytechnic University with nearly one third of its impact case studies receiving a rating of 4* and more than half received a rating of 3*. All 3 universities also have no impact case studies receiving a rating below 2*.

At this moment, it is difficult to provide a detailed analysis of the performance of the universities, other than observing that the majority of the submissions were successful in transferring their research to policies and solutions that bring substantial benefits to the economy, environment and society and making tangible changes to stakeholders’ lives both locally and globally. Comparing impact scores between subject panels is challenging as the criteria can vary significantly. In addition, some of the subject panels have only one university submitting impact case studies (i.e. UoA 4 – clinical dentistry and UoA 23 – hotel management & tourism).

To have a better insights on the universities’ impact of research, we need the UGC to make impact case studies available, just like the REF in United Kingdom and EIA in Australia to allow both the sector and the society to have a better understanding on the positive changes and valuing benefits of the research as well as, the level of engagement with stakeholders (or beneficiaries) that were established and the use of evidence. This is important for both universities and faculties to learn from the experience, to have a more holistic and integrated picture of engagement with community, businesses and government, and to start embedding impact into the research lifecycle, if this has not already been done.

Moving forward with Research Impact in Hong Kong

Both the universities and faculties now need to plan ahead for the next round of the RAE, which is scheduled to take place in either 5 or 7 years’ time. Universities need to review their strategy and plans for achieving and enabling impact from their research. A successful strategy might be considered as follows:

Successful approaches must incorporate culture and behaviour change as a system-only approach will be minimally effective. It is imperative that organisations draw on the expertise of information managers (for system architecture and data processing), impact specialists (for expert knowledge of the pathways and metrics), central administrative and business support (for institutional reporting requirements), staff whose roles align with impact (e.g. knowledge transfer, marketing, learning and development) and academics (as super users). The ultimate solution must also break down academic-administrative/business divides alongside cementing partnerships with external stakeholders to maximise impact opportunities. . . .[T]he impact agenda requires solid solutions as it grows in both assessment weighting and international visibility. A combined approach of technical, person-based and impact specialism offers the most promising approach to realising impact at the highest level.

And for faculties, identifying impact early on is very important in order to effectively embed it into the project lifecycle. Retrospective gathering of impact data and evidence is very hard.

We have experienced this in the United Kingdom on a large scale with the last REF 2014 Exercise (and current REF 2021 Exercise) and in Hong Kong as well with RAE 2020. By considering impact early on, you can plan for activities in advance, thinking about practical elements like funding, time and other resources, as well as the evaluation activities you want to embed in order to capture evidence of impact.