On 20th April 2023, the Final Report for the Review of the Australian Research Council Act 2001 was published by the Australian Department of Education.
The report, produced by a team led by Margaret Sheil (vice-chancellor of Queensland University of Technology) set out ten recommendations “which aim to enhance the trust in the ARC by the government and the research community”. These range from advancing support for Indigenous Australian academics through to introducing a legislative basis for the ARC – and importantly recommend for the cyclical ERA (Excellence in Research for Australia) and EIA (Engagement & Impact Assessment) to be discontinued.
Meanwhile, a separate review of Australian higher education in general is currently in progress and will deliver an interim report in June 2023, followed by a final report in December.
What Do the Recommendations Mean for Impact Assessment?
The quality of Australian research has so far been assessed by the ERA, the country’s national assessment, which took place periodically in 2010, 2012, 2015 and 2018. The 2023 iteration was postponed by the ARC back in September 2022, pending review.
The ARC introduced the EIA in 2018 as a counterpart to specifically examine impact and engagement, using the same dataset from ERA submissions. The results from this exercise are available in an online report here.
In the ARC review report, 55 per cent of the respondents agreed that the ERA and EI have served their purpose and should be halted; however the report does consider that the “initiatives have played a valuable and important role in raising the quality and relevance of research in Australian universities”.
Crucially, the authors are “explicitly not recommending ERA and EI be replaced by a so-called light touch metrics-based exercise”, with additional concerns surrounding the rigour of Impact Case Studies (ICS) submitted to the EIA, which “risk privileging style and particular kinds of research outcomes, especially were they to be linked to government funding”.
Furthermore, the authors consider that “both systems are inherently retrospective, when Australia’s research sector and the broader national innovation system are well equipped to pivot to a prospective paradigm”. In the global context of higher education impact assessment, this echoes the school of thought that prioritises a formative assessment – one that continually optimises the quality of activities, such as the SEP (Strategy Evaluation Protocol) in the Netherlands – rather than a summative assessment, such as the UK’s REF (Research Evaluation Framework), which examines impact over time and is used as a reference to allocate funding to institutions.
One suggestion is to include assessment of engagement and impact portfolios to complement quality assessments when universities undergo their seven-yearly re-registration to TEQSA, which would form part of an annual evaluation of approximately 60 units of evaluation (UoEs) – rather than the 2,603 UoEs assessed in one go during the last ERA.
Universities Should Continue to Capture Research Impact
While the recommendations are digested and the exact future of impact assessment in Australia remains unclear, universities should continue to capture and measure their research impact as standard practice.
The report notes that “Engagement and Impact (EI) has a narrow reference window for research to application, relative to the typically longer periods observed in the real-world history of research and innovation”. hivve’s analysis of Hong Kong’s RAE (Research Assessment Exercise) 2020 supports this observation, showing that research takes an average of 7-19 years to create impact.
In fact, the Australian Academy of the Humanities contributes “We think there is merit, in place of ERA, of a ‘State of the Research Sector’ approach. As a retrospective assessment the ERA is not necessarily a good indicator of strengths at the time of assessment or of future performance. The ARC could continue to collect data annually and … produce a ‘State of the Research Sector’ report which would provide evidence of value to the Minister and the sector – for workforce development, capability assessment, and strategic planning. A thorough report would include indicators on gender equity, workforce diversity, and research environments.”
With this in mind, it is likely that any future form of impact assessment in Australia will consider longer time periods, and therefore institutions should be continually capturing quantitative and qualitative data to support their impact claims.
The best way to do this is to set tangible metrics and regularly capture supporting data and evidence, which helps to build an organisational memory for use in communications around how universities support their local communities. This can then feed into annual reports, and even provide a portfolio of examples of the sorts of impacts created by the institution.
Furthermore, capturing impact data in real-time requires a cultural shift, which can be supported through technology and bottom-up/top-down implementation strategies.
Impact Could Become Increasingly Important for Academic Careers
One of the ten recommendations of the report asks for more clarity around the role of the ARC in supporting academic careers.
Given the possibilities of a move to an annual, lighter-touch assessment model, one of the approaches taken by the ARC could reflect the New Zealand PBRF Quality Evaluation (QE), which looks set to include impact within its assessment criteria for 2026. The PBRF considers the evidence portfolios of individual researchers rather than groups in its evaluation process.
Although the ARC recommendations have only just been published, there is the possibility of a more integrated approach to career support. It can therefore be useful to collect impact data and evidence holistically, using a system that can break down information into institutions or even individual levels.
Impact continues to be important in grant applications, including for the NHMRC, where having a readily available record of previous impact is essential; and the growing importance of impact within global assessments means that being able to prove credentials is important to remain competitive.
Furthermore research by Times Higher Education (THE) (published in May 2021) shows that for students, the “university’s commitment to, and reputation for, sustainability” is now more important than the location of the institution, creating further incentives to be able to demonstrate economical, environmental, cultural and societal impact.
Next Steps
The report has been submitted to education minister Jason Clare, with next steps likely to be announced in due course.
However, although the nature of impact assessment in Australia is likely set to change, the importance of impact will continue to stay – especially with regards to tracking sustainability over long periods of time.
As James Cook University contributes to the report, “a showcase to the public of projects that were funded 5, 10, 15 and 20 years ago and the benefits they have delivered to society could increase the support for publicly funded research in Australia.”
Universities should continue to capture relevant impact data and evidence as part of business-as-usual for future proofing and aligning to key trends, weaving impact into the fabric in preparation for any future evaluations, but also for the multitude of different platforms where showing transparent and wide-ranging impact is vital.
If you want to learn more about simplifying how you capture impact over time, please get in touch.