1. Resources and Strategies for Publishing Evaluation Data
“Another benefit of collecting the evaluation data etc is publishing on it, are there any resources or thoughts for how to go about this? Or the use of this?”
Again it is important to think about what you hope to achieve from sharing your evaluation findings. Are you wanting to share new ideas/ thinking to inform others in their work? Do you want to have your evaluation work assessed by others? Do you want to contribute to the academic knowledge base about public engagement practice? Are you seeking recognition for your work?
Depending on your purposes, you will also need to think about:
- How you choose to share your work – from writing blogs, creating short film, to sharing project reports, or journal articles and
- Where you choose to share your work – from your own project website; institutional website; funders websites; organisations that engage the people you how we will engage with your work e.g. NCCPE Research for All journal
For evaluation, you will need to consider GDPR and ethics, including if and how people have assented to the use of their data. So it is really important to think about your plans for publication right at the beginning so you can ensure you have addressed these issues carefully and proactively.
2. Public Engagement for Young Academics
“For young academics who just started their academic careers, their focuses are publishing and obtaining research funding. How would they start engaging the public/practitioners?”
Start small and build from there. It helps if engagement follows your interests and passions. Don’t assume all engagement is event based, and consider what you have the time and the resources to try.
Start by participating in an activity that is run by someone else (e.g. I’m a scientist) or a local science festival. These formats are already tried and tested, and you can get support to be involved. Having tried something, take time to reflect on it, and consider if and how you want to do more. This might be doing more supported events, or developing an approach bespoke to your research area. Your centre or institution is likely to have a Public Engagement team – talk to them! They may well run training events you can take part in too.
There are lots of resources to help get you started. The NCCPE website is a great place to start: https://www.publicengagement.ac.uk/do-engagement
If you are interested in working with schools and young people, I would recommend becoming a STEM Ambassador as you get a DBS check and you are sent a range of opportunities in your area, these could be giving a school assembly, judging a competition, giving careers advice. All great activities to get a feel for PE. Also don’t feel like you have to reinvent the wheel, look for resources and activities that are already out there. For DNA and genomes related activities www.yourgenome.org is a good starting point.
3. Contextualising Impact Contribution in Complex Areas
“Any advice or examples of how to talk about or contextualise impact contribution in complex areas like policy? Often, researchers have been cited and been involved in discussions and been consulted but the link to eg. Gov reports etc is not direct as there are no mentions. So for REF impact levels of evidence this can be challenging.”
There are two essentials for effective REF case studies. A compelling narrative, that offers links in the impact journey, that provide confidence that the research has made a contribution to the impact claimed and effective evidence of this impact. For impacting policy, this might include narrative about the involvement of the researcher within relevant dialogues and discussions; testimonials from policy makers and/ or civil servants evidencing the value of the contribution; specific references to papers, or to grey literature relating to the research including research briefings; testimonials from partner organisations etc.
Given the REF assessment is a peer review process, you are seeking enough evidence to give confidence to the panel (which will include researchers in your discipline) that the impact you are claiming is one that you can legitimately say you made a contribution to.
4. Evaluating Engagement Intervention vs Longer-Term Impacts of The Intervention
“Very helpful to hear both Francesca and Tony distinguish on the difference between evaluating the engagement intervention, vs. the longer term impacts/outcomes of the intervention. Do they have any useful hints/tips on this point?”
The NCCPE’s guide to evaluating public engagement considers the evaluation of the process, enabling you to make improvements to it; evaluation of the outcomes at the point of intervention; and consideration of how you might evaluate longer term outcomes and impacts. It uses a theory of change approach, modeled using a logic model. Doing a longitudinal study with a small pilot project is a great way of targeting evaluation effort, and exploring if and how the intervention has led to longer term change. This may enable you to come up with some proxy indicators of impact, that you can use for the overall project.
I would say if you are planning anything do a Logic Model or Theory of Change as this is really helpful for identifying what questions you want to answer in the short, medium and long term (and therefore the type of data and information you need to collect). Evaluating a specific activity is still important but if you need to report on the bigger picture of what impact that activity had you need to think longer term.
5. Considering Expectations of Evaluation Studies
“I worry that the average sociologist or psychologist would despair at the conclusions we draw from our evaluation studies. Do we need to be more realistic about our expectations? Our studies are usually poorly controlled and rely on accurate self-reporting by the respondent. We are so often expected to collect data that in fact is just too difficult to achieve with the resources available (eg re learning outcomes). Do we need to change expectations?”
Evaluation needs to be rigorous, and methodologically sound. Whilst we need to recognise that the evaluation we do of projects is not a fully costed research project, and manage expectations about that, we can ensure that we use robust approach to evaluation by either learning about it ourselves; working alongside others in the university who have these skills eg public engagement staff; or commissioning an evaluator to do the work for us.
Currently there is a significant amount of resource that is spent on poor evaluation, that has little value for learning or for measuring impact, and we need to ensure that we think strategically about the evaluation we use; we do this effectively; and we report on it appropriately.
6. Evaluating Outreach in terms of Diversity
“I work for a public-sector research institute that is trying to get a more diverse workforce (we are physics/engineering and so match the typical levels of high white and cis-male identifying individuals). Part of our outreach is focused on getting women and minority ethnic backgrounds; So my question is how can we sensitively evaluate how/whether our outreach is effectively getting to these audiences? [We do some events targeted specifically at these groups, but is there a way to evaluate our general events that are open to all with this angle in mind ?]”
There are quite a few interesting elements in this question. The first is to what extent the general events are suitable, relevant and engaging for the diversity of audiences you hope to engage with. For me this is the starting point.
The second is about reach. You are seeking to evaluate if you are attracting specific groups to the event. Standard monitoring information invited at the time of booking the event could help you do this, but am not sure what you would learn from this? Attendance at the event may well be down to marketing, and if and how the event suits the interests, needs and availabilities of those you are seeking to engage.
The third is about good practice in engaging with diverse audiences. In this case working alongside specific community organisations who are already working with the groups you hope to engage would help, and you could work alongside them to understand better how to develop effective interventions and also how these could be effectively evaluated.
7. Collaborating with Scientists
“I liked your tracking of scientist behaviour from dipping toes in to doing more and more, any ideas on how to track this when people might be working outside of your institution etc?”
Think about how keeping in contact with the scientists could benefit them, for example could you develop a series of events for people to come back together to reflect on their work; problem solve; and share learning and insights?
Having some system where they can log what they are doing is useful; some organisations use a Google Form to collect data on their staff’s PE activities such as date, audience, activity and more, which goes into a central database. When appraisal time comes around they can generate a report in their activities for the year, or if they want to add this as a portfolio for their CV. There is mutual benefit in this sort of system.
8. Public Engagement in an Online World
“Delivering online events is much more difficult to generate a 2 way conversation. Do the panel have any tips to achieving this online?”
The NCCPE created a guide to developing online events which includes some great guidance, and there is also a guide from Involve which has some great pointers.
Again you need to think about why and who in thinking about the format and approach. This includes the choice of platform for the dialogue. You will want to consider whether you want to develop an dialogue through an event, or whether it could be better supported asynchronously, with people adding their ideas and thoughts at times that are convenient to them.
Remember it can take longer to build confidence to engage in an online forum so offer different ways to participate including through chat; on an online message board; by sharing audio and/ or camera. Consider the implications of chat being anonymous or not, and how this might encourage or inhibit participation.
Using small groups can help, but make sure that you facilitate these well, to enable everyone to get involved. Consider asking everyone to introduce themselves with a picture and a one-sentence story about themselves on a padlet board or similar. Consider speaking to all event participants before the event, or have a session before the event, so people can familiarise themselves with the platform, and meet a few people. Recognise all contributions, including in chat; on padlet; in the room, and share them.
.
JOIN OUR MAILING LIST
and keep up-to-date with everything impact related.