Could AI revolutionise Impact Evaluation?

Last week, I was delighted to present to peers working on educational partnership projects at Christina Astin's online Breakfast Briefing. The theme was Impact and Evaluation, and as with the best CPD sessions, I took away several unexpected threads to unravel afterwards. Not least Most Significant Change Theory (MSC), thanks to Rob Southwell-Sander. This blog is all about my own upskilling, so I was fascinated to learn something new in my reading this week: it is clear that MSC can harness student voice and bring rigour to focus group feedback and qualitative data capture. I have been particularly inspired by the body of work of Rick Davies at Swansea University around MSC - more on this in a future post. I think this methodology could suit several of our partnership projects and we will aim to start small with a pilot of the approach. 

Barriers and Solutions

The barriers to rigorous evaluation work were explored in the Breakfast Briefing - not least: how do we find the time to do all of this properly?  

This was further brought home to me while taking part in a recent partnership project with Newcastle University - using a Theory of Change methodology, the 'Sustainable House Design' project for Year 9s has benefited in its design and evaluation from the input of graduate researchers, who have given time and expertise to put a robust process in place. The success criteria and quality assurance have been 'baked in' from the start and I have learned a great deal. As highlighted in the research of Rick Davies and others, Theory of Change and Most Significant Change can be take up a huge amount of time and resource if we do such qualitative (and quantitative) data capture properly.

Many Directors of Partnerships and School Improvement Leads have no dedicated admin support around Impact Evaluation. However, this arguably one of the most important aspects of school improvement work - how down we know if we have been successful in our interventions? One solution to the resource barrier could be the fostering of links and partnerships with Universities. Our link with Newcastle University has opened my eyes to what a high quality Theory of Change process looks like on an individual project level: there are certainly opportunities here to expand our links with post-grads and education departments. We are lucky to have two fantastic Universities on our doorstep, but even without this luxury, much of the admin work can be done remotely. An alternative solution to the resource barrier is via technology, and particularly the growth of open AI.

I won't add to the explosion of blog posts on ChatGPT here - there are plenty of other places to find this on the internet. However, I will rather plant a seed by asking the question:

How can AI support Impact Evaluation in Charities and Schools?

In other areas, such as Law and Medicine, AI is increasingly integrated. Any profession reliant on processes that involve significant human effort in trawling through data will likely transform over the next decade. In fact, an AI application was due to represent a defendant in court recently, and as the New Scientist reports, while this approach has since been blocked, it won't be the last time we read of AI appearing in the 'human' realm. Academics are increasingly looking to the power of AI to do the heavy lifting in literature reviews, with some fascinating debates on the ethics of this across social media. Moreover, we seemingly can't scroll through LinkedIn without seeing a version of the quote that, 

'...you won't be replaced by AI, but by a human using AI.' 

In the charity and education space, there is little doubt that AI could go some way to solving the (false) perception that funding shouldn't go on admin. If a donor is making a big contribution, a reasonable question in the future could be around 'how is AI being used effectively in order to reduce admin costs'? Similarly across schools, the big opportunity with AI is around reducing teacher work-load: surely the days of producing a sheet of 10 multiple choice questions for a homework task is over: ChatGPT can do this in a matter of seconds. Similarly, putting together a survey that is appropriate for young readers taking part in a Literacy Intervention programme could be produced to an accessible level in a matter of seconds. Those using AI smartly are likely to make gains and efficiencies around admin that will leave others behind. If we can harness this to improve outcomes for our young people, why wouldn't we experiment with it? 

In my daily work, I already use smart apps to reduce admin, such as Calendly (life-changing), Canva, Trello (dabbling with this) and others. ChatGPT is now another tab open in my browser, and more tabs will surely follow. I'll be using 'video/audio to script' convertor applications for data capture in the coming months. 

While we may still need humans to think, 'can I automate this?', we will increasingly find the answer is 'yes', leaving more time for the good stuff - working with our young people.