Counting and Counterfactuals: Theory of Change 4
This post is the fourth in a series on Theory of Change. In previous posts, I have written about the need for a Long Term Impact Goal, the Outcome level and the Activities planned to meet the outcomes. In this post I will address three questions around impact posed by Caroline Fiennes (2012):
"...to cut through the complexity we need to know 3 things:
- what happened?
- to what extent was that due to us?
- how do our results compare to other charities?"
1. What Happened?
'not everything that can be counted counts, and not everything that counts can be counted'
In my view, whether we deem a particular piece of data counts or not, we should count it anyway.
As Fiennes notes, we are obliged to document what happened when using charitable funds. It may be that different audiences are interested in different metrics and numbers: for example, a funder of our STEM programme might be interested in a slightly different outcome or 'output' than a school governor. It seems crucial to have all of the data at our finger tips.
The best example that I have seen of this is the work of Graeme Lawrie MBE at ACS International Schools. The numbers are staggering, and an exemplar for cross sector 'outputs'. However, even more interesting to me is the way the stats are embedded in a Theory of Change framework:
"This data represents the impact of ACS International Schools Partnerships over a 5 year period... this data represents the 'Outputs' (The amount of something we produce as a result of our activities). From this data we calculate our 'Outcomes' (which are experienced by our stakeholders) and 'Impact' (The broader change that occurs as a result of our activities)."
2. To what extent was it due to us?
The counterfactual is the scourge of education researchers and reflective senior leaders:
What would have happened if we had done nothing?...
What if we hadn’t implemented the intervention groups In Year 11?...
What if we hadn’t marked any homework below Year 10? ...
What if we dedicated more time to Sport and the Arts and took time from the ‘core’ subjects?...
In other (scientific) disciplines, we have the control group and Randomised Control Trials (RCTs) to deal with the counterfactual. However, implementing RCTs in schools is notoriously tricky. This is for a number of reasons, not least the ethical implications of withholding an intervention from a group of students. That is before we even try to unpick the myriad factors and inputs that we are trying to control for. My reading of Complexity Theory and its use in Social Sciences has helped my understanding of how we might navigate this obstacle in educational settings. As Byrne concludes, the best we can do in a non-simple system is to look for clues from the outputs and work backwards: where students are successful, what do the activities have in common? Where schools are unusually successful when compared to similar schools across various socio-economic demographics, what have they done well?
A classic example of this kind of analysis in teaching is Doug Lemov’s work, a seminal piece of work that builds on data from the most successful teachers across the US and works backwards to naming the common strategies observed and making them explicit. For me, it is the one book every teacher should read.
"...solutions to education challenges exist in the classrooms of real life teachers, that exceptional practitioners of the art of teaching are the true experts. Our job is to find them, study them, and share what they do, so others can copy, practice and adapt it to their own teaching".
3. How do we compare?
An ongoing question within third sector work and for those of us working across lots of different educational settings must be, "what are we bringing of additional value here, and could another charity do this better?" As Fiennes asserts, if someone is approaching our Outcomes and Impact among the same population in a more successful and rigorous way, perhaps we are best served enhancing their work - the power of Theory of Change is in making the need for such partnerships explicit. The long term impact is all important: if we are serious about inspiring and motivating disadvantaged students to achieve their potential, sometimes the most helpful thing is to empower and support those doing it well already. In my own area, we have done just this with fantastic charities and institutions such as Success 4 All, AMSP, Common Room of the North and many others.
It is only through partnerships that we can tackle the complex issues facing the sector at this difficult moment for schools, children and parents.
Happy half term!
Reading
Fiennes, C. (2012) It ain't what you give, it's the way that you give it. [Giving Evidence]
Link
Comments
Post a Comment