Theory of Change - Outcomes, Assumptions and Causality

Theory of Change: Outcomes, Assumptions and Causality

In this blog I aim to 'think in ink': the hope being that by writing (and reading) every week, solutions will emerge to strategic problems in my day-to-day work. 

This post is the second in a series on Theory of Change (ToC). In the previous post, I wrote about the need for a Long Term Impact Goal, and how we might place this above an 'accountability ceiling'. Here, I will write about how we've worked backwards to the Outcome level below, which consists of the measurable outcomes we set internally. In the various examples available in the literature, these outcomes can be several, or perhaps even just one, with sub-outcomes.

In our ToC, there are four pillars.

The 'Outcomes' level of a ToC should be constructed such that the causal links to the Long Term Goal are water-tight. We need to back up those arrows with evidence that the assumptions of causality are plausible. It is the job of Trustees and Governors to probe these assumptions as critical friends. 

Let's take the four Outcomes I have placed in the ToC above and probe a little:

1.1 Developing Teachers 

Is it plausible to assume that improving teachers through training will have a positive effect on aspiration and attainment? I think so, especially if done in a targetted way and where we have particular expertise to add to the local offer. As Dylan Wiliam summarises in the superb Leadership for Teacher Learning   , 

"...over the last forty years, evidence has accumulated that the quality of teachers is one of the most important influences on how much children learn at school...The major contribution to improving teacher quality must come from improving the quality of teachers already working in our schools."

In my view, the causal link between teacher quality and student attainment is well established and plausible. We aim to do this through upskilling non-specialists, boosting trainee numbers in STEM through our SCITT and providing quality CPD offers for local teachers.

1.2 Outcomes in STEM

Is it plausible that improving outcomes in STEM will have a long term impact on student attainment? Again, I think this is uncontroversial, but the devil is in the detail. If we can provide qualifications that wouldn't otherwise be available to students in their school then the counter-factual is clear . We are doing this with Further Maths GCSE webinars, by upskilling Physics non-specialists while providing content and practical days, and putting on an extra STEP session locally to enhance the AMSP offer. The other projects around CREST award and Digital Schoolhouse have their own in-built structure and impact measurements. More on this in the next post.

1.3 Literacy at Key Stages 1 and 2

"I don’t remember a single occasion of anyone ever reading to us, and we certainly had no books in the house." Hashi Mohamed, People Like Us, 2020.

The words of author Hashi Mohamed speak to a growing problem among disadvantaged children: they are not being read to as often as their advantaged peers.

In their updated guidance report of November 2021, ‘Improving Literacy in Key Stage 2′, the EEF state that, ‘too many children, particularly those from disadvantaged homes, fall behind in literacy. Disadvantaged pupils are 20% less likely than their peers to reach the expected standards in reading, writing and maths by the end of primary school, and the gap in literacy attainment grows substantially during Key Stage 2, with pupils making less progress than their peers in both reading and writing’.

Speaking to Headteachers from our partner Primary schools, we have been told that it is often the students who need the reading support the most who are unable to get it. This could be for a variety of reasons, not least that adults at home may also be learning English as an additional language, or unable to devote time to reading 1-1 due to job commitments and the pressure that a post-pandemic cost of living increase brings to bear.

“There can be few things as powerful as regularly reading to a young child” [National Literacy Trust]

This is where our Voluntary Service 1-1 Reading scheme can be so powerful, reaching 3000 young children last year.

1.4 Cultural Capital

Can we be sure that building cultural capital has a long term impact on attainment and aspiration? This is a tricky one to track, but is something teachers on the ground actively ask for - this demand is reassuring, and is part of the 'partnerships' approach to joint planning. As the recent ‘Destination University’ publication shows, critical thinking and non-transactional teaching (to the test) can make a real difference to students' aspirations and awarenes of future pathways: 'you can't be what you can't see'.

Next Post: how will we know we've been successful? Measuring outcomes via Key Performance Indicators (KPIs).

Books

Hashi Mohamed, People Like Us, 2020.

We had the privilege of hearing from Hashi Mohamed as a visiting speaker last year, as part of the 'RGS Talks' series.

Links

The Better Evaluation website is a treasure trove for Theory of Change resources and Evaluation ideas in general:  https://www.betterevaluation.org/en/evaluation-options/TOC_software

The NCVO step-by-step explainer for ToC is very clear and helpful: How to build a theory of change | NCVO

Listening to:

Eureka! Podcast - Is Going to School a Waste of Time? For a light hearted discussion around some of the big issues in education.