17:14:58 Evidence rich impact:1. Design projects for impact2. Take ownership of project evaluation3. Use accurate evidence from your work to understand the difference you have made and to learn how you might improve4. Share learning to help others improve their work You can read more about all this in the Cultural Learning Evidence Champion’s Handbook
17:16:03 This talk is based on the Evidence Champions Handbook.
17:18:59 When we ask that question, “what difference do I make?”, we’re asking about efficacy – does what I do work to produce a desired result?We’re currently surrounded by questions about what does or doesn’t workSo what should I be doing if I want to know if what I’m doing is working and how much difference is it making?
17:20:32 design projects for impact1.	How will the activity respond to the needs of the students in this school? 2.	What difference is the activity expected to make for students and/or teachers (‘outcomes’)? 3.	What makes us think that the activity will lead to these outcomes (i.e. existing evidence)? 4.	What will students miss out on by taking part in the project? 5.	What will the school and the cultural learning organisation put in place to make sure the project can work as intended?
17:22:00 People work with young people in order to effect change in something – like wellbeing. But getting them to explain what happens between ‘working with young people’ and ‘improved wellbeing’ might as well be magic. A Theory of Change unpicks what’s in that black box which is most certainly not magic.
17:22:43 How do I, plus the activity, plus external factors lead to the change I want to see.
17:25:37 In the Evidence champions guide there are examples of Theories of Change
17:27:16 Activity - what you do and for whom, Mechanism - what is needed for your approach to suceed, Outcomes - the values, attitudes, knowledge, skills and behaviours that result directly from your work, Aim - what drives you to do what you do.
17:29:12 Theory of Change is just the first step. 1: theory of change and evaluation plan - we know what we want to achieve2: before and after study – we have seen there is a change
17:30:28 3: counterfactual of some kind - we belive the change is caused by us4: manualisation and replication in a different places - we know how and why it works5: replicated in multiple cities - we know how and why it works - it works everywhere
17:32:25 Take ownership of project evaluation1.What do we want to learn from the evaluation? Can we align these objectives with the school’s development plans?
17:32:34 2.Why do we need to collect participant data? What data will be collected, how and by whom? How will it be analysed and by whom? What existing data can be used?
17:32:40 3.How will data collection costs be met?
17:32:44 4.How and when will requirements and timing of any data collection be communicated?
17:32:49 5.How can participants opt-out if they want to?
17:35:52 Collecting data that can tell me about the difference my work makes
17:36:09 •Collect both qualitative data (stories) and quantitative data (numbers).
17:36:34 •To understand the difference you are making, you will need different kinds of data about:1.	Who is taking part2.	How they engaged with your activity3.	What they thought about it4.	Whether it has made any difference
17:37:01 Consider when you collect data as well as how you collect data
17:37:54 Make sue you collect data at both the beginning and the end of the project.
17:38:19 Compare your participants with people who aren't participating your project.
17:39:19 Collect data from users.  This is information about who is taking part in your activities.
17:39:47 What can it tell me?  Engagement data can tell you how well you are engaging participants over time.
17:40:35 Engagement data can also tell you: if you've delivered what you have promised, how regularly participants attend or return to your activities, what kind of activity they engage in most often.
17:42:07 What should I do?  Feedback data and outcomes data are often conflated, but relying on feedback to demonstrate that you have achieved an outcome is not particularly reliable.
17:43:23 Getting feedback from a wide range of participants, staff and stakeholders should be a reguar part of routine.  You can collect it using surveys, questionnaires etc.  Mentions on social media, conversations, managements meetings, board meetings.
17:43:41 OUtcomes data will tell you if what you have done made a difference to participants.
17:44:20 Collecting outcomes data is likely only to happen occasionally.
17:45:00 Collecting it can be done through a wide range of approaches, which will give you different levels of accuracy and reliability.
17:45:41 Reporting the results of your evaluation:
17:46:15 1. How will you share the results of the evaluation, and what tone do you want to strike?
17:47:01 2. Commit to sharing findings, even if the expected outcomes have not been achieved.
17:47:11 3. Agree to report on what you hoped to achieve; what you did; what you monitored; any impact observed and what you learnt.