“I need to prove that my program makes a difference!”

How do I prove my program makes a difference?

“I need to prove that my program makes a difference!”
Wow, if I had a dollar for every time I heard a program manager say something like that I would have a lot more dollars. When I first started working in program evaluation I used to get pretty jazzed by that sentiment. After some time went by and I was more “experienced”, I would hear this and I felt less positive and started to get pretty anxious. I got overwhelmed by the not at all carefully considered reasons why this unique, singular program may not have much influence at all on anyone. Of course the program manager wants an evaluation that gets at causation – we want to presume that there is a relationship between two variables. Especially if they are correlated, have temporal priority, and there is a lack of spurious correlation. On the other hand, part of our undeniable experience of living in the world is that we “know” that lots of stuff just seems to happen, poof!

The Realist Evaluation Approach
Programs designed to help people are really, really complex – there are lots of moving parts. I think good program evaluation helps to describe how programs work and how program components and the people in that context are influenced. I like the Realist Evaluation approach originated by Pawson and Tilley (1997), for more information on this approach and a great website and resource for anyone interested in evaluation, see Realist Evaluation. One key idea from this approach is the notion that “interventions work (or not) because actors make particular decisions in response to what is provided by the intervention (or not)”. The people served by the program and the program staff – all are actors whose actions influence the effectiveness of the intervention. Their actions are driven by their response to the resources or opportunities provided to them from the program or intervention. Context also matters as it influences how people act and what kinds of resources programs can offer.

So now when program managers talk to me about wanting to prove how their program “makes a difference”, I am a little less anxious. I know that there are many mechanisms at work amongst many actors and if we can build a relationship of trust with one another, my work as an evaluator may be able to help the program manager find lots of ways that her program and those touched by her program “make a difference”.



About the Author:
Michel Lahti, Ph.D., is the CEO of LeCroy & Milligan Associates. He has an extensive background in program evaluation working in both the public and nonprofit sectors. In addition, he provides consultation to agencies implementing performance measurement systems.

“I need to prove that my program makes a difference!”

How do I prove my program makes a difference?

“I need to prove that my program makes a difference!”
Wow, if I had a dollar for every time I heard a program manager say something like that I would have a lot more dollars. When I first started working in program evaluation I used to get pretty jazzed by that sentiment. After some time went by and I was more “experienced”, I would hear this and I felt less positive and started to get pretty anxious. I got overwhelmed by the not at all carefully considered reasons why this unique, singular program may not have much influence at all on anyone. Of course the program manager wants an evaluation that gets at causation – we want to presume that there is a relationship between two variables. Especially if they are correlated, have temporal priority, and there is a lack of spurious correlation. On the other hand, part of our undeniable experience of living in the world is that we “know” that lots of stuff just seems to happen, poof!

The Realist Evaluation Approach
Programs designed to help people are really, really complex – there are lots of moving parts. I think good program evaluation helps to describe how programs work and how program components and the people in that context are influenced. I like the Realist Evaluation approach originated by Pawson and Tilley (1997), for more information on this approach and a great website and resource for anyone interested in evaluation, see Realist Evaluation. One key idea from this approach is the notion that “interventions work (or not) because actors make particular decisions in response to what is provided by the intervention (or not)”. The people served by the program and the program staff – all are actors whose actions influence the effectiveness of the intervention. Their actions are driven by their response to the resources or opportunities provided to them from the program or intervention. Context also matters as it influences how people act and what kinds of resources programs can offer.

So now when program managers talk to me about wanting to prove how their program “makes a difference”, I am a little less anxious. I know that there are many mechanisms at work amongst many actors and if we can build a relationship of trust with one another, my work as an evaluator may be able to help the program manager find lots of ways that her program and those touched by her program “make a difference”.



About the Author:
Michel Lahti, Ph.D., is the CEO of LeCroy & Milligan Associates. He has an extensive background in program evaluation working in both the public and nonprofit sectors. In addition, he provides consultation to agencies implementing performance measurement systems.