Measuring Program Outcomes: Too Often we Use the Wrong Measures


Measuring Program Outcomes

One of the biggest challenges facing the program evaluator is finding a good outcome measure. This is the bread and butter of program evaluation. To demonstrate that a program is beneficial one must measure outcomes, that is, capture the change that takes place with participants. Whatever changes that take place they must be measured and quantified. Measurement is so critical to what we do that Nunnaly (1978) simply stated, “the central issue in social science is measurement.” And while there are lots of measures or scales or instruments in existence a lot of them are not good outcome measures. Why not? Mostly because program evaluators use instruments that were designed to measure concepts but not designed to measure outcomes. At LeCroy & Milligan Associates we have done a lot of work in home visitation and in this field professionals have identified over 100 measures being used! But are they all equally good measures of outcome? Are some programs determined to be effective and others not because of a difference in the outcome measures not in the programs?

One example is the Parenting Stress Index, which sounds like it would be a good outcome measure for a home visitation program. But, this instrument was designed to measure parenting stress—as a concept and as such some of the items in the measure do not allow for capturing any changes over time. For example, “my phone number is unlisted” this questionnaire item will not change from pretest to posttest. Or consider an item on the Child Abuse Potential Inventory—which sounds like a terrific measure for a child abuse prevention program, “I enjoy having pets.” The program theory does not have anything to do with pets!

Many therapists and social workers will readily tell you “change is hard.” As program evaluators we must do our part to find outcome measures that are not only valid and reliable but also sensitive to change—an important criterion often overlooked when selecting measures. The following table suggests some considerations for guidance when selecting an outcome measure.

Key guidelines for Selecting an Outcome Measure

LeCroy, C.W. (in preparation). Handbook of Free outcome measures for Program Evaluation.


About the Author:
Craig LeCroy, Ph.D., is Co-founder of the evaluation firm of LeCroy & Milligan Associates, Inc. Dr. LeCroy is also Professor of Social Work, Arizona State University, in the College of Public Programs. Dr. LeCroy has an extensive background in program evaluation and has been conducting program evaluations since 1983 and worked extensively with state and federal agencies conducting program evaluations. He is a past president of the Arizona Evaluation Network, a state affiliate of the American Evaluation Association.

http://www.craiglecroy.com

Categories

Measuring Program Outcomes: Too Often we Use the Wrong Measures


Measuring Program Outcomes

One of the biggest challenges facing the program evaluator is finding a good outcome measure. This is the bread and butter of program evaluation. To demonstrate that a program is beneficial one must measure outcomes, that is, capture the change that takes place with participants. Whatever changes that take place they must be measured and quantified. Measurement is so critical to what we do that Nunnaly (1978) simply stated, “the central issue in social science is measurement.” And while there are lots of measures or scales or instruments in existence a lot of them are not good outcome measures. Why not? Mostly because program evaluators use instruments that were designed to measure concepts but not designed to measure outcomes. At LeCroy & Milligan Associates we have done a lot of work in home visitation and in this field professionals have identified over 100 measures being used! But are they all equally good measures of outcome? Are some programs determined to be effective and others not because of a difference in the outcome measures not in the programs?

One example is the Parenting Stress Index, which sounds like it would be a good outcome measure for a home visitation program. But, this instrument was designed to measure parenting stress—as a concept and as such some of the items in the measure do not allow for capturing any changes over time. For example, “my phone number is unlisted” this questionnaire item will not change from pretest to posttest. Or consider an item on the Child Abuse Potential Inventory—which sounds like a terrific measure for a child abuse prevention program, “I enjoy having pets.” The program theory does not have anything to do with pets!

Many therapists and social workers will readily tell you “change is hard.” As program evaluators we must do our part to find outcome measures that are not only valid and reliable but also sensitive to change—an important criterion often overlooked when selecting measures. The following table suggests some considerations for guidance when selecting an outcome measure.

Key guidelines for Selecting an Outcome Measure

LeCroy, C.W. (in preparation). Handbook of Free outcome measures for Program Evaluation.


About the Author:
Craig LeCroy, Ph.D., is Co-founder of the evaluation firm of LeCroy & Milligan Associates, Inc. Dr. LeCroy is also Professor of Social Work, Arizona State University, in the College of Public Programs. Dr. LeCroy has an extensive background in program evaluation and has been conducting program evaluations since 1983 and worked extensively with state and federal agencies conducting program evaluations. He is a past president of the Arizona Evaluation Network, a state affiliate of the American Evaluation Association.

http://www.craiglecroy.com

Categories