Does monitoring alone lead to change?

This past Christmas my husband got me a Fitbit. He wasn’t being rude; I asked for it. Now I am smart enough to know that just wearing the darn thing alone was not going to automatically lead to lost pounds. No more than just attending my Weight Watchers meetings each week leads to losing weight. And as I predicted, in the last few months I have not wilted to my high school weight. So what’s the deal? Whatever happened to, “What gets measured gets done?”

Screen Shot 2015-05-19 at 1.53.56 PM

You often hear people say that you have to measure the right things. I think it’s more accurate to say that you have to measure the right set of things. I can walk 10,000 steps every day, but I if eat lots of things that are high in calories I will be no better off.

Similarly, if we only track things that happen at point A and point C, ignoring the implementation of the intervention (i.e. point B), we may also miss the boat because we were hyper-focused on outcomes alone. In my example, if I don’t track the calories I put in my body along (my Point B) along with the calories expended, I won’t reach my goal. I once heard someone at a shall-not-be-named very large federal agency say that a surveillance system is all that is needed. That attitude sounds like a cartoon that is familiar to most evaluators.

Screen Shot 2015-05-19 at 1.53.46 PM

What happens in between points A and B is important. E. Jane Davison, in Actionable Evaluation Basics, advises that too often we get lost in the details and have no clear sense of what we really want to know. “Too often, evaluations don’t have any of these big picture questions; they leap to measurement or data gathering. Or they ask incredibly narrow questions, at the indicator level,” says Davidson. Monitoring alone is not necessarily an evaluative endeavors.

In my evaluation world, we deal with complex social problems. Measuring complex social problems is a challenge; measuring the effectiveness of our prevention efforts is even harder. We also have to decide what to measure at each stage of the intervention.

In the past we have been guilty of collecting too much data that inevitably didn’t answer evaluative questions. We have clients who are guilty of measuring only what funders want to know even though what funders want to know doesn’t always reflect the “big picture” questions either.

Choosing what to measure is hard. One thing is for sure, monitoring alone does not lead to change. We have to measure the right things at the right time in the right combination. Walk on friends……

Screen Shot 2015-05-19 at 1.53.20 PM