A recent Innovations for Poverty Action (IPA) newsletter reported new study findings from Ghana on using SMS reminders to ensure people complete their course of anti-malaria pills. The researchers concluded that the intervention worked. More research is needed to tailor the messages to be even more effective.

The proportion of those completing the pack of pills was 62 per cent in the control group and 66 per cent in the treatment group.  My first reaction was a rather different one to that of the researchers. I thought, Well, that didn’t work very well.  They had better look for something else. The researchers seem to be falling into the all too common trap of mistaking statistical significance for practical significance.

But then I remembered an interview I heard in the wake of the Lance Armstrong drug doping scandal.  “Without the drugs”, a fellow cyclist said, “Armstrong would have been in the top ten rather than first.”  But in many races just a second or two separates the person coming in first and the one in tenth place.

How much difference do performance enhancing drugs really make in athletics? This rather nice blog lays out lots of data to answer this question. Performance enhancing drugs clearly do work. The improvement for the 1500 metres is 7-10 seconds.  Given the world record of 3 minutes 26 seconds, that may not sound like a lot. It is just three to five per cent.  But it took thirty years for the world record to improve by 10 seconds from 3.36 to 3.26.  The women’s world record for the 400 metres was set, mostly likely with the assistance of performance-enhancing drugs, back in the 1980s. No one has come close since.

As a runner myself, I know the big difference between running 10 kilometres in 39 minutes 50 seconds compared to 40 minutes 20 seconds.  Or a marathon (42.2 kilometres) in 2 hours 59 minutes 40 seconds rather than 3 hours 1 minute 10 seconds. It is very frustrating speaking to non-runners who say, “Well, that’s not much of a difference.”

So, going back to the IPA study in Ghana, who am I to say if the increase from 61 to 65 per cent is big or not? I simply don’t have enough sector knowledge to make that judgment. I don’t know what else has been tried and how well it worked. And in particular, I don’t know which of the approaches to get people to complete their malaria treatment is most cost effective. As my 3ie colleague Shagun argued in her recent blog, it takes specialist sector knowledge to know how big an effect needs to be for it to matter, and knowing this is crucial in performing accurate power calculations. Impact evaluation study teams often don’t have that knowledge.  They should be consulting sector policymakers to find out how big is big enough.