Proof-of-concept evaluations: Building evidence for effective scale-ups

I delivered a talk at 3ie’s Delhi Seminar Series on a recently published PLoS ONE paper) and follow-up research. This project was a randomised experiment evaluating the potential for text messages to remind malaria patients to complete their treatment course of antimalarial medication.

Myths about microcredit and meta-analysis

It is widely claimed that microcredit lifts people out of poverty and empowers women. But evidence to support such claims is often anecdotal. A typical micro-finance organisation website paints a picture of very positive impact through stories: “Small loans enable them (women) to transform their lives, their children’s futures and their communities. 

Demand creation for voluntary medical male circumcision: how can we influence emotional choices?

This year in anticipation of World AIDS Day, UNAIDS is focusing more attention on reducing new infections as opposed to treatment expansion. As explained by Center for Global Development’s Mead Over in his blog post, reducing new infections is crucial for easing the strain on government budgets for treatment as well as for eventually reaching “the AIDS transition” when the total number of people living with HIV begins to decline.

How big is big? The need for sector knowledge in judging effect sizes and performing power calculations

A recent Innovations for Poverty Action (IPA) newsletter reported new study findings from Ghana on using SMS reminders to ensure people complete their course of anti-malaria pills. The researchers concluded that the intervention worked. More research is needed to tailor the messages to be even more effective.

Calculating success: the role of policymakers in setting the minimum detectable effect

When you think about how sample sizes are decided for an impact evaluation, the mental image is that of a lonely researcher laboring away on a computer, making calculations on STATA or Excel. This scenario is not too far removed from reality. But this reality is problematic. Researchers should actually be talking to government officials or implementers from NGOs while making their calculations

“Well, that didn’t work. Let’s do it again.”

Suppose you toss a coin and it comes up heads. Do you conclude that it is a double-headed coin? No, you don’t. Suppose it comes up heads twice, and then a third time. Do you now conclude the coin is double-headed? Again, no you don’t. There is a one in eight chance (12.5 per cent) that a coin will come up heads three times in a row. So, though it is not that likely, it can and does happen.

Requiring fuel gauges: A pitch for justifying impact evaluation sample size assumptions

We expect researchers to defend their assumptions when they write papers or present at seminars. Well, we expect them to defend most of their assumptions. However, the assumptions behind their sample size, determined by their power calculations, are rarely discussed. Sample sizes and power calculations matter. Power calculations determine sample size requirements, which match budget constraints with minimum sample size requirements.

Unexpected evidence on impact evaluations of anti-poverty programmes

The first email that caught my eye this morning as I opened my inbox was Markus Goldstein’s most recent World Bank blog post, “Do impact evaluations tell us anything about reducing poverty?” Having worked in this field for four years, I too have been thinking that we were in the business of fighting poverty, and like him, I expected that impact evaluations, especially impact evaluations of anti-poverty programmes, would tell us whether we are reaching the poor and helping

How 3ie is tackling the challenges of producing high-quality policy-relevant systematic reviews in international development

At its annual colloquium being held in Hyderabad, India, the Cochrane Collaboration is focusing on evidence on the global burden of disease of mostly treatable illnesses that are concentrated among populations living in low- and middle-income countries (L&MICs).  We already have a lot of systematic review evidence about what works to prevent and treat them.  Yet they remain prevalent due to the lack of resources, implementation capacity and population attitudes.

Making impact evidence matter for people’s welfare

The plenary session at the Making Impact Evaluation Matter conference in Manila made clear that impact evidence – in the form of single evaluations and syntheses of rigorous evidence – do indeed matter. Two key themes were (1) strong evidence about the causal effects of programmes and policies matter to making decisions that improve the welfare of people living in low- and middle-income countries and (2) that, to make impact evaluation matter more, we need to continue to make efforts to build capacity to generate, understand, and use such evidence in those same countries.