Most impact evaluations capture short-term (1-2 year) results, but development interventions often aim to shape outcomes over entire lifetimes or even future generations. For example, designers of a program to support childhood health, nutrition, or education might hope that improving immediate outcomes for children will translate into improved labor market outcomes and incomes for those people as adults. There may be strong theoretical reasons to expect these long-term effects, but empirical evidence from long-term impact evaluations 5, 10, or even 20 years later is scarce.
This lack of long-term evidence has obvious implications. How can value-for-money be calculated when a project's value is only measured over a fraction of the time when it might be bearing fruit?
The handful of long-term impact evaluations that have been conducted have often yielded surprising results.
One example is the Targeting the Ultra Poor program, implemented in several countries by BRAC, where a long-term impact evaluation showed a durable positive effect. The program provided households with substantial asset transfer, consumption support, savings, and training. A study on the intervention in West Bengal, India showed that positive effects on consumption, food security, income, and health persisted 10 years after the intervention. A separate study on the same intervention in Bangladesh showed similar positive results on asset accumulation after 11 years.
The authors of the West Bengal study note that the initial costs of the intervention were "considerable," but the long-term results show that they're worthwhile:
"The program breaks even by year four. Its benefit-cost ratio is 379 percent by year 10 and would be 1,110 percent if year 10 gains are sustained in perpetuity. These estimates are larger than in Banerjee et al. (2015), which extrapolated year three gains over time, because treatment effects rise between years three and ten." (Banerjee, Duflo, and Sharma 2021)
"The program breaks even by year four. Its benefit-cost ratio is 379 percent by year 10 and would be 1,110 percent if year 10 gains are sustained in perpetuity. These estimates are larger than in Banerjee et al. (2015), which extrapolated year three gains over time, because treatment effects rise between years three and ten." (Banerjee, Duflo, and Sharma 2021)
However, long-term follow-up studies don't always show such lasting positive effects.
A nine-year follow-up study of Uganda's Youth Opportunities Program, which provided cash grants to help youth become craftspersons, showed mixed results. For that project,
"These start-up grants acted more as a kick-start than a lift out of poverty. Grantees' investment leveled off; controls eventually increased their incomes through business and casual labor; and so both groups converged in employment, earnings, and consumption. Grants had lasting impacts on assets, skilled work, and possibly child health, but had little effect on mortality, fertility, health or education." (Blattman, Fiala, and Martinez 2020)
There may be some extra challenges or costs when conducting long-term impact evaluations, largely because of the difficulties with tracking down participants more than a decade after a project has ended. Some participants may not be found at all, meaning that only studies that started out with a large enough number of participants to allow for attrition are good candidates for long-term follow-ups.
Still, the potential for long-term follow-ups is huge, because a large number of impact evaluations are now more than 10 years old. The Development Evidence Portal hosts more than 4500 impact evaluations published between 2000 and 2015. Many of those studies may represent good candidates for long-term follow-ups.
At 3ie, we're hoping to encourage the development and humanitarian fields to conduct more of these long-term follow-up impact evaluations. If you would like to know more, have ideas for long-term follow-ups, or are interested in collaborating, let us know! Leave a comment below or email anagy[at]3ieimpact[dot]org.
