'Impact evaluation in peace building situations: Realistic?
July 6, 2011
Speaker: Annette Brown
Poverty rates are 20 percent higher in countries affected by conflict and violence, according to the World Development Report 2011. Conflict and fragility are major obstacles to poverty reduction and the achievement of the Millennium Development Goals. Many agencies, such as the US Agency for International Development, the UK Department for International Development, and multilaterals such as the World Bank and the United Nations, fund stabilization interventions, which can also be thought of as “peace building” interventions.
Stabilization interventions in fragile states aim to prevent organized violence between groups in conflict or protect civilians from exposure to violence or intimidation. Given the nature of these programmes, designing and conducting an impact evaluation is a daunting task.
“For example, for some impact evaluation designs, you need to plan the study before the intervention starts. This is difficult given that most stabilization interventions ramp up quickly and aim to provide assistance rapidly,” said Dr. Annette Brown, 3ie Chief Evaluation Officer.
Good monitoring and baseline data for conducting impact evaluations in this case may seem an ill-afforded luxury to programme implementers. Hence identifying who has been ‘treated’ and who has not becomes a problem for evaluators. The selection of beneficiaries for such interventions can also become a sensitive issue depending on the drivers of conflict in the country. Warring factions in a community could object to the selection and exacerbate the conflict situation, explained Dr. Brown.
These challenges have meant that little is known about the impact of these stabilization programmes. Although the United States has recently funded at least 165 such interventions, only one impact evaluation of a U.S. government funded programme has been carried out. Overall, Dr. Brown and her co-author have identified 23 impact evaluations that have been completed or are currently underway in post-conflict or fragile states like Rwanda, Afghanistan, Liberia and Democratic Republic of Congo.
The key to getting credible results lies in the careful design of the impact evaluation. In Rwanda, a randomized control trial of a radio soap opera to deliver messages of peace was conducted. While the treatment group was encouraged to get together and listen to the radio show “New Dawn”, the control group listened to a health programme. Interestingly, the outcomes were measured in terms of observable behavior like people’s willingness to speak out or disagree with other members of the group and the community members behavior in deciding how to share the gifts of a cassette player.
Dr. Brown offered several pointers for getting around the complexities of designing impact evaluations in fragile context. “The theory of change for a stabilization programme is complicated and hence outcomes need to be tested at various levels of change. We need to be open to quasi-experimental methods and correctly use statistical methods to address problems like the lack of baseline data or small samples. Ensuring there are well-maintained records of the programme information would also go a long way in interpreting the findings,” she concluded.
For more information, download Dr. Brown's presentation Evaluating stabilization interventions (1.5 MB)