Impact evaluation
Development programmes are fundamentally about improving outcomes: boosting incomes, increasing productivity, encouraging learning, improving health and protecting the environment, to name a few. Understanding whether or not an intervention accomplishes its objectives – and why or why not – is crucial for accountability, informed decision-making and the efficient use of resources. Impact evaluations – rigorous studies that measure the effects of international development programmes – are at the heart of 3ie's work. Our holistic approach to designing and conducting impact evaluations provides a rich set of information for decision-makers.
3ie’s approach to impact evaluation starts by identifying the set of policy-relevant research questions that will drive decision-making. We then identify a rigorous research design, suitable for the programme’s operational context, that will provide credible answers to those questions. In addition to establishing rigorous quantitative estimates of a programme's effect, we believe the most useful impact evaluations seek to understand why a program worked and at what cost. Our focus on high-quality impact evaluations also drives our efforts to strengthen evaluation capacity; ensure research is transparent, ethical, and replicable; use innovative data sources; and promote evidence uptake and use.
What is an impact evaluation?
Impact evaluations are designed to answer the question: "What was the effect of an intervention on an outcome?" Measuring such an effect is not as simple as it might initially seem. Consider a training programme for female entrepreneurs, like this one in Kenya we helped evaluate. Simply measuring participants’ business performance before and after the programme does not necessarily measure the programme’s impact, because nothing happens in a vacuum. Many factors affect business performance from month to month, from seasonal variations to technological change to global health pandemics. So how can we separate the effect of the programme from all those other factors? Impact evaluations provide a toolkit of methods to measure the effects of that programme, and that programme alone.
In the Kenya example, the research team used a randomized control trial, a method similar to the studies doctors use to test the effectiveness of new medicines. By randomly assigning some individuals to participate in a given programme while others are not, we can compare the outcomes across the two groups to see if the programme works. Because random assignment produces two groups that are similar except for the presence of the programme, we rule out other factors that might otherwise account for the differences between the two groups, leading to a causal interpretation of impact.
However, not everything can (or should) be randomized. Evaluations of other interventions like food aid to conflict-affected families in Mali or an environmental programme in Mexico rely on other tools: quasi-experimental methods. With careful research designs, advanced statistical techniques, and innovative approaches to data collection, we can identify an appropriate comparison group and measure an intervention's effect.
Process evaluations and implementation research
Our experience has shown that impact evaluation findings are most informative when they stand alongside additional data developed through process evaluations, formative evaluations, and other implementation research. 3ie is a pioneer in showing how impact evaluations and implementation research can work together to promote evidence-informed action. Using primarily qualitative approaches, these evaluations provide rich contextual data to colour in the backdrop around impact evaluation findings. Our new process evaluation guidelines provide impact evaluators with tools and ideas on how to explore and add relevant elements of process evaluations to experimental and quasi-experimental impact evaluation designs. Examples of our work combining impact and process evaluations include our research on India's National Rural Livelihoods Mission and our evidence programme on agricultural insurance.
Cost analysis
Including an analysis of a programme's costs in an impact evaluation provides another essential piece of information policymakers need. Unfortunately, too many impact evaluations omit this essential component. Incorporating costs with rigorous estimates of impact allows for the comparison of two or more interventions with cost-effectiveness analysis, or for the comparison of the value of benefits generated by an intervention relative to its costs, using a cost-benefit analysis. Both cost-benefit and cost-effectiveness analyses provide important insights for deciding whether to invest in a programme and what approach is more cost-effective.
Quality assurance
In addition to conducting impact evaluations, we also offer quality assurance services which ensure that impact evaluations conducted by other organizations meet the highest standards. Our approach to quality assurance verifies that an impact evaluation is designed appropriately to identify a programme's effect and data collection strategies are suitable for the research context. We also make sure that evaluations are conducted transparently and that all analytic findings are replicable. We have quality assured impact evaluations from organisations including Save the Children, the Population Fund of India, and Peru's Ministry of Education, among others.
Want to learn more? Drop us a line at info@3ieimpact.org for a personalized consultation with an Evaluation Specialist.