Integrating Cost Evidence in Research and Evaluation
Public resources have competing usages, and efficient allocation of resources for development projects is key to improving social outcomes. Policymakers need information on both effectiveness and costs to support prioritization of scarce resources, budget optimization and project management. 3ie’s Cost Evidence Group designs and implements rigorous cost analysis, including as an integrated part of impact evaluation, provides training on measuring and analyzing costs, and supports partners in better use of cost evidence in decision-making.
The Cost Evidence Group at 3ie has developed a handbook for Measuring Cost Effectiveness in Impact Evaluation. In this blog we explain how the handbook provides a comprehensive ‘how-to’ guide for implementing cost-effectiveness analysis and can be adapted for different programs by researchers, implementation teams, monitoring and evaluation officers, and funders, such as government and donors.
Sounds good... but what will it cost? Making the case for rigorous costing in impact evaluation research
Integrating cost-effectiveness analysis into impact evaluations is rarely done due to insufficient training, unclear standards, and limited demand. Organizations like CEGA and 3ie aim to address these challenges by studying market demand, establishing best practices, and promoting cost research sustainability. This blog explores the crucial role of cost evidence in evaluating the effectiveness of development programs.
Cost analyses should be required by donors and get more respect from researchers, 3ie’s expert panel says
Despite policymakers' interest in cost evidence, barriers exist to analyzing program costs in evaluation. While health researchers commonly conduct cost analyses, achieving similar standards across social interventions poses challenges due to lack of standardized outcomes and transparency in reporting costs. This blog summarizes an insightful discussion where experts call for donor pressure, best practice guidelines, and enhanced training to improve cost analysis integration.
Overview
3ie has advocated for the inclusion of cost evidence in research by contributing to multilateral costing initiatives, developing publicly available guidance materials and trainings, and producing cost evidence to inform decision-making. This followed the work of Brown and Tanner (2019) who identified cost evidence as a critical gap in impact evaluation research.
Our work aims to promote the use of cost evidence in evaluation both internally within 3ie and externally among policymakers, funders, and researchers.
Key objectives of 3ie’s Cost Evidence Group:
- Design and implement rigorous cost analysis, including as an integrated part of impact evaluation
- Provide training on measuring and analyzing costs
- Maintain ‘living’ guidelines and tools for implementing cost analysis as part of impact evaluation
- Assess the quality of cost analyses
- Advocate for better use of cost evidence in impact evaluation and related evidence approaches, including by contributing to sector-wide costing communities of practice, and
- Support 3ie partners to identify cost analysis applications that meet their unique needs and context
Measuring costs
Evaluating the outcomes of interventions is only one side of the coin; the other, equally crucial, aspect involves analyzing the costs associated with achieving these outcomes. 3ie champions the integration of detailed cost analysis in impact evaluations to enhance decision-making and foster transparency and accountability.
Our approach:
- Cost analysis plays a pivotal role in informing policy and programmatic decisions. By measuring and analyzing the costs involved in achieving specific outcomes, stakeholders are equipped with a clearer understanding of the value generated by different interventions to optimize resource allocation.
- Rigorous cost analyses provide a detailed account of how resources are utilized to inform programmatic decisions such as program budgeting, design, and scale-up.
- By systematically analyzing and reporting costs, 3ie advocates for a culture of accountability where decisions are made based on evidence of cost-effectiveness, reinforcing the commitment to prudent and responsible use of resources.
Some of the initiatives we are working on include:
- Piloting our approach on 3ie Evaluation and Evidence, Policy and Learning (EPL) projects
- Standards for assessing CEA in evidence synthesis
CEA handbook
3ie has developed a handbook that lays out guidelines providing impact evaluators with tools and ideas on how to design, implement and report metrics of cost-effectiveness in experimental and quasi-experimental impact evaluation designs. The 3ie guide enables you to identify the most appropriate way to collect cost data as part of the evaluation after accounting for uncertainty, missing data, and sensitivity analyses.
We continue to refine and iterate these guidelines based on piloting and user feedback; to that end, we’re currently expanding our scope to include other common CEA applications, such as scale-up decision-support, using incremental cost-effectiveness ratio (ICER) thresholds, and budget impact analysis. We welcome all inquiries, critiques, questions and suggestions. Reach out to a 3ie cost specialist to unpack and calculate the actual costs or true costs of your project at info@3ieimpact.org.
Journal | Special issue on costing
Our Journal of Development Effectiveness’s recent issue features seven papers that provide an insightful discussion of and understanding on a range of challenges of collecting cost data, how to overcome these and different approaches to doing this work.
It includes the following papers:
- The State of Cost-Effectiveness Guidance: Ten Best Resources for CEA in Impact Evaluations
- Cost-effectiveness analysis and joint public production of outputs for development: a preliminary framework
- Facilitating real-time cost collection and evaluating cost-effectiveness in a multi-armed study with government partners in Ghana
- Harnessing cost data to improve early grade reading: cost evidence from a large-scale literacy initiative in Pakistan
- Infections averted by a comprehensive HIV prevention intervention and its cost-effectiveness: a prospective cohort study of persons who inject drugs in Delhi, India
- Benefit-cost analysis of iron fortification of rice in India: modelling potential economic gains from improving haemoglobin and averting anaemia
- Cost and cost-efficiency of unconditional cash transfers in Tahoua, Niger
- Evaluating spillovers and cost-effectiveness of complementary agricultural and social protection interventions: evidence from Lesotho
Read the special issue.
News and features
Vox | We’re finally figuring out if foreign aid is any better than handing out cash
In this article, 3ie’s Daniel Handel underscores the need for evidence-based aid programming, highlighting ongoing efforts to enhance evaluation quality and foster evidence-driven decision-making within USAID and beyond.
Learn more about 3ie’s Commitment to Evidence in this blog.
