Replication
We set up our Replication Program to address the need for a freely available global public good that helps improve the quality and reliability of impact evaluation evidence used for development decision-making. Replication is the most established method of research validation in science, yet it has not been fully embraced by the research community or development donors, leading to this gap.
3ie funded five replication studies on influential impact evaluations of HIV prevention and treatment interventions. The studies included cash transfers, country-level development assistance programs, HIV education programs and antiretroviral therapy administration. This brief highlights lessons learned and recommendations for donors, policymakers and program implementers.
What is replication?
The 3ie program highlights the benefits of replicating impact evaluations of development studies to incentivize replication of selected influential, innovative or controversial impact evaluations. Since 2012, we have funded more than 20 internal replications. This type of replication uses data from the original study and possibly existing secondary datasets from the same location to check the validity and robustness of the estimations and recommendations.
Benefits of replication
Where 3ie-funded replication studies confirm that the impact evaluation findings are valid and robust, they lend additional credibility to these findings for use in policy and program decision-making.
For impact evaluations revealed to have invalid or non-robust findings, the replication results caution decision-makers in the use of those findings. The overall benefit of 3ie’s program, though, is to improve the incentives for all impact evaluators to conduct careful analysis leading to credible findings in the first place.
This selected bibliography attests to the fact that there is a growing number of replications in economics. For a multidisciplinary list of previous research on the practice of replication, see our replication methodologies bibliography. A comprehensive database of replication studies can also be found on the University of Göttingen's replication wiki website.
Replication program on HIV prevention
3ie aims to improve the evidence base around HIV prevention and treatment in low- and middle-income countries. To this end, we funded five internal replications of influential or innovative impact evaluations of biomedical, behavioural, social, and structural HIV prevention and treatment interventions. Internal replication is the reconciliation and re-analysis of the findings of an original study using the study’s own raw data.
Interventions include antiretroviral therapy, conditional cash transfers, country-level HIV treatment, and prevention and education programs. If you have additional queries about this program, please write to replication@3ieimpact.org.
Background
Three decades of work on HIV prevention have yielded a series of effective biomedical, behavioural and structural interventions among specific populations in small-scale settings and/or in randomized controlled trials. To date, few of them have been adopted at a population level to prevent HIV. Although most of them show clear potential for a major impact in reducing HIV and AIDS and the associated human and economic costs, the importance of the effort to prevent HIV and the magnitude of effort required to scale up any evidence-based intervention require the study results to be carefully reviewed, understood and confirmed.
Researchers conduct replications of an original study that is included in 3ie’s Impact Evaluation Repository. Accepted replications are published as part of the 3ie Replication Paper Series. This program was funded by the Gates Foundation.
Award winners
Replication researchers: Fang Yu, Nicholas A Hein and Danstan S Bagenda
Original paper title: The Regai Dzive Shiri Project: results of a randomised trial of an HIV prevention intervention for Zimbabwean youth
Original researchers: Frances M Cowan and colleagues
Original publication: Lancet
Read more
Replication researcher: Eric Djimeu
Original paper title: Timing of antiretroviral therapy for HIV-1 infection and tuberculosis
Original researchers: Diane V Havlir and colleagues
Original publication: The New England Journal of Medicine
Read more
Replication researchers: Jiangtao Luo, Nicholas A Hein and Danstan S Bagenda
Original paper title: HIV development assistance and adult mortality in Africa
Original researchers: Eran Bendavid and colleagues
Original publication: JAMA
Read more
Replication researchers: Baojiang Chen and Morshed Alam
Original paper title: Task shifting of antiretroviral treatment from doctors to primary-care nurses in South Africa (STRETCH): a pragmatic, parallel, cluster-randomised trial
Original researchers: Lara Fairall and colleagues
Original publication: Lancet
Read more
Replication researchers: Lynette M Smith, Nicholas A Hein and Danstan S Bagenda
Original paper title: Effect of a cash transfer programme for schooling on prevalence of HIV and herpes simplex type 2 in Malawi: a cluster randomized trial
Original researchers: Sarah Baird and colleagues
Original publication: Lancet
Read more
Key documents and templates
3ie replication program document
Request for expressions of interest
RW3 individual consultant budget template
RW3 institutional consultant budget template
RW3 review form
RW3 individual consulting agreement
RW3 institutional consulting agreement
Replication program on financial services for poor
Over the last four decades, financial services for the poor have become a priority in L&MICs. This has led to the development of multiple financial instruments, including microlending, microsavings, microinsurance, alternative collateralization, prize-linked savings accounts, and others with different sets of risk profiles and targeted at poor populations. Although financial services for the poor have seen significant uptake, questions remain regarding whether impacts vary based on population targeted, product design and delivery, risk management practices of the service provider, country of operation, associated skills transfer, and other aspects of the program design.
Background
3ie funded seven replications to improve the evidence base on financial services for the poor. This was done through the funding of internal replications of influential or innovative impact evaluations of financial interventions designed for low-income populations in low- and middle-income countries. Internal replication is the reconciliation and re-analysis of the findings of an original study using the study’s own raw data.
Replication researchers selected a study included in the candidate studies list. This list includes impact evaluations of mobile money, cash transfers, bank deposits, and other financial service interventions targeted towards underserved and unbanked populations in developing countries.
The financial services for the poor replications were funded by the Bill & Melinda Gates Foundation.
Award winners
Replication researcher: Nazila Alinaghi
Original paper title: Risk sharing and transactions costs: evidence from Kenya’s mobile money revolution
Original researchers: William Jack and Tavneet Suri
Original publication: American Economic Review
Read more
Replication researcher: Akinwande Atanda
Original paper title: Building state capacity: evidence from biometric smartcards in India
Original researchers: Karthik Muralidharan, Paul Niehaus and Sandip Sukhantankar
Original publication: American Economic Review
Read more
Replication researchers: Edmundo Beteta, Giovanna Aguilar, Oliver Elorreaga, Jean Pierre Meneses, Edgar Ventura and César Huaroto
Original paper title: Payment mechanisms and anti-poverty programs: evidence from a mobile money cash transfer experiment in Niger
Original researchers: Jenny Aker and colleagues
Original publication: Economic Development and Cultural Change
Read more
Replication researcher: Stefan Lhachimi
Original paper title: Cash, food, or vouchers? Evidence from a randomized experiment in northern Ecuador
Original researchers: Melissa Hidrobo and colleagues
Original publication: Journal of Development Economics
Read more
Replication researcher: Maira Reimao
Original paper title: Cash or condition? Evidence from a cash transfer experiment
Original researchers: Sarah Baird, Craig McIntosh and Berk Ozler
Original Publication: The Quarterly Journal of Economics
Read more
Replication researcher: Jesper Stage
Original paper title: Facilitating savings for agriculture: field experimental evidence from Malawi
Original researchers: Lasse Brune and colleagues
Original publication: Economic Development and Cultural Change
Read more
Replication researcher: Hongmei Wang
Original paper title: The short-term impact of unconditional cash transfers to the poor: experimental evidence from Kenya
Original researchers: Johannes Haushofer and Jeremy Shapiro
Original publication: The Quarterly Journal of Economics
Read more
Key documents and templates
3ie replication program document
Request for expressions of interest
RW4 FSP individual consultant budget template
RW4 FSP institutional consultant budget template
RW4 FSP review form
RW4 FSP individual consulting agreement
RW4 FSP institutional consulting agreement
3ie Replication Program’s contracts notification and communication policy
