Transparent, Reproducible and Ethical Evidence (TREE)
3ie champions the research transparency and reproducibility movement as a means of understanding and mitigating challenges to the credibility of social science research, while also working toward stronger integration of ethical principles into practice. Through integration of TREE principles and practices in our research workflow, 3ie aims to ensure: (i) the rigor and credibility of research activities by asking demand-driven questions and mitigating intentional and unintentional scientific misconduct; and (ii) research activities follow foundational principles of research ethics: respect for persons, beneficence, and justice.
Five lessons from our Transparent, Reproducible, and Ethical Evidence (TREE) reviews
Our work to pilot test and implement our TREE Review Framework has been underway since December 2021 with four 3ie-managed evaluations and one external client. In this blog, we discuss lessons from these five TREE Reviews as part of which we have engaged with more than 15 study team members working on evaluations in four countries across three continents.
Call for submission - Journal for Development Effectiveness
3ie is inviting papers for a special issue of the Journal for Development Effectiveness titled ‘Trends in Research Transparency, Reproducibility, and Ethics for Development Effectiveness’. The special issue will focus on projects, programs, and policies and practices that aim to improve the transparency, reproducibility, and ethical conduct of impact evaluations and/or test whether improvements in these areas improve development effectiveness. If you have a paper that fits our special issue criteria, please send it to us by 31 March 2023. Read about the special issue's scope and themes.
TREE Review Framework – Reimagining ethics review and oversight
At 3ie, we are refining a process to help research teams consider the ethics questions raised in social science research and document their decisions. In this blog, read about how our Transparent, Reproducible and Ethical Evidence (TREE) Review Framework complements the necessary work of the Institutional Review Board (IRBs) while ensuring we do not outsource our ethical judgment to them, we update our ethics review system to respond to changing needs of the 21st century, and we continue to foster scientific integrity.
Our Transparent, Reproducible and Ethical Evidence (TREE) Policy articulates the organization’s commitment to delivering the resources and expertise research teams need to achieve TREE objectives: to ensure rigor and credibility of research and adhere to the principles of ethical research. The TREE Policy defines how TREE work is guided by 3ie’s ethical principles:
- Everyone with whom 3ie research staff comes into contact, regardless of sex, age, gender identity, disability, sexual orientation, ethnic origin, religion, or caste, has the right to be protected from all forms of harm, abuse, neglect and exploitation.
- Research transparency, reproducibility, and ethics are integral to research rigor.
- Foundational ethics – respect for persons, beneficence, and justice – are standards based on fundamental human rights.
- Ethics should be as focused on the purpose of the research activity and producing meaningful analyses for improving development effectiveness as it is to ensure the protection of the health and safety of the research participants and team.
- Ethics requires considering the potential unintended consequences of conducting the research and weighing these as additional benefits and/or costs.
- Ethical standards require the means to monitor and address concerns throughout the research life cycle.
The primary focus of the TREE Helpdesk is to support research teams on:
- Training on incorporating TREE within the research life cycle;
- Training in protection of human subjects;
- Study registration on 3ie’s RIDIE, as well as other relevant registries;
- Pre-analysis Plan (PAP) or protocol review and publication on RIDIE and
- Adherence to standardized reporting and citation requirements for publications.
The Helpdesk is also available to support researchers on data management, de-identification, and push-button replication through the TREE Data Center and on navigating ethical issues through TREE Review.
The primary focus of the TREE Data Center is to support research teams on:
- Defining who has access to identifiable data and when and how that data may, or may not, be shared through Data Management Plans and/or Data Sharing Agreements
- Establishment of secure data transfer and storage practices
- Review of data de-identification protocols to mitigate re-identification risk and adhere to promises of confidentiality
- Integration of Push-Button Replication (PBR) protocols into analysis workflow to improve computational reproducibility
- Data sharing on the 3ie Dataverse, or other data sharing platforms.
TREE Review Framework
The primary focus of the TREE Review is to support research teams on:
- Strengthening the informed consent review, pilot, and implementation process to ensure appropriate engagement with research participants on key issues related to research purpose, risk, benefits, and data-sharing practices
- Engaging with Institutional Review Board (IRB) for specific research activities based on local, journal, funding, and other requirements
- Conducting independent review of 3ie-defined TREE practices to assess research design, implementation, and dissemination for alignment with ethical principles
- Ensure alignment between promises of confidentiality and data de-identification and sharing practices.
Our TREE Review Framework is based on 10 ethical standards for evaluations. Its objectives are to: (i) establish ethical standards for evaluations, (ii) better integrate best practices for transparency, reproducibility, and ethics into evaluation workflow; and (iii) establish timely, independent review of risks facing the evaluation team to produce credible, meaningful, ethical evidence and the efforts to mitigate such risks.
The Framework Questionnaire is designed to pose specific questions regarding evaluation teams’ actions and decisions related to achieving these 10 ethical standards. Teams may use the TREE Review Framework Questionnaire for two purposes:
- Documentation only: Team members may use this Questionnaire as a guide for actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle.
- Documentation and submission for independent assessment and feedback: Team members may use this Questionnaire to document their actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle and submit to the TREE Review Committee for assessment and feedback.
3ie understands that evidence-based policy to improve development outcomes requires credible, rigorous research methods. Just as 3ie called attention to the need for evidence to inform project design and implementation, there is a need for evidence to inform certain decisions in research design and implementation. TREE research aims to answer specific questions, including:
- Does incorporating TREE into research practice improve research quality and credibility? What evidence exists and where are the evidence gaps?
- What ethical concerns do research teams have? What evidence exists on potential best practices and where are the evidence gaps?
- How are TREE publications – registries, PAPs – used to strengthen meta-analysis and systematic reviews?
- Do data-sharing efforts enable additional analysis and learning opportunities? What questions related to development effectiveness can be answered with open data?
For more information, please write to TREE@3ieimpact.org mentioning '3ie TREE' in the subject line.
Working paper | Integrating PBR into workflow
This paper examines the portfolio of 179 3ie impact evaluations carried out from 2008 to 2020. We first assess if research transparency alone improves the credibility of research findings and determine that while transparency is a necessary condition, it is not sufficient for computational reproducibility. We then examine how incorporation of both research transparency and reproducibility – through push-button replication (PBR) – into research workflow leads to almost 100% computational reproducibility of annually completed research within two years. Read more
Data and code | Incentives for child immunization in Ethiopia
Through 3ie Dataverse, the research team submitted de-identified data alongside analysis code for public use. The data is available to facilitate reproducibility of analysis and has passed through 3ie’s push-button replication (PBR) assessment. Read more