Transparent, Reproducible and Ethical Evidence (TREE)

Our Transparent, Reproducible and Ethical Evidence (TREE) Policy articulates the organization’s commitment to delivering the resources and expertise research teams need to achieve TREE objectives: to ensure rigor and credibility of research and adhere to the principles of ethical research. The TREE Policy defines how TREE work is guided by 3ie’s ethical principles: 

• Everyone with whom 3ie research staff comes into contact, regardless of sex, age, gender identity, disability, sexual orientation, ethnic origin, religion, or caste, has the right to be protected from all forms of harm, abuse, neglect and exploitation.
• Research transparency, reproducibility, and ethics are integral to research rigor.
• Foundational ethics – respect for persons, beneficence, and justice – are standards based on fundamental human rights.
• Ethics should be as focused on the purpose of the research activity and producing meaningful analyses for improving development effectiveness as it is to ensure the protection of the health and safety of the research participants and team.
• Ethics requires considering the potential unintended consequences of conducting the research and weighing these as additional benefits and/or costs.
• Ethical standards require the means to monitor and address concerns throughout the research life cycle.

The primary focus of the TREE Helpdesk is to support research teams on: 

1. Training on incorporating TREE within the research life cycle;
2. Training in protection of human subjects;
3. Study registration on 3ie’s RIDIE, as well as other relevant registries;
4. Pre-analysis Plan (PAP) or protocol review and publication on RIDIE and 
5. Adherence to standardized reporting and citation requirements for publications.

The Helpdesk is also available to support researchers on data management, de-identification, and push-button replication through the TREE Data Center and on navigating ethical issues through TREE Review. 

The primary focus of the TREE Data Center is to support research teams on:  

1. Defining who has access to identifiable data and when and how that data may, or may not, be shared through Data Management Plans and/or Data Sharing Agreements
2. Establishment of secure data transfer and storage practices
3. Review of data de-identification protocols to mitigate re-identification risk and adhere to promises of confidentiality
4. Integration of Push-Button Replication (PBR) protocols into analysis workflow to improve computational reproducibility
5. Data sharing on the 3ie Dataverse, or other data sharing platforms. 

The primary focus of the TREE Review is to support research teams on:  

1. Strengthening the informed consent review, pilot, and implementation process to ensure appropriate engagement with research participants on key issues related to research purpose, risk, benefits, and data-sharing practices
2. Engaging with Institutional Review Board (IRB) for specific research activities based on local, journal, funding, and other requirements
3. Conducting independent review of 3ie-defined TREE practices to assess research design, implementation, and dissemination for alignment with ethical principles
4. Ensure alignment between promises of confidentiality and data de-identification and sharing practices. 

Our TREE Review Framework is based on 10 ethical standards for evaluations. Its objectives are to: (i) establish ethical standards for evaluations, (ii) better integrate best practices for transparency, reproducibility, and ethics into evaluation workflow; and (iii) establish timely, independent review of risks facing the evaluation team to produce credible, meaningful, ethical evidence and the efforts to mitigate such risks.

The Framework Questionnaire is designed to pose specific questions regarding evaluation teams’ actions and decisions related to achieving these 10 ethical standards. Teams may use the TREE Review Framework Questionnaire for two purposes:

1. Documentation only: Team members may use this Questionnaire as a guide for actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle.
2. Documentation and submission for independent assessment and feedback: Team members may use this Questionnaire to document their actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle and submit to the TREE Review Committee for assessment and feedback.

Access the toolkit here

3ie understands that evidence-based policy to improve development outcomes requires credible, rigorous research methods. Just as 3ie called attention to the need for evidence to inform project design and implementation, there is a need for evidence to inform certain decisions in research design and implementation. TREE research aims to answer specific questions, including: 

1.  Does incorporating TREE into research practice improve research quality and credibility? What evidence exists and where are the evidence gaps? 
2. What ethical concerns do research teams have? What evidence exists on potential best practices and where are the evidence gaps? 
3. How are TREE publications – registries, PAPs – used to strengthen meta-analysis and systematic reviews?
4. Do data-sharing efforts enable additional analysis and learning opportunities? What questions related to development effectiveness can be answered with open data?

For more information, please write to TREE@3ieimpact.org mentioning '3ie TREE' in the subject line. 

Working paper | Integrating PBR into workflow

This paper examines the portfolio of 179 3ie impact evaluations carried out from 2008 to 2020. We first assess if research transparency alone improves the credibility of research findings and determine that while transparency is a necessary condition, it is not sufficient for computational reproducibility. We then examine how incorporation of both research transparency and reproducibility – through push-button replication (PBR) – into research workflow leads to almost 100% computational reproducibility of annually completed research within two years.  Read more

Data and code | Incentives for child immunization in Ethiopia

Through 3ie Dataverse, the research team submitted de-identified data alongside analysis code for public use. The data is available to facilitate reproducibility of analysis and has passed through 3ie’s push-button replication (PBR) assessment. Read more