Innovations in data for impact evaluation

In alignment with our mission, 3ie promotes rigorous, efficient, and ethical use of innovative data sources for impact evaluations, including in those conducted by 3ie, by 3ie research partners, and in the global development community more broadly. This includes stock-taking and systematic mapping of sources and methods; research capacity development for partner institutions, especially in low-and middle-income countries; convening and collaborating with leaders in this space; and joining the global conversation about how these sources can be used to advance evidence-informed equitable, inclusive and sustainable development. To this end, 3ie has supported a number of impact evaluations that use innovate data sources and is actively collaborating on projects with key partners.

Background

In recent years, a confluence of trends has generated a substantial increase in the use of non-traditional data sources for impact evaluations. For example:

  • Rapidly advancing technology has created new options for data collection and analysis in impact evaluation, opening the door to more rigorous study designs and further exploration of under-researched topics and locations.
  • The global COVID-19 pandemic has accelerated remote data collection to minimize the risk of spreading the virus and underscored the urgency of getting accurate data quickly.
  • Policymakers, program implementers, and funders of international development increasingly demand faster, cheaper, and more customized evidence to inform decision-making.
  • A growing number of multidisciplinary research teams and multi-sectoral initiatives have spurred the development of increasingly sophisticated analysis methods to better account for complexity in social, organizational, environmental, and economic systems.

These non-traditional sources include remotely sensed data, geospatial data, big data, and others. Though some of these data types have been around for decades, we refer to them as innovations because of their relative novelty in impact evaluations. There has also been an increased interest in using these data sources in the field of international development over the past several years. Since there is less collective experience with these data sources, there is still much to be learned about how best they can be leveraged to promote better impact evaluation.

The need for innovation in impact evaluation data sources is driven both by long-term trends and urgent needs

While the urgency of COVID-19 pandemic response has prompted the rapid development of new, innovative ways of gathering information on human health, welfare, and development - for description, prediction, and causal attribution - these innovations also address long-term needs in impact evaluation research. These include strengthening the research design; increasing the scale, speed, and affordability of impact evaluation; and enabling a greater focus on difficult contexts (e.g., conflict-affected areas, humanitarian emergencies, pandemics), among others.

Rapid advances in big data sources and methods bring new opportunities and considerations to impact evaluation

Increased generation and accessibility of big data are prompting the use of new tools and approaches at the intersection of data science and impact evaluation. These include predictive analytics, machine learning, and increasingly sophisticated study designs, which are being used to better account for complexity in programs and interventions.

Widely touted benefits include faster and cheaper studies; the potential for increased variety and geographic scale of measured variables; and the ability to generate more robust comparison groups, or counterfactuals, which strengthens the evidence that a particular intervention had a causal effect on a targeted social outcome. At the same time, there are new challenges related to informed consent data privacy and security, transparency of methods, underrepresentation of certain populations, and broader questions related to the validity and ethics of evaluations conducted entirely from afar.

Amid this discussion lie considerable, sometimes crucial, and often under-appreciated differences of the relative benefits and challenges of different methods and different types of big data, including human-sourced (e.g., social media, crowd-sourcing, citizen-reporting); process-mediated (e.g., administrative data, call detail records, e-transactions); and machine-generated (e.g., from satellites, sensors, drones).

3ie promotes rigorous and ethical application of innovations in data for impact evaluations, with an emphasis on building research capacity in low-and middle-income countries

Key work focusing on innovative data sources includes conducting, quality assuring, and funding impact evaluations; stock-taking and systematic mapping of sources and methods; research capacity building for partner institutions; convening and collaborating with leaders in this space; and joining the global conversation about how these sources can be used to advance evidence-informed equitable, inclusive and sustainable development.

Big data systematic map

Gaps exist in terms of access to reliable data to monitor and evaluate the progress of development outcomes and targets such as sustainable development goals (SDGs) and credible evidence to decide on future resource allocation to achieve the targets. Data gaps are particularly significant for the populations and countries where the need for evidence informed policy decisions are perhaps the greatest.

The big data systematic map, funded by the Centre for Excellence for Development Impact and Learning (CEDIL), aims to address this gap in information. In this map we visualize the use of big data to evaluate development outcomes across the world with a special focus on challenging contexts. It identifies and appraises rigorous impact evaluations, systematic reviews and the studies that have innovatively used big data to measure development outcomes.

View Big Data Systematic map

To access the submaps, use the links below:

CEDIL Working paper | Using big data for evaluating development outcomes: a systematic map

CEDIL Working paper brief | Using big data for impact evaluations

CEDIL Blog | Big Data in the time of a pandemic

Related activities

Events

Publications | 3ie has already been actively working in this space. As of December 2019, 3ie has funded 13 impact evaluations that used innovative data sources, such as satellite data, digital sensors, mobile technology, drones, and other techniques. Links to the published studies can be found in the related content section below.

Partnership | 3ie is also a partner of the Geo4Dev initiative, along with CEGA, New Light Technologies, and the Development Impact Evaluation (DIME) group at the World Bank.

To learn more about how geospatial analysis can be integrated into impact evaluations, check out this flyer here. To commission geospatial impact evaluations, contact data@3ieimpact.org or rs@nltgis.com.

Related content

Impacts of the Stimulate, Appreciate, Learn and Transfer community engagement approach to increase immunization coverage in Assam, India

Impact evaluation 3ie 2020
 
Authors of this report evaluate the effectiveness of a community engagement intervention that strives to promote community ownership to increase uptake of vaccinations in Assam, India.

Impacts of a novel mHealth platform to track maternal and child health in Udaipur, India

Impact evaluation 3ie 2020
 
Authors of this report evaluate the impact of a mobile health tracking system, featuring digital maternal and child health records, and voice call reminders on immunisation outcomes for maternal and child health infants in Udaipur, India.

Using big data to evaluate the impacts of transportation infrastructure investment: the case of subway systems in Beijing

Impact evaluation 3ie 2020
 
Using big data, authors of this impact evaluation report examine the social, economic and environmental impacts of the rapid expansion of the subway system in Beijing, China.

Evaluating the impact of infrastructure development: case study of the Konkan Railway in India

Impact evaluation 3ie 2020
 
Authors of this impact evaluation report present empirical evidence on the long-term socioeconomic and environmental impact of the Konkan Railways project, a large-scale infrastructure project in India, on the local ecosystem of the Konkan region.

Impacts of community monitoring of socio-environmental liabilities in the Ecuadorian and Peruvian Amazon

Impact evaluation 3ie 2019
 
In this impact evaluation report, authors evaluated a technology-enabled community intervention to enhance local communities’ detection, monitoring and reporting capabilities, and their ability to stake socio-environmental claims that result in adequate compensation.

Impacts of key provisions in Ghana’s Petroleum Revenue Management Act

Impact evaluation 3ie 2019
 
The authors evaluated the impact of using public meetings and an ICT platform used by the Public Interest and Accountability Committee to promote transparency and accountability.

Cash For Carbon: A Randomized Trial Of Payments For Ecosystem Services To Reduce Deforestation

Impact evaluation 3ie 2019 Publication type : 3ie grantee final report
Author : Seema Jayachandran, Joost De Laat, Eric F. Lambin, Charlotte Y. Stanton, Robin Audy, Nancy E. Thomas
Sector : Agriculture, fishing, and forestry

Impacts of the Stimulate, Appreciate, Learn and Transfer community engagement approach to increase immunization coverage in Assam, India

Impact evaluation 3ie 2020
 
Authors of this report evaluate the effectiveness of a community engagement intervention that strives to promote community ownership to increase uptake of vaccinations in Assam, India.

Impacts of a novel mHealth platform to track maternal and child health in Udaipur, India

Impact evaluation 3ie 2020
 
Authors of this report evaluate the impact of a mobile health tracking system, featuring digital maternal and child health records, and voice call reminders on immunisation outcomes for maternal and child health infants in Udaipur, India.

Using big data to evaluate the impacts of transportation infrastructure investment: the case of subway systems in Beijing

Impact evaluation 3ie 2020
 
Using big data, authors of this impact evaluation report examine the social, economic and environmental impacts of the rapid expansion of the subway system in Beijing, China.

Evaluating the impact of infrastructure development: case study of the Konkan Railway in India

Impact evaluation 3ie 2020
 
Authors of this impact evaluation report present empirical evidence on the long-term socioeconomic and environmental impact of the Konkan Railways project, a large-scale infrastructure project in India, on the local ecosystem of the Konkan region.

Impacts of community monitoring of socio-environmental liabilities in the Ecuadorian and Peruvian Amazon

Impact evaluation 3ie 2019
 
In this impact evaluation report, authors evaluated a technology-enabled community intervention to enhance local communities’ detection, monitoring and reporting capabilities, and their ability to stake socio-environmental claims that result in adequate compensation.

Impacts of key provisions in Ghana’s Petroleum Revenue Management Act

Impact evaluation 3ie 2019
 
The authors evaluated the impact of using public meetings and an ICT platform used by the Public Interest and Accountability Committee to promote transparency and accountability.

Cash For Carbon: A Randomized Trial Of Payments For Ecosystem Services To Reduce Deforestation

Impact evaluation 3ie 2019 Publication type : 3ie grantee final report
Author : Seema Jayachandran, Joost De Laat, Eric F. Lambin, Charlotte Y. Stanton, Robin Audy, Nancy E. Thomas
Sector : Agriculture, fishing, and forestry
There are no systematic reviews

Big Data Systematic Map

Evidence gap map

This systematic map shows where big data has been used to evaluate development outcomes across the world. This map is different from 3ie’s evidence gap maps, and innovatively uses the same methodology to map data sources to development outcomes. Given the scale, we created five sub-maps associated with outcomes including: economic development and livelihoods, health and well-being, governance and human rights, urban development and environmental sustainability.

There are no replication studies