Innovations in data for impact evaluation

In alignment with our mission, 3ie promotes rigorous, efficient, and ethical use of innovative data sources for impact evaluations, including in those conducted by 3ie, by 3ie research partners, and in the global development community more broadly. This includes stock-taking and systematic mapping of sources and methods; research capacity development for partner institutions, especially in low-and middle-income countries; convening and collaborating with leaders in this space; and joining the global conversation about how these sources can be used to advance evidence-informed equitable, inclusive and sustainable development. To this end, 3ie has supported a number of impact evaluations that use innovate data sources and is actively collaborating on projects with key partners.

Background

In recent years, a confluence of trends has generated a substantial increase in the use of non-traditional data sources for impact evaluations. For example:

  • Rapidly advancing technology has created new options for data collection and analysis in impact evaluation, opening the door to more rigorous study designs and further exploration of under-researched topics and locations.
  • The global COVID-19 pandemic has accelerated remote data collection to minimize the risk of spreading the virus and underscored the urgency of getting accurate data quickly.
  • Policymakers, program implementers, and funders of international development increasingly demand faster, cheaper, and more customized evidence to inform decision-making.
  • A growing number of multidisciplinary research teams and multi-sectoral initiatives have spurred the development of increasingly sophisticated analysis methods to better account for complexity in social, organizational, environmental, and economic systems.

These non-traditional sources include remotely sensed data, geospatial data, big data, and others. Though some of these data types have been around for decades, we refer to them as innovations because of their relative novelty in impact evaluations. There has also been an increased interest in using these data sources in the field of international development over the past several years. Since there is less collective experience with these data sources, there is still much to be learned about how best they can be leveraged to promote better impact evaluation.

The need for innovation in impact evaluation data sources is driven both by long-term trends and urgent needs

While the urgency of COVID-19 pandemic response has prompted the rapid development of new, innovative ways of gathering information on human health, welfare, and development - for description, prediction, and causal attribution - these innovations also address long-term needs in impact evaluation research. These include strengthening the research design; increasing the scale, speed, and affordability of impact evaluation; and enabling a greater focus on difficult contexts (e.g., conflict-affected areas, humanitarian emergencies, pandemics), among others.

Rapid advances in big data sources and methods bring new opportunities and considerations to impact evaluation

Increased generation and accessibility of big data are prompting the use of new tools and approaches at the intersection of data science and impact evaluation. These include predictive analytics, machine learning, and increasingly sophisticated study designs, which are being used to better account for complexity in programs and interventions.

Widely touted benefits include faster and cheaper studies; the potential for increased variety and geographic scale of measured variables; and the ability to generate more robust comparison groups, or counterfactuals, which strengthens the evidence that a particular intervention had a causal effect on a targeted social outcome. At the same time, there are new challenges related to informed consent data privacy and security, transparency of methods, underrepresentation of certain populations, and broader questions related to the validity and ethics of evaluations conducted entirely from afar.

Amid this discussion lie considerable, sometimes crucial, and often under-appreciated differences of the relative benefits and challenges of different methods and different types of big data, including human-sourced (e.g., social media, crowd-sourcing, citizen-reporting); process-mediated (e.g., administrative data, call detail records, e-transactions); and machine-generated (e.g., from satellites, sensors, drones).

3ie promotes rigorous and ethical application of innovations in data for impact evaluations, with an emphasis on building research capacity in low-and middle-income countries

Key work focusing on innovative data sources includes conducting, quality assuring, and funding impact evaluations; stock-taking and systematic mapping of sources and methods; research capacity building for partner institutions; convening and collaborating with leaders in this space; and joining the global conversation about how these sources can be used to advance evidence-informed equitable, inclusive and sustainable development.

Big data systematic map

Gaps exist in terms of access to reliable data to monitor and evaluate the progress of development outcomes and targets such as sustainable development goals (SDGs) and credible evidence to decide on future resource allocation to achieve the targets. Data gaps are particularly significant for the populations and countries where the need for evidence informed policy decisions are perhaps the greatest.

The big data systematic map, funded by the Centre for Excellence for Development Impact and Learning (CEDIL), aims to address this gap in information. In this map we visualize the use of big data to evaluate development outcomes across the world with a special focus on challenging contexts. It identifies and appraises rigorous impact evaluations, systematic reviews and the studies that have innovatively used big data to measure development outcomes.

View Big Data Systematic map

To access the submaps, use the links below:

CEDIL Working paper | Using big data for evaluating development outcomes: a systematic map

CEDIL Working paper brief | Using big data for impact evaluations

CEDIL Blog | Big Data in the time of a pandemic

Related activities

Events

Publications | 3ie has already been actively working in this space. As of December 2019, 3ie has funded 13 impact evaluations that used innovative data sources, such as satellite data, digital sensors, mobile technology, drones, and other techniques. Links to the published studies can be found in the related content section below.

Partnership | 3ie is also a partner of the Geo4Dev initiative, along with CEGA, New Light Technologies, and the Development Impact Evaluation (DIME) group at the World Bank.

To learn more about how geospatial analysis can be integrated into impact evaluations, check out this flyer here. To commission geospatial impact evaluations, contact data@3ieimpact.org or rs@nltgis.com.

Related content

Harnessing transparency initiatives to improve India’s environmental clearance process for the mineral mining sector

Impact evaluation 3ie 2019
 
This study evaluated the impact of a set of reforms to the environmental clearance process in India, which included subjecting larger projects to scrutiny from regulators, independent experts and the public.

A triple win? The impact of Tanzania’s Joint Forest Management programme on livelihoods, governance and forests

Impact evaluation 3ie 2016
 

This study by Persha and Meshack assesses the impact of Tanzania’s Joint Forest Management (JFM) programme on three sets of outcomes -- restoring forests, improving livelihoods and

Impacts of payments for ecosystem services programme in Mexico

Impact evaluation 3ie 2015
 

Policies for reducing emissions from deforestation and forest degradation (REDD) have been a crucial part of international climate change negotiations.

A rapid assessment randomised-controlled trial of improved cookstoves in rural Ghana

Impact evaluation 3ie 2013
 
This impact evaluation, by David I. Levine and Jason Burwen, uses a randomised controlled trial to quantify the impact of an improved wood cookstove on changes in fuel use, exposure to smoke and self-reported health in the upper west region of Ghana.

Harnessing transparency initiatives to improve India’s environmental clearance process for the mineral mining sector

Impact evaluation 3ie 2019
 
This study evaluated the impact of a set of reforms to the environmental clearance process in India, which included subjecting larger projects to scrutiny from regulators, independent experts and the public.

A triple win? The impact of Tanzania’s Joint Forest Management programme on livelihoods, governance and forests

Impact evaluation 3ie 2016
 

This study by Persha and Meshack assesses the impact of Tanzania’s Joint Forest Management (JFM) programme on three sets of outcomes -- restoring forests, improving livelihoods and

Impacts of payments for ecosystem services programme in Mexico

Impact evaluation 3ie 2015
 

Policies for reducing emissions from deforestation and forest degradation (REDD) have been a crucial part of international climate change negotiations.

A rapid assessment randomised-controlled trial of improved cookstoves in rural Ghana

Impact evaluation 3ie 2013
 
This impact evaluation, by David I. Levine and Jason Burwen, uses a randomised controlled trial to quantify the impact of an improved wood cookstove on changes in fuel use, exposure to smoke and self-reported health in the upper west region of Ghana.

There are no systematic reviews
There are no evidence gap maps
There are no replication studies
There is no related content.