Innovations in Data for Impact Evaluation

In recent years, a confluence of trends has generated a substantial increase in the use of non-traditional data sources for impact evaluations. For example:

• Rapidly advancing technology has created new options for data collection and analysis in impact evaluation, opening the door to more rigorous study designs and further exploration of under-researched topics and locations.
• The global COVID-19 pandemic has accelerated remote data collection to minimize the risk of spreading the virus and underscored the urgency of getting accurate data quickly.
• Policymakers, program implementers, and funders of international development increasingly demand faster, cheaper, and more customized evidence to inform decision-making.
• A growing number of multidisciplinary research teams and multi-sectoral initiatives have spurred the development of increasingly sophisticated analysis methods to better account for complexity in social, organizational, environmental, and economic systems.

These non-traditional sources include remotely-sensed data, geospatial data, big data, and others. Though some of these data types have been around for decades, we refer to them as innovations because of their relative novelty in impact evaluations. There has also been an increased interest in using these data sources in the field of international development over the past several years. Since there is less collective experience with these data sources, there is still much to be learned about how best they can be leveraged to promote better impact evaluation.

The need for innovation in impact evaluation data sources is driven both by long-term trends and urgent needs

While the urgency of COVID-19 pandemic response has prompted the rapid development of new, innovative ways of gathering information on human health, welfare, and development - for description, prediction, and causal attribution - these innovations also address long-term needs in impact evaluation research. These include strengthening the research design; increasing the scale, speed, and affordability of impact evaluation; and enabling a greater focus on difficult contexts (e.g., conflict-affected areas, humanitarian emergencies, pandemics), among others.

Rapid advances in big data sources and methods bring new opportunities and considerations to impact evaluation

Increased generation and accessibility of big data are prompting the use of new tools and approaches at the intersection of data science and impact evaluation. These include predictive analytics, machine learning, and increasingly sophisticated study designs, which are being used to better account for complexity in programs and interventions.

Widely touted benefits include faster and cheaper studies; the potential for increased variety and geographic scale of measured variables; and the ability to generate more robust comparison groups, or counterfactuals, which strengthens the evidence that a particular intervention had a causal effect on a targeted social outcome. At the same time, there are new challenges related to informed consent data privacy and security, transparency of methods, underrepresentation of certain populations, and broader questions related to the validity and ethics of evaluations conducted entirely from afar.

Amid this discussion lie considerable, sometimes crucial, and often under-appreciated differences of the relative benefits and challenges of different methods and different types of big data, including human-sourced (e.g., social media, crowd-sourcing, citizen-reporting); process-mediated (e.g., administrative data, call detail records, e-transactions); and machine-generated (e.g., from satellites, sensors, drones).

3ie promotes rigorous and ethical application of innovations in data for impact evaluations, with an emphasis on building research capacity in low- and middle-income countries

Key work focusing on innovative data sources includes conducting, quality assuring, and funding impact evaluations; stock-taking and systematic mapping of sources and methods; research capacity building for partner institutions; convening and collaborating with leaders in this space; and joining the global conversation about how these sources can be used to advance evidence-informed equitable, inclusive and sustainable development.

Gaps exist in terms of access to reliable data to monitor and evaluate the progress of development outcomes and targets such as sustainable development goals (SDGs) and credible evidence to decide on future resource allocation to achieve the targets. Data gaps are particularly significant for the populations and countries where the need for evidence informed policy decisions are perhaps the greatest.

The big data systematic map, funded by the Centre for Excellence for Development Impact and Learning (CEDIL), aims to address this gap in information. In this map we visualize the use of big data to evaluate development outcomes across the world with a special focus on challenging contexts. It identifies and appraises rigorous impact evaluations, systematic reviews and the studies that have innovatively used big data to measure development outcomes.

View big data systematic map

To access the submaps, use the links below:

• Economic development and livelihoods
• Health and well-being
• Governance and human rights
• Urban development
• Environmental sustainability

CEDIL Working paper | Using big data for evaluating development outcomes: a systematic map

CEDIL Working paper brief | Using big data for impact evaluations

CEDIL Blog | Big Data in the time of a pandemic

Events

• Night time lights: applications and impact evaluation (Part I), 3rd Annual Geo4Dev Symposium & Workshop, 10-11 December | Watch recording
• Do you have to ask? Innovations in data collection and analysis for impact evaluation when sample surveys and face-to-face interviews are challenging, Asian Evaluation Week | Watch recording
• Using Big Data for evaluating development outcomes: a systematic map, KDIS-3ie-ADB-ADBI Conference on Impact Evaluation | Watch recording

Publications | 3ie has already been actively working in this space. As of December 2019, 3ie has funded 13 impact evaluations that used innovative data sources, such as satellite data, digital sensors, mobile technology, drones, and other techniques. Links to the published studies can be found in the related content section below.

Partnership | 3ie is also a partner of the Geo4Dev initiative, along with CEGA, New Light Technologies, and the Development Impact Evaluation (DIME) group at the World Bank.

To learn more about how geospatial analysis can be integrated into impact evaluations, check out this flyer here. To commission geospatial impact evaluations, contact data@3ieimpact.org or rs@nltgis.com.

Visit Geo4Dev website | Read media article