Building a culture of evidence is a tall order, one that demands the engagement of different stakeholders committed to evidence-informed policy. While we embed capacity-building activities in our grant programmes, we continue to explore alternative approaches beyond our grants to increase local researchers’ familiarity with impact evaluation, so the pool of research centers able to provide impact evaluation services in a given country expands. Such an approach has the potential of building a community of professionals around the generation and use of rigorous evidence; and even building or expanding the local market for impact evaluation services.

We initially attempted to increase capacity with workshops, sharing knowledge and awarding bursaries. But we deemed our response insufficient, given the potential demand and the need for advisory services customized to local needs.

Our answer? 3ie’s policy window advanced (PWA) program.

Local researchers have a deep understanding of context, political economy and the development challenges in their countries. It makes total sense to commission local research teams to design and implement impact evaluations of local interventions. However, one of the key challenges is the limited number of local researchers familiar with the wide range of methodological tools at their disposal for impact evaluation. Even where strong and experienced research centres exist in L&MICs, such as GRADE in Peru, they may be unable to respond the local and regional demand for impact evaluation services. This limitation often forces government agencies and others to look abroad for such expertise.

In the last couple of years, 3ie collaborated with the National System for Evaluation of Management and Results (SINERGIA-DNP) in Colombia, to tackle this challenge. SINERGIA-DNP understands the importance of impact evaluation and funds several studies each year. However, they have struggled to find enough local researchers familiar with innovative approaches to assess government programs being evaluated.

PWA is one answer to the problem. The program we piloted in Colombia paired local research teams with an international expert, who serves as a team mentor or advisor throughout the design and/or implementation of the impact evaluation. The expert became part of the research team and collaborated on design ideas, suggested alternative methodological approaches, and helped solve implementation challenges. Under PWA 3ie contracts the international experts directly, and they report to 3ie. This set-up reaffirms the independence of the evaluation and the perception of the expert as a peer.

Staff from 3ie and SINERGIA-DNP first reviewed upcoming evaluation tenders and identified three projects for the PWA pilot: a family housing subsidy, the Mandatory Healthcare Plan, and an internet access expansion initiative (summaries of each experience can be found here). We posted descriptions of the programmes to be evaluated and a request for expressions of interest from international experts on the 3ie’s website.

3ie and SINERGIA-DNP then selected the mentors based on both their impact evaluation and sectoral expertise, and matched them to respective projects. In the case of Colombia, it was crucial for the mentors to be fluent in Spanish. Our first mentors were Jose Galdo (Carleton University), Antonio Trujillo (John Hopkins University) and Alberto Chong (Georgia State University).

The experience has been very encouraging. The experts gave helpful guidance and inputs to revise the evaluation research strategies, drawing from their own experiences with similar programmes elsewhere. They guided teams through several approaches to robustness checks, set up protocols on how to implement matching techniques, and worked with their teams to address statistical issues. Throughout implementation, the research teams acknowledged the value experts added.

Aligning incentives

PWA has concrete benefits for all parties involved. The model allows government agencies to commission local teams to conduct impact evaluations and still benefit from the contextual knowledge such teams bring to the studies. The commissioning agency also feels more confident about the quality of the final evaluation. Local research teams gain experience in and further exposure to impact evaluation approaches, analysis and reporting that the mentor offers as advice and guidance, which they can use in future contracts. They also have a greater probability of publishing their work. Finally, the experts have a chance to build new or strengthen existing relationships in the country, as well as have an additional publication opportunity.

The PWA model supports 3ie’s objective of strengthening the capacity of impact evaluation researchers in L&MICs, and responds to local needs for expertise cost-effectively and collaboratively. We look at PWA as an instrument for increasing the drive for impact evaluation in L&MICs, by strengthening the supply side of the local impact evaluation market. Additionally, PWA allows 3ie to reach a large number of countries at a fraction of the cost of a full impact evaluation.

The pilot PWA has generated interest amongst other 3ie L&MIC members. Spurred by these initial inquires, 3ie has recently launched a new request for qualifications for international experts. We look forward to collaboratively find different ways to enable the local environment for rigorous evidence and its use in policy. Stay tuned.