On 1 February 2020, Marie Gaarder took over as 3ie’s new executive director. She sat down for a quick interview on her new role, challenges and opportunities in the evaluation sector, the persistent gaps in evidence, and what 3ie has planned in the coming year.

Q1. What are you looking forward to in your role as executive director of 3ie?

I look forward to continuing to grow 3ie as an innovator and amplifier organization, which pushes the frontiers of evidence for sustainable social progress. With 3ie’s amazing team, I am sure we will find exciting new opportunities in the coming years to build strong partnerships and help improve more lives. We will strive to do this at numerous levels:

1. We will support and strengthen the capacity to produce, translate and use evidence;
2. We will help fill evidence gaps and make evidence more accessible; and
3. We will engage with researchers to improve the quality and relevance of what we are doing, for example, by tackling issues of cost-effectiveness of interventions, generalizability, ethics, understanding of context and root causes.

This will require us to be innovative and spur innovation by others. For example, we will continue to innovate in mapping, where 3ie is recognized as an international leader after having invented the evidence gap maps. Other areas include evidence translation, brokerage and the use of evidence. We will also need to think about the carbon footprint of research, and innovate tools and methods (e.g. big data, virtual field monitoring tools; virtual conferences) to decrease it. Our organization’s unique setup across three continents and our international board, puts us in an exciting position to play a role across the developing world. I have seen 3ie grow from the beginning, and I am thrilled to be able to lead us to do even more.

Q2. What are some of the main challenges and opportunities facing the evaluation community today and what role will 3ie play in addressing these?

3ie has always risen to emerging needs and challenges. After all, 3ie was created in 2008 as a response to the lack of evidence described in the 2006 report of the Evaluation Gap Working Group (When Will We Ever Learn? Improving Lives Through Impact Evaluation). Some other examples: In response to a challenge by IFPRI to present everything we knew about interventions in agriculture and their effects on nutritional outcomes, 3ie developed the first ever evidence gap map in 2010. When we found that some interventions were not ready for impact evaluations, or were taken to scale although they should not have been, 3ie developed phased evaluation programs that included formative and process evaluations.

Now to the new challenges.

First, I think there is still a tendency in the development evaluation or research field to draw up caricature versions of a method or approach in order to then prove that another method or approach is better. For example, the 2019 Nobel Prize in economics has seen a fair amount of negative reactions. While we have to recognize limitations of RCTs, I have argued elsewhere how the clarity in approach brought by RCT’s may have contributed to the increased credibility and funding available to development evaluation. 3ie has not been immune to this kind of ‘strawperson’ attacks. A number of people still think we are only about RCTs when in fact we never were. Since its very inception, 3ie has emphasized a theory-driven, mixed-methods approach, as well as the importance of synthesizing across a body of evidence before making policy recommendations with broader implications (Gaarder and White, 2009). I think there is an opportunity for 3ie to be a bridge-builder – between research and evaluation, between impact evaluation and more traditional program evaluation, and between relevant and rapid evidence production and quality evaluation.

Second, there exists an accountability vacuum: when interventions do not have the intended outcomes, nobody is really held to account (Gaarder and Bartsch, 2015). Multilateral loans get repaid regardless, the aid is still counted as ODA even if it did not aid, and the developing country governments we work with are often not accountable to the populations they are supposed to serve. If we want that evidence-informed adaptation becomes embedded in all development processes, we need to find ways to increase this accountability. The question is, which funder or government is willing to fund work to make itself more accountable? We at 3ie believe such funders exist, and hope to develop partnerships and innovative solutions to this accountability conundrum.

Third, I think we still have a way to go to make development evaluation locally relevant, and thereby ensure its usefulness. Relevant evaluations need to be ‘locally sourced’. This has the potential to make them better informed by local contexts, and local evaluators are more likely to be effective evidence-brokers or have the networks that can play this role. This is the right thing to do also from an ethical and a sustainability point of view (think of the carbon footprint). We are not quite there yet, capacity strengthening and mentoring initiatives are still needed. There are also barriers in place for L&MIC evaluators and researchers that need to be lifted: for example, their access to networks, ethics reviews and IRB approvals, data hosting services, replication training, registries, journals, project and financial management training courses are all limited. I think 3ie can help provide better access to research-related support services for L&MIC researchers. We are planning to make this a key aspect of our new strategy.

Finally, we are at a different place than we were 10-12 years ago. Thousands of development effectiveness studies have been implemented, and hundreds of systematic reviews have been produced. 3ie has been an important contributor to this achievement. Most of these studies, which now constitute a large public good, and will be available in 3ie’s newly revamped searchable repositories. What amazes me is how little use is being made of this treasure-trove, even by organizations that know of its existence. At the start of this new decade, 3ie therefore launched a new campaign, aptly named the 2020 hindsight campaign in which we are challenging ourselves to tell stories in layperson terms about what has been found to work and for whom. In parallel, our medium-term goal is to work with partners to see how best to translate and contextualize the evidence so that it can support and inform decision-making processes locally and internationally in a timely way.

Q3. What are the key areas where evidence gaps persist?

Since the invention of the first evidence gap map (EGM) in 2010-11, we have pioneered further advancements, including developing our interactive online map platform. EGMs are now one of our most popular products as they are visually intuitive and can improve decision-making around where to make investments in producing more evidence or synthesising existing evidence.

While the evidence is increasing, there are still some sectors and topics that remain largely under-researched, including climate change, humanitarian assistance and fragile contexts, governance and institutional reforms, and infrastructure to name but some. There are also cross-cutting gaps that remain in what type of outcomes have been looked at: environmental, climate change and other unintended impacts are largely ignored, gender- an equity-related findings are scant, and longer-term social and welfare outcomes are underrepresented compared to the intermediate outcomes. There are also huge regional and country-level differences in terms of where the evaluations have taken place, with the Middle-East and North Africa (MENA) being the most evidence-poor region.

Q4. What can we expect to see from 3ie in the coming months?

• We are in the process of preparing our new strategy fit for the new decade. It is currently in its consultative phase but expect to see many of the ideas already mentioned above reflected in the new strategy.
• As our 2020 hindsight campaign continues, expect to read many more blogs from us than you have in previous years.
• We are also about to launch the new incarnation of our evidence repositories, the 3ie Development Evidence Portal. This is part of a large, ongoing initiative to upgrade the Impact Evaluation and Systematic Review Repositories to make them more comprehensive and more user-friendly, as part of our overall efforts to reduce barriers to evidence use.
• 3ie will soon bring senior fellows on board. The fellowship program provides experienced researchers an opportunity to work with 3ie’s global team on a range of new and ongoing projects in their area of expertise.
• As many of you know, 3ie started the tradition of holding evidence weeks, usually in conjunction with our Board meeting and members’ conference, and this year is no exception. The public conference and Howard White lecture will take place in London on the 23 April and will have cost-analysis in development effectiveness research as a core theme.
• Some of our new programs in Asia will start to take off. We will be launching a request for proposals in relation to women's economic empowerment in India (supported by the Gates Foundation), and we will be preparing a new phase of close collaboration with the Philippines government, evaluating more government programmes (supported by Australia’s DFAT). The government’s commitment to strengthening the culture of evaluation is really encouraging. We have been fortunate to contribute to similar positive trends in Uganda, another long-time partner of 3ie.

Q5. 3ie has been advocating for enhanced research transparency and encouraging stakeholders to join the movement. What is your message to the international development community?

My overall message is that the end-beneficiary of development research and the ultimate owner of the data is the population that is supposed to benefit from development policies and interventions. It is not the funder, nor the researcher, nor is it the implementing agency. If we use this as our ‘north’ in terms of how, with whom, and when we share data and research findings, we should be on the right track.

From the beginning, 3ie has been committed to research transparency, requiring researchers to make their reports publicly available and sharing their data and codes. In 2018 we launched our official Research Transparency Policy, since then we have carried out push button replication on all the 3ie supported impact evaluations, and we will aim to continue to be at the cutting-age of research transparency.