Highlights from 3ie London Evidence Week Conference, November 2018

3ie’s Emmanuel Jimenez delivered the opening remarks, highlighting some of 3ie’s achievements, including supporting the generation of evidence in challenging contexts and sectors; how 3ie-funded evidence has been useful for decision makers; and, how our members have helped form a community of like-minded champions of evidence. He also asserted that 3ie’s innovations, including evidence gap maps and repositories have contributed to strengthening the enabling environment for evidence-use. He outlined three key challenges: ensuring that rigorous evidence is cost-effective and timely; ensuring that the number and capacity of development country researchers continues to grow; and, continuing to be an insistent voice for evidence-informed decision-making.

Ruth Levine (Hewlett Foundation and 3ie board chair) moderated a discussion on the role of donors and evaluators in development. She credited 3ie for playing a part in strengthening the body of rigorous evidence and listed key challenges to ensuring evidence use. 

Alastair Ager (DFID) emphasised that there needs to be a focus on what works to sustain development funding. While this might seem like a challenge, he pitched it as an opportunity for evaluators to show how quality programming can produce quality evidence. Adeline Sibanda (African Evaluation Association) said that evaluation is not neutral; it is about power relations. She called on the community to raise questions about who commissions, conducts and implements evaluations in developing countries. Howard White (Campbell Collaboration and 3ie founding executive director) spoke about the fourth evidence revolution, knowledge brokering and translation, and how organisations like 3ie need to ensure that evidence produced is easy to access and use. He addressed some of the challenges raised by other panellists, giving examples of how portals, EGMs, and other tools helped make evidence more useful and accessible to decision makers.

Moderator: Ruth Levine, director, global development and population program, the William and Flora Hewlett Foundation, and chair, 3ie board of commissioners
 
Speakers: Alastiar Ager, deputy chief scientific adviser, DFID; Adeline Sibanda, president, African Evaluation Association; Howard White, CEO, Campbell Collaboration

The discussion focused on various challenges in ensuring evidence-informed decision-making by governments. Andrew Osei Asibey (Ghana Ministry of Monitoring and Evaluation) shared the story of how a change in leadership helped the country institutionalise monitoring and evaluation, including setting up a dedicated ministry. Timothy Lubanga (Uganda Office of Prime Minister) explained how a strong and invested political leadership, and a dedicated team of technocrats helps ensure the use of evaluation evidence. Esmie Kainja (Malawi Ministry of Gender, Disability and Social Welfare) explained how the government’s interest in commissioning and conducting evaluations depends on several factors, including political will, cost-effectiveness of evaluations and how evaluations align with government demands. She called on donors to engage more Malawians as experts, not just as data collectors. She also shared that positive findings from a 3ie-supported evaluation of a cash transfer programme caught the attention of cabinet ministers interested in the potential of evaluation evidence to inform programmes. While all panellists faced similar challenges, such as resource constraints and low interest in evaluation, they agreed that forming strong partnerships is central to ensuring evidence use.

Panellists discussed the state of social protection evidence and ways to ensure uptake. Sudanshu Handa (University of North Carolina at Chapel Hill) shared key ingredients for uptake, including building trust between evaluators and programme managers, providing timely findings, and partnering with national researchers. Esmie Kainja (Malawi Ministry of Gender, Disability and Social Welfare) outlined challenges, such as a lack of political will and readiness of the country and leadership to use evidence, especially to scale up the programme. Orazio Attanasio (University College London) highlighted the need to look at behaviour change as a component of social protection programmes. He also mentioned ongoing challenges, such as aligning interests of policymakers and evaluators, especially in changing political landscapes, how to scale up programmes and the risk of making blind generalisations from impact evaluations. Carola Alvarez (Inter-American Development Bank and 3ie board commissioner) talked about how social protection programmes have evolved. She emphasised the need to understand different perspectives and operating environments of researchers and evaluators and finding common ground to deliver a useful programme. Marie Gaarder (3ie) chaired the session and provided an overview of the findings from 3ie’s DFID-funded social protection programme.

Chair: Marie Gaarder, Director of the Evaluation Office and Global Director for Innovation and Country Engagement, 3ie

Speakers: Carola Alvarez, Chief of Development Effectiveness and Strategy, Inter-American Development Bank and 3ie Board Commissioner ; Esmie Kainja, Principal Secretary for Gender in the Ministry of Gender, Disability, and Social Welfare, Malawi ; Orazio Attanasio, Professor of Economics, University College London and Research Director, Institute for Fiscal Studies ; Sudhanshu Handa, Professor of Public Policy, The University of North Carolina at Chapel Hill

Panellists discussed global learning crisis and what can be done to address it. George Werner (former Liberian education minister) narrated how securing a grant to study the state of education after the Ebola outbreak provided evidence that helped inform the country’s education strategy. Ju-Ho Lee (former South Korean education minister and 3ie board commissioner) talked about the struggle of finding evidence on what works in new approaches to education, such as using artificial intelligence, social learning through digital platforms and others. Girindre Beeharry (Gates Foundation) referenced 3ie’s education effectiveness review, saying while synthesis is useful, there was a need to have context-specific evidence on what works and where. Panellists agreed that timeliness of evidence was a key challenge. It is important to invest more in context-specific, rigorous evidence that is useful for decision-making. They also discussed the increased use of technology to improve learning and the mixed evidence on its effectiveness. Sally Gear (DFID) emphasised that technology is a tool, not an end and it was important to make decisions on using technology to improve teaching and learning. One of the challenges, is to build capacity of partners to use technology in L&MICs. She also challenged the term learning crisis. Whether there is one depends on where you live and your income

Chair: Elizabeth M King, senior fellow, Brookings Institute and 3ie board member

Speakers: George Werner, former education minister for Liberia; Girindre Beeharry, director, Global Education Learning Strategy, Gates Foundation, Ju-Ho Lee, professor and former education minister, South Korea, and 3ie board commissioner; Sally Gear, senior education advisor, DFID

Institutionalising evaluations at the national level: the power of storytelling

Gonzalo Hernández Licona (CONEVAL and former 3ie board commissioner) presented challenges to setting up national monitoring and evaluation units and how evaluators have to work within political systems to create impact. He stressed that evidence needs to be narrated in a way that is accessible and easily understood by all stakeholders. People respond to stories they can understand and identify with, so evidence needs to be part of the story. He discussed the idea of institutionalising the everyday use of evidence in a systematic way. He also spoke about his own experiences setting up 3ie and being part of the CONEVAL, and his efforts at explaining the technicalities of evaluation to people in politics.

Democratising evidence: innovations in access and measurement

Rachel Glennerster (DFID) chaired this World Café session, where participants were encouraged to engage with speakers on four respective topics: implementation research, evaluating humanitarian interventions, evidence databases and platforms, and lessons from an institutional model of evidence-based decision-making. Each speaker shared a key takeaway from their group’s discussion.

• Implementation research
  Anna Heard (3ie) led an engaging discussion on implementation research. She explained how implementation research, including formative and process evaluations, can help inform programme design during implementation, as opposed to impact evaluations. Participants also discussed how consistent reporting of implementation processes, feedback from beneficiaries in shaping interventions, as well as technology can help improve programme design.
• Evaluating humanitarian assistance programmes
  Tara Kaul (3ie) and Ernest Guevara (Valid International) discussed innovative approaches to challenges in conducting humanitarian impact evaluations. Participants outlined measurement and ethical considerations as some of the key challenges they face when evaluating programmes. They also discussed challenges in ascertaining external validity of results in the humanitarian context.
• Lessons from an institutional model of evidence-based decision-making
  Shobhini Mukerji (J-PAL South Asia) encouraged participants to share various challenges in engaging with decision makers. Lack of time and availability, complex communication processes, and lack of accessibility to government officials were some of the barriers highlighted. Using examples from J-PAL’s work, Shobhini talked about different ways in which evaluators can collaborate with governments, including identifying champions, building capacity, using and sharing data analytics, and facilitating the creation of advisory committees.

• Evidence databases and platforms
  Birte Snilstveit (3ie) introduced participants to 3ie’s repositories and elaborated on our plan to upgrade them to ensure that relevant evidence is easily accessible to researchers, project managers and policymakers, particularly in L&MICs. She invited ideas from participants on how to visualise evidence, making categories more informative and what kind of filters are useful to get desired results

Chair: Rachel Glennerster, chief economist, DFID 
Speakers: Anna Heard, senior evaluation specialist, 3ie; Birte Snilstveit, senior evaluation specialist, 3ie; Ernest Guevara, head of measures, Valid International; Shobhini Mukerji, executive director, J-PAL South Asia; Tara Kaul, evaluation specialist, 3ie

Beryl Leach (3ie) led an engaging and interactive discussion on research transparency, accountability, replicability and reusability. Arnaud Vaganay, (National Centre for Social Research) emphasised reusability of evidence and data, along with the need to identify third-party users to make it useful and relevant for them. 
Neeta Goel (3ie) shared some lessons from 3ie’s ongoing replication work, including the challenges faced with de-identification of data, particularly qualitative data, and challenges with producing the same results that are presented in final evaluation reports. Adeline Sibanda (African Evaluation Association) highlighted challenges like inadequate infrastructure and poor internet access in some African countries. Kristin Komives (ISEAL Alliance) spoke about the challenge in working with corporations who fund and conduct interesting research, but are reluctant to share it, leading to duplication and spending unnecessary resources. She also mentioned ISEAL was developing a repository of research standards and an online portal that systematically maps of evidence and codes research in a way that lets users extract relevant information and visualise results. During the question and answer, participants brought up the need to address language barriers to democratise access to evidence. Participants also echoed the need to improve readability of data and its translation for evidence users as a key priority.

Chair: Beryl Leach, director and head, policy and advocacy, 3ie
Panellists: Adeline Sibanda, president, African Evaluation Association; Arnaud Vaganay, research director at National Centre for Social Research, and director of Meta-Lab; Neeta Goel, senior evaluation specialist, 3ie; Kristin Komives, director, impacts, ISEAL Alliance

Ruth Levine and Emmanuel Jimenez shared some of their key takeaways from conference and outlined the upcoming challenges for 3ie, and the evaluation community’s over the next decade.

Ruth Levine referenced American political scientist Robert Putnam’s study of social capital to explain how best to build trust among people who are producing and using impact evaluation findings. Bonding is when we are operating in a like-minded group, creating a shared vocabulary, shared framework that we all understand and reinforcing the same set of ideas. Bridging is when we extend ourselves to others not just like ourselves, seeking points of connection around a shared goal in pursuit of this connection, we are willing to go the extra mile to speak in a language familiar to others and listen to other points of view. While 3ie’s first decade was spent in closing the evaluation evidence gap, the next challenge is bridging the chasm between the generation and use of evidence. She recommended that more time be spent on building bridges with people who use other methods and have different perspectives, rather than bonding mainly or exclusively with people in the impact evaluation community.

Emmanuel Jimenez summarised three key takeaways including the need to recognise possible trade-offs in commissioning and conducting evaluations; the need to define 3ie’s role as a knowledge broker; and ensure evidence is accessible, shareable, digestible and most importantly, appetising.