At a public forum on impact evaluation a couple of years ago, Arianna Legovini, head of the World Bank’s Development Impact Evaluation programme (DIME), declared that ‘dissemination is dead’. But her statement does not imply that we should stop the dissemination of impact evaluation findings for influencing policy. Quite the opposite: impact evaluation studies should be published in academic journals, be distilled into effectively formulated policy briefs, be disseminated at workshops and havesocial media fanfare. Legovini’s point was that this approach was not enough. For influencing policy, we need to move beyond dissemination and have strategies that include ongoing communication and active engagement with key actors. At a minimum, policymakers, programmemanagers and field staff, need to be involved in the evaluation from the design stage.
This view was echoed at the recent CGD-3ie event, Impact Evaluation: Can We Learn More? Better? Participants were asked to answer two questions: (1) What are the reasons that impact evaluation findings sometimes fail to influence policy, and (2) What actions can be taken to improve the likelihood that impact evaluation findings will influence policy?
We do know, as many pointed out, that many factors aside from research evidence always influence policymaking and change. The challenge is to make evidence available to policymakers at the time it is needed and in forms that policymakers can digest. Participants at the CGD-3ie event gave effectively summarised and designed policy briefs as an example of getting the product right for policymakers. And most participants’ answers to the second question about how to better ensure policy influence were related to getting the process right.
Securing the buy-in from the implementing agency is needed to carry out a study in the first place. Further, getting an implementing agency involved becomes particularly important if the study design has implications for implementation. As 3ie has found, and as many participants confirmed, getting the buy- in of the top officials in an implementing agency, whilst important, is not enough. Policy implementation depends on the people responsible for implementing it in the field, not the higher level policymakers. So, field staff need to be convinced of the worth of the evaluation and be substantively involved. If ignored, the evaluation itself, let alone changes based on eventual findings, are at risk of being undermined.
Buy-in also matters for the use of the findings from an impact evaluation. At 3ie, we ask whether a study is answering the questions that policymakers are asking? These questions are not just going to be about the impact of a development programme. They are going to be about the targeting, delivery mechanisms, and cost effectiveness of a programme. Policymaker interest is also always influenced by their political contexts. Studies will be ignored if they don’t address questions that policymakers want answered or are answers on which they cannot or will not act.
Policy influence is also about relationship building. Participants also pointed out that findings may also be ignored if policymakers feel that they don’t come from a trusted source. Policymakers, like most other people, often turn to friends and trusted advisors and colleagues for advice. Researchers and 3ie need to be those trusted advisors and friends who can offer impartial advice.
I saw this approach being implemented in practice while visiting the Beijing-based staff of Rural Education Action Program (REAP). One of the staff members told me his job was to go drinking with local government officials, as in China, this is what relationship building requires. REAP invests in relationships with these officials so that their studies can be carried out, and so that the findings of these studies are used by their key audience. And REAP has been successful at building relationships ‘at both ends’. At the CGD-3ie event, Scott Rozelle from REAP talked about their success at getting policy briefs signed off by the Premier of China and having recommended changes incorporated into state policy. The REAP staff clearly have a good understanding of the policymaking process and the inputs required to influence policy.
At 3ie, we consider it important for researchers to engage with policymakers throughout the life cycle of an impact evaluation. We encourage grantees to think about policy influence through policy influence plans, which they develop and implement with our technical support and through ongoing communication about them.
The search for answers to the question about how to influence policy and evidence for what works and why is a crucial part of 3ie’s current reflection and planning for its new 3-year strategy. We will be putting a draft strategy out for consultation in due course, but if you have any ideas in the meantime feel free to email them to us at strategy@3ieimpact.org.
To read my co-host William Savedoff’s reflections on this event, click here.