Highlights from Evidence 2018, Pretoria, South Africa

Invisibility and evidence: time’s up for evidence that doesn’t consider gendered drivers of inequality: the complacent, compliant or complicit test

This talk show style session engaged the audience on persistent challenges of and barriers to improving the gender and equity responsiveness of research and evaluation. The session began by the group interrogating whether or not they agreed that gender is a cross-cutting determinant in poverty.  Everyone agreed that the starting point for the discussion would be that gendered social norms always matter.  Not looking at them adequately in development harms women and girls and can harm men and boys.  
Adeline Sibanda (AfrEA) emphasised that sex-disaggregated data is the first necessary step, but that is not looking at gender, which requires incorporating gender analysis frameworks into methodologies. Researchers often cite cost considerations as a reason to omit gender analysis in their studies. She suggested that this can be mitigated by ensuring that cost of carrying out this analysis is incorporated in the budgets and fighting with donors to fund it. AfrEA’s experience shows that just providing training on gender analysis does not overcome the gender blind culture in evaluation.  She cautioned against setting up separate gender task forces or groups in evaluation, as they become marginalised.  

Mark Abrahams (Africa Evaluation Journal) shared insights on staff composition at the African Evaluation Journal. The composition of the board remains inequitable, with one-third more men than women. In terms of authors, there are more women than men. However, women prefer to work collaboratively in producing articles as opposed to men producing (AEJ) more single author articles. Men also outnumber registered women reviewers of the journal. He stated that the publishing sector has become complicit in perpetuating inequality. There is an urgent need to restructure institutions in order to ensure they are more equitable. 

Beryl Leach (3ie) chaired the session and encouraged the audience to reflect on entrenched power dynamics that dictate the research, evaluation and publishing sector. She emphasised the need to see gender and equity responsiveness as standards for ethical and quality research and evaluation. The session concluded with each participant sharing their key takeaways, as well as committing to one action they will take when they go home that is more gender responsive.  Actions included doing more reading, training staff on gender analysis, refusing to take any evaluation work that is not gender responsive and improving participation of women in the AEJ and looking at how to improve the review process. 

Strengthening evidence use in Uganda: what’s working and what’s next

Radhika Menon (3ie) chaired a talk-show style session on Uganda’s success in institutionalising evaluation capacity and using evidence to inform policymaking. The panellists included Josephine Watera (Uganda Parliament M&E Unit), Rhona Mijumbi-Deve (Africa Centre for Systematic Reviews and Knowledge Translation) and Timothy Lubanga (OPM, Uganda). They credited the Ugandan government for investing in creating a strong institutional infrastructure, which supports an effective enabling environment for evidence production and use. This includes identifying evidence champions in the government, creating the right frameworks and allocating sufficient resources to M&E. Panellists discussed 3ie’s role in promoting evidence-informed policymaking, especially in the public sector. 3ie-supported evaluations have led to programmatic changes and a new plan to restructure a number of public sector agencies. They also discussed challenges to using evidence, including a lack of access to high-quality data for policymakers, and a gap in communication between producers and users of evidence. 

Measuring evidence use: the value of contribution tracing

How can we measure evidence use in an unbiased and rigorous way? What tools and methodologies can we apply to increase the confidence we have in claims on evidence use from evaluations and reviews? This session, led by 3ie's Kanika Jha Kingra, Kirthi Rao and Radhika Menon drew on our experience with applying contribution tracing to measure evidence uptake and use from a portfolio of completed impact evaluations and systematic reviews. The presenters started by focusing on the myriad factors, apart from evidence, that influence decision-making. They underlined the importance of strengthening early and ongoing engagement to improve the relevance and usefulness of evidence being generated. These approaches have improved 3ie’s processes for monitoring and measuring evidence use, and improved the rigour of validating evidence use claims from 3ie-supported studies. Participants worked in groups to identify uptake or use from research that they have commissioned or been part of. They were encouraged to identify ways to monitor and measure changes with the tools and processes they already have. Participants queried how this methodology can be adapted to inform their own work, without it being resource intensive. The presenters emphasised the value of collecting relevant monitoring data from the very beginning to ensure that it is cost-effective and so that evidence use claims can be validated more robustly.  

Joint AEN-3ie workshops

Networking between Francophone and Anglophone Africa: bridging a post-colonial divide

Beryl chaired the first-eversession on barriers and challenges to networking between Francophone and Anglophone Africa. Deo, John Njovu (AfrEA), Olfa Soukri Cherif (Member of Parliament, Tunisia) and Siziwe Ngcwabe (AEN) were panellists. The most prevalent reason perpetuating the barriers was language—the majority English speakers in the region do not learn French. French speakers often have to learn English as adults due to work or career demands.  There was also a discussion on how institutional structures vary between these countries, such as differences in the way governments operate because of their colonial legacies. The panellists and audience members highlighted how institutions can work together to find cost-effective ways, such as using voluntary translators, simultaneous translations at conferences, and Swahili as a common language to create an environment to facilitate networking and collaboration. There was a strong call for organisations producing publications to offer them in both languages.  

Capacity development to use evidence in decision-making: working toward partnerships and building space to use approaches that work

Participants shared multiple perspectives to take forward the regional dialogue on developing capacity to produce and use evidence. Beryl, Adeline Sibanda (AfrEA), Caitlyn Blaser (CLEAR-AA), Henk Serfontein (DPME) and Emily Hayter (INASP) were the main speakers and Ruth Stewart (AEN) was the session facilitator. They highlighted the need to focus on not just developing capacity, but also co-producing evidence and the value of learning from all actors involved in this process. 

Speakers shared various examples from the region on their efforts to embed systems within legislatures and government, to improve understanding of and therefore use of evidence. There was an acknowledgement that it isn’t the lack of skills among decision makers, but the absence of opportunities to network, learn from peers and engage to access rigorous evidence. The audience also highlighted the need to move beyond just conducting workshops, to carrying out assessments and diagnostics to be responsive to the needs of decision makers.  

Other sessions

Understanding HIV research in Sub-Saharan Africa: analysis of 340 HIV studies from the Impact Evaluation Repository

As part of a panel on evidence in healthcare, 3ie’s Jorge Miranda presented key findings from an analysis of impact evaluations of HIV programming included in 3ie’s Impact Evaluation Repository. He used several charts and infographics to show the top ten countries producing these evaluations, top researchers in the field and the change in publications on HIV over time. He also explained the inclusion criteria and methodology, shared results of topic extraction, network analysis using authorship and citation data. 

Strengthening the supply side of evidence-informed policymaking

In this session, Aloysisus Ssennyonjo shared three interrelated approaches being used in the EU-funded, SPEED partnership to achieve universal health coverage in Uganda: multi-stakeholder engagements to understand problems and identify solutions, policy implementation monitoring through conducting a policy implementation barometer, and capacity building for faculty, students and partner institutions. 
Nasreen Jessani of the Centre for Evidence-based Healthcare introduced the centre’s two-day elective course on knowledge translation. This course is an excellent example of the importance of using adult learning principles for this type of capacity development. Mixing researchers and decision-makers in the workshops improved value to participants and having an experienced facilitator is critical for effective learning. Enhancing knowledge and skills ‘just in time’ rather than ‘just in case’ has demonstrated increased potential for immediate relevance, uptake and sustainability. 

Over the past decade, numerous interventions aimed at enhancing knowledge translation in L&MICs have been tested. However, most efforts have focused on policymaker perspectives. Violet Murunga (PhD student at the Liverpool School of Tropical Medicine) shared findings from her literature review in this area. There is a need for more studies and interventions focusing on understanding and addressing the barriers for researchers. 

Watch this video for an overview of 3ie's sessions at Evidence 2018