When 3ie set up the Innovations in Increasing Immunisation Evidence Programme, we realised early on that we had to walk the talk. The evidence programme funded formative and impact evaluations of community-based approaches for boosting coverage in countries with low or stagnating immunisation rates. But community members and local decision makers had not been figuring prominently in our approach to how our grantees engage with stakeholders. This programme therefore became a catalyst for us to improve our approach.

Three years into this evidence programme, we are able to share some lessons on how we perceive powerholders and decision-making, community members, and the crucial role of stakeholder engagement at every stage of the evaluation cycle.

Early engagement with local stakeholders improves intervention design

For 3ie-supported formative evaluation teams, early engagement with local stakeholders has been particularly important in designing interventions that are context-appropriate. Mapping power dynamics among stakeholders has also been crucial in developing a contextualised theory of change.

An intervention extending the Ethiopian government’s health development army network needed to reflect the specific characteristics of pastoral communities. They are dispersed across large, remote geographic areas and their constant mobility and low demand for health services makes it particularly challenging to deliver health services.

The initial intervention design focussed on training women within pastoral communities. However, discussions with district administrators and community members made it clear that it was important to include the Aba Ollas or male traditional leaders to mobilise female network members. Pastoral communities are organised as clusters of families, usually residing in close proximity to and governed by a common traditional leader. Not only do pastoral families migrate outside their village, they also move within their village from one settlement area to another. Aba Ollas were therefore important for ensuring the sustainability of the network organisation as families migrated.

Strategic and ongoing engagement improves evaluation implementation

Securing implementer buy-in is crucial in ensuring that the intervention is implemented as the evaluation design requires.

However, securing implementation support and building evaluation buy-in is no small task when the government is the implementer. Securing permissions and convincing government officials at multiple levels in the hierarchy about adherence to the evaluation design is tough and time-consuming. The high attrition and transfers of health department officials also doesn’t make it any easier.

To address these issues, several 3ie-supported study teams have invested considerable time and effort in building relationships with health department officials at the local, regional and national levels. Most teams have been strategic in their engagement approach and have gone beyond meetings and presentations. An impact evaluation team in India, for example, has institutionalised its partnership with a state government early on by signing a memorandum of understanding. The team roped in the chief minister of the state to launch the project – an event that was covered by the local media. They also had a state government health department official join the study team. All of these measures signalled to the district administration that the project had high-level government support. Needless to say, district staff became highly supportive of project-related activities. What could have been a long and cumbersome process became more straightforward and easier to implement.

A few study teams have dedicated personnel for building and maintaining partnerships with the government. These teams provide useful health monitoring data to local health department officials on a regular basis. This has not just piqued policymaker interest but has also gradually built a solid commitment to using evidence for decision-making.

Some study teams have had other ideas for engaging with their implementing partners and finding champions for promoting evidence use. While a formative evaluation team in Nigeria worked with a senior policymaker to jointly produce a policy brief, a study team in Uganda set up an advisory group with key stakeholders. The advisory group, which included government officials and health sector experts, provided guidance to the research team and became study champions.

Stakeholder engagement matters for ethical research and evidence uptake

Evaluators have substantively engaged with key stakeholders, including community members, through the implementation phase, as well as after study completion. 3ie expects part of this engagement to be final dissemination workshops with decision makers. Increasingly, and in part because of the focus of this evidence programme, 3ie is stressing community stakeholder workshops during the study, as well as in communicating the evaluation findings and implications.

For instance, a study team working on a formative evaluation of a collaborative community checklist intervention in Myanmar, engaged extensively with female health volunteers, village leaders, auxiliary midwives, midwives, village health committees, checklist assistants, pregnant women, mothers and caregivers throughout the study period.

For the study's results validation workshop, they used participatory exercises (such as ten seed technique and role play) for sharing the evaluation findings and collecting feedback on ways to improve the use of the checklist. The team also shared the study findings directly with community stakeholders. Their feedback helped them identify implications of the study that informed the final evaluation report.

The intensive work that has gone into stakeholder engagement with both local and national actors is paying off as evidence is being used to inform programming. In Nigeria, a formative evaluation of the vaccine indicator and reminder bands for infants showed that traditional and religious leaders helped address community apprehensions about the band and encouraged caregivers to get their infants immunised.

Engaging traditional birth attendants also contributed to an improvement in routine immunisation uptake. These findings have informed the Kebbi state primary healthcare agency’s community engagement strategy and action plan. The agency has decided to involve community leaders to improve referring newborns to health facilities and in tracking defaulters. As part of the plan, Kebbi state has established a Juma’at Mosque Committee to involve religious leaders in raising awareness on routine immunisation during their weekly sermons. They also plan to use traditional birth attendants to promote health education on routine immunisation among mothers.

Walking the talk

3ie works with our grantees to ensure that they understand the contexts in the complex interventions they are evaluating, including mapping and engaging with all of the key stakeholders involved in the intervention. 3ie needed to ensure that our guidance adequately focused on community stakeholders and a wide range of decision makers affecting implementation. Because of this programme, we adapted our guidance and have monitored it proactively with these complexities in mind. By requiring and promoting the value of ongoing engagement, we have helped our teams ensure stakeholder buy-in for smooth implementation. Moreover, we are stressing the ethical, as well as practical value of engaging with community actors, especially programme participants. This approach has helped increase the likelihood that evaluations have community, as well as government support, and that findings are available and understood by all key stakeholders, not just powerholders.

Avantika Bagai provided valuable inputs to earlier drafts and Beryl Leach helped shape and finalise this version.