On World AIDS Day 2015, we are marking the culmination of 3ie’s third thematic window, which funded seven pilot programmes and rapid impact evaluations looking for ways to increase the demand for voluntary medical male circumcision (VMMC). In late 2013, we awarded grants to project teams of implementers and researchers to pilot innovative programs for increasing VMMC demand and to conduct rapid impact evaluations of those programmes. All seven studies were completed in early 2015 and will be published soon in the Journal of Acquired Immune Deficiency Syndromes, which unfortunately means that the results of the individual studies are currently embargoed. But we can share what we learned from the overall grants programme – a highly focused thematic window to test pilot HIV prevention programmes in a rapid way can help to answer both decision-focused and knowledge-focused programme questions.
There were three important features of this grants window. The first is that we designed it from the beginning to support rapid impact evaluations. Our funder, the Bill & Melinda Gates Foundation, wanted the window to produce evidence that can be used to inform programming as soon as possible. The research question – what innovations can increase the demand for VMMC – lends itself to rapid impact evaluation, because the primary outcome, circumcision, is a discrete behavioral response that only needs to happen once and is expected to occur relatively soon after the interventions. In addition, we anticipated that many project teams would be able to use randomised controlled trial designs and be able to randomise treatment assignment at the individual level, both of which allow for smaller sample sizes and thus quicker studies. Based on these factors, we set the funding ceiling and the timelines to require rapid impact evaluations.
The second feature was the specificity of the theme. Although 3ie has many thematic windows, the breadth of the themes for the windows varies widely. In the case of the VMMC window, our funder wanted to know the answer to a very specific question: what innovations can help increase the demand for VMMC? In our scoping report, which informed the request for proposals, we laid out some barriers and facilitators affecting the demand for VMMC that should be considered when proposing innovations. The funded studies included a range of interventions that addressed barriers and facilitators both from a rational choice standpoint (i.e. what are my costs and benefits) and from an emotional choice standpoint (i.e. what might influence me to do this thing I do not feel like doing), as discussed in our blog post last year.
There were similarities across some VMMC interventions. For example, two interventions used a mobile phone lottery (or raffle) as an incentive. Two interventions relied on intimate partners either directly (sending messages about VMMC through women attending ante-natal clinics) or indirectly (using messaging for men about the preferences of intimate partners for circumcised men). Two interventions relied on some kind of peer effects, one through a sports-based program and one through peer referrals. Two interventions used a financial incentive intended to compensate men for some of the costs incurred. You can learn more about one of the interventions and its impact evaluation in this video.
The differences across the seven interventions allow us to examine many possible VMMC programmes, each of which could be a candidate to be scaled up in the same or very similar context. In this way, each study is what Shah, et al. (2015) call a decision-focused evaluation, an evaluation “driven by implementer demand, tailored to implementer needs and constraints, and embedded within implementer structures”. However, the narrow focus of the theme, the similarities between studies, and the grounding on a common framework of barriers and facilitators allow us to examine some of the underlying questions – the answers to which could inform the design of other similar programmes. Taken together, the seven studies then also play the role of what Shah, et al. (2015) call knowledge-focused evaluations, “those primarily designed to build global knowledge about development interventions and theory”. The restriction of the window to a set of countries in Sub-Saharan Africa also narrows the focus of the window in a way that increases the comparability of results across these studies for this region.
The third important feature was the coordination. For this funding window, we held 3ie’s first post-award workshop. This event brought together the study teams for two days to refine their intervention and impact evaluation designs based on the feedback from other teams, but also with a view to harmonising the aspects of the projects that would allow for better cross-project comparisons upon completion. Awardees discussed the theories behind their interventions, their sampling and randomisation methods and the measurement of their primary and secondary outcomes. In some cases, the teams continued their communication with each other throughout the implementation of their projects. As the timing of the studies was pre-determined, the studies were all completed at about the same time. This makes it possible for us to synthesise and disseminate the results so that both the decision-focused findings and the knowledge-focused findings can be made available at the same time to complement evidence uptake activities.
The primary challenge for us in implementing a funding window with these features was the long delays many grantees faced in getting their Institutional Review Board (IRB) approvals. These delays slowed the timing of the rapid impact evaluations and interfered with the coordination of completion. We have encountered the same challenge with our other rapid impact evaluation windows; our grantees miss their early project milestones due to delays in IRB approvals.
The results of the VMMC funding window are enlightening. While we cannot share the findings from the individual studies, we can mention some conclusions from our forthcoming synthesis paper “Identifying interventions to increase the uptake of male circumcision in Southern and Eastern Africa: evidence from seven impact evaluation studies”, which is being presented as a poster on World AIDS Day 2015 at the International Conference on AIDS and STIs in Africa:
- The impact that peers and intimate partners can have in encouraging circumcision appears to be limited. One exception may be sports-based interventions. (See here for results from a precursor study to that funded under our window.)
- There is evidence that how information about male circumcision is framed and presented can make a difference to the impact it can have.
- Material incentives based on chance do not seem to have an impact, but financial incentives framed as compensation for costs incurred due to circumcision can have a positive impact on uptake.
3ie is not alone in running highly focused grants windows for impact evaluations with a view to increased comparability moving towards external validity. Evidence in Governance and Politics (EGAP), a 3ie partner organisation, has developed an innovative approach called a metaketa. This approach focuses even more on a single or small set of knowledge-focused evaluation questions by having the research teams work together in the design of the pilot interventions as well as on the design of the experimental evaluations. The principal investigators for the grant programme also plan in advance how they will synthesise the evidence across the individual studies to look for more generalisable answers. 3ie’s VMMC funding (as well as our other rapid impact evaluation windows) and EGAP’s metaketa are two similar approaches to funding a group of impact evaluations with a view to maximising external validity.