Leiby, K, Connor, A, Tsague, L, Sapele, C, Koanga, A, Kakaire, J and Wang, P, 2016. Measuring the impact of SMS-based interventions on uptake of voluntary medical male circumcision in Zambia, 3ie Impact Evaluation Report 49 New Delhi: International Initiative for Impact Evaluation (3ie)
Link to Source
Two five-month SMS campaigns designed to increase voluntary medical male circumcision (VMMC) uptake were piloted in the urban and peri-urban areas in Lusaka with Zambia U-Report, a free and confidential short message service (SMS) platform to access information on sexual health issues.
Zambia is one of 14 countries in east and southern Africa with high HIV prevalence and low circumcision rates prioritised to expand VMMC services and uptake. However, efforts to promote VMMC in Zambia have produced only modest results.
This evaluation sought to determine whether VMMC uptake, measured in terms of self-reported and verified uptake increased due to the impact of a conventional SMS campaign or a tailored SMS campaign.
Two campaigns were implemented on the Zambia U-Report platform aimed at getting subscribers to opt for VMMC. Each campaign variant consisted of sending 21 SMS messages to relevant users of the U-Report service over a five-month period. These campaigns sought to motivate U-Report users for VMMC uptake by providing information and encouraging broader interaction with SMS counsellors around VMMC and promoting participants to go for a procedure.
In the ‘conventional’ campaign, all participants received a comprehensive package of messages employing different behaviour-change tactics with information relevant to participants across a spectrum of intention levels and of VMMC awareness. In the ‘tailored’ campaign, participants received targeted messages employing the same behaviour change tactics but containing information relevant to their specific self-reported intention to go for VMMC.
Theory of change
The conventional campaign’s theory of change rested on the assumption that a standard comprehensive package of messages that covered an array of tactics and stages of change, in addition to confidential access to SMS counselling, would result in higher uptake of VMMC as compared to access to SMS counselling alone.
The tailored campaign’s theory of change rested on the notion that participants would be more likely to undergo VMMC if they received messages that are targeted at one’s intention level to go for VMMC. Therefore, a more targeted campaign may have a greater influence on intentions and uptake.
The study was designed as a three-armed randomised trial. The three arms included a control arm, a conventional arm and a tailored arm. The final sample size was limited by eligible participants and their responsiveness to the baseline screening survey. 2,312 participants met the original eligibility criteria and were enrolled in the study and 1,652 remained after attrition.
The intervention failed to produce statistically significant differences in uptake across study arms. However, self-reported outcomes — controlling for differences in consistency across reports indicated non-significant but positive associations between both treatment arm conditions and higher uptake. The findings suggest that neither campaign strategy was effective in generating demand but the campaigns were effective in increasing engagement with SMS counsellors on the U-Report platform.
Cost efficiency analysis
On average, U-Report costs $195,370 to operate annually, excluding the salary for the programmer. Six months of running U-Report plus additional U-Report engagement would be around $99,000. After accounting for costs of transport to the facility and the cost of the procedure itself, total costs without any VMMC campaign would be $237,588. A ‘conventional’ campaign scenario would cost $274,303, while a ‘tailored’ campaign scenario would cost $278,130. The incremental cost-effectiveness ratio (ICER) accounts for the incremental cost per additional circumcision in each campaign scenario. The ICER of a ‘tailored’ campaign ranges from $177.74 to $920.48 per additional circumcision, while that of a ‘conventional’ campaign ranges from $187.78 to $505.40 per additional circumcision.
Implications for implementers
Self-reported data was even more unreliable than expected and would be hard to interpret without some degree of verifiability. Furthermore, VMMC referral should be focused on facilities providing the most reliable services as many sites in the study area provided variable services over the course of the evaluation.
Implications for policy and practice
Based on the results, it is recommended that the U-Report platform not be used to scale-up either of the campaigns evaluated. However, SMS-tools provide easy routes to low-cost follow-up. Participants engaged more intensively through alternative interventions could continue to be promoted and encouraged to go for VMMC through SMS messages.
Implications for further research
The evaluation provided the first narrow examination of a focused, sustained campaign on Zambia U-Report. This research should serve as a starting point for evaluating the broader value and cost-effectiveness of the U-Report platform. Future research should examine the platform’s impact in other service areas and produce evidence that can be used to constantly improve its offerings.