Measuring the impact of SMS-based interventions on uptake of voluntary medical male circumcision in Zambia

Publication Details

 Leiby, K, Connor, A, Tsague, L, Sapele, C, Koanga, A, Kakaire, J and Wang, P, 2016. Measuring the impact of SMS-based interventions on uptake of voluntary medical male circumcision in Zambia, 3ie Impact Evaluation Report 49 New Delhi: International Initiative for Impact Evaluation (3ie)


Link to Source
Author
Kevin Leiby, Alison Connor, Landry Tsague, Crispin Sapele, Albert Kaonga, Joshua Kakaire, Paul Wang
Country
Zambia
Region
Sub-Saharan Africa (includes East and West Africa)
Sector
Health Nutrition and Population, Social Protection
Subsector
Nutrition
Gender analysis
No
Equity Focus
None specified
Evaluation design
Randomised Control Trials (RCT)
Status
3ie Series Report
3ie Funding Window
Voluntary Medical Male Circumcision Thematic Window

Synopsis

Two five-month SMS campaigns designed to increase voluntary medical male circumcision (VMMC) uptake were piloted in the urban and peri-urban areas in Lusaka with Zambia U-Report, a free and confidential short message service (SMS) platform to access information on sexual health issues.

Context

Zambia is one of 14 countries in east and southern Africa with high HIV prevalence and low circumcision rates prioritised to expand VMMC services and uptake. However, efforts to promote VMMC in Zambia have produced only modest results.

Research questions

This evaluation sought to determine whether VMMC uptake, measured in terms of self-reported and verified uptake increased due to the impact of a conventional SMS campaign or a tailored SMS campaign.

Methodology

Intervention design

Two campaigns were implemented on the Zambia U-Report platform aimed at getting subscribers to opt for VMMC. Each campaign variant consisted of sending 21 SMS messages to relevant users of the U-Report service over a five-month period. These campaigns sought to motivate U-Report users for VMMC uptake by providing information and encouraging broader interaction with SMS counsellors around VMMC and promoting participants to go for a procedure.

In the ‘conventional’ campaign, all participants received a comprehensive package of messages employing different behaviour-change tactics with information relevant to participants across a spectrum of intention levels and of VMMC awareness. In the ‘tailored’ campaign, participants received targeted messages employing the same behaviour change tactics but containing information relevant to their specific self-reported intention to go for VMMC.

Theory of change

The conventional campaign’s theory of change rested on the assumption that a standard comprehensive package of messages that covered an array of tactics and stages of change, in addition to confidential access to SMS counselling, would result in higher uptake of VMMC as compared to access to SMS counselling alone.

The tailored campaign’s theory of change rested on the notion that participants would be more likely to undergo VMMC if they received messages that are targeted at one’s intention level to go for VMMC. Therefore, a more targeted campaign may have a greater influence on intentions and uptake.  

Evaluation design

The study was designed as a three-armed randomised trial.  The three arms included a control arm, a conventional arm and a tailored arm. The final sample size was limited by eligible participants and their responsiveness to the baseline screening survey. 2,312 participants met the original eligibility criteria and were enrolled in the study and 1,652 remained after attrition.  

Main findings

The intervention failed to produce statistically significant differences in uptake across study arms. However, self-reported outcomes — controlling for differences in consistency across reports indicated non-significant but positive associations between both treatment arm conditions and higher uptake. The findings suggest that neither campaign strategy was effective in generating demand but the campaigns were effective in increasing engagement with SMS counsellors on the U-Report platform.

Cost efficiency analysis

On average, U-Report costs $195,370 to operate annually, excluding the salary for the programmer. Six months of running U-Report plus additional U-Report engagement would be around $99,000. After accounting for costs of transport to the facility and the cost of the procedure itself, total costs without any VMMC campaign would be $237,588. A ‘conventional’ campaign scenario would cost $274,303, while a ‘tailored’ campaign scenario would cost $278,130. The incremental cost-effectiveness ratio (ICER) accounts for the incremental cost per additional circumcision in each campaign scenario. The ICER of a ‘tailored’ campaign ranges from $177.74 to $920.48 per additional circumcision, while that of a ‘conventional’ campaign ranges from $187.78 to $505.40 per additional circumcision.

Implications for implementers

Self-reported data was even more unreliable than expected and would be hard to interpret without some degree of verifiability. Furthermore, VMMC referral should be focused on facilities providing the most reliable services as many sites in the study area provided variable services over the course of the evaluation.

Implications for policy and practice

Based on the results, it is recommended that the U-Report platform not be used to scale-up either of the campaigns evaluated. However, SMS-tools provide easy routes to low-cost follow-up. Participants engaged more intensively through alternative interventions could continue to be promoted and encouraged to go for VMMC through SMS messages.

Implications for further research

The evaluation provided the first narrow examination of a focused, sustained campaign on Zambia U-Report. This research should serve as a starting point for evaluating the broader value and cost-effectiveness of the U-Report platform. Future research should examine the platform’s impact in other service areas and produce evidence that can be used to constantly improve its offerings.  

Additional publications

Scroll to Top