The Original Study
The original paper by Hidrobo and colleagues analyzes a cluster-randomized controlled trial (cRCT) comparing the effectiveness of different modes of food assistance (cash, food, and vouchers) with a control mode (i.e. no assistance). The statistical analysis was done using an ANCOVA model and included several food-related outcome measures (food consumption, several indices of food security, and diet). Robustness checks are made and the effect estimates, adjusted for co-variates, are reported. Finally, the costs associated with each mode of assistance are calculated and cost-effectiveness measures presented.
All three modes investigated by the paper (cash, in-kind, vouchers) are under policy discussion or even used in several countries and it is widely agreed that all three modalities work in absolute terms, i.e. increasing calorie intake. Many existing studies on this topic are either observational or do compare one of the three interventions against a non-intervention only. However, comparing these three modes directly against each other is of vital importance for policy makers as these three modes are thought to have distinct advantages and/or disadvantages in terms of efficacy, public acceptance, and cost. For example, direct cash transfers are seen by some as a more efficient way of providing help as the disbursement costs are relatively low and allow the recipient to buy goods that truly increase the recipient’s utility. On the other hand, recipients may spend cash not necessarily solely on beneficial goods (e.g. for tobacco instead). In-kind transfers, for example, may have more beneficial health effects if the quality and quantity of food provided exceeds that bought from a cash transfer. Yet in-kind transfers are costly to administer and reduce the agency of the recipients. Hence, some argue that vouchers occupy a middle ground between these two modes of assistance. The paper by Hidrobo and colleagues is one of very few studies that report the results of a head-to-head comparison of all three modes.
The Replication
The first objective of this replication research is to conduct a pure replication of the study. That is, establish whether the published findings can be reproduced using the study’s own data and methods. The second objective is to investigate the robustness of the findings through additional analysis of the data, in particular investigation of possible contamination during the sampling process and explicitly modeling the hierarchical structure of the sampling frame. A primary aim of the study was to establish which of the three modes of food assistance is superior (in terms of food security, costs, and acceptance), the sampling frame and size of the study appear to be underpowered to identify a meaningful difference between the three modes (although the study is sufficiently powered to reveal a difference between the control mode and the interventions). We will extend the analysis of the cost-effectiveness through simulation analysis. This can then inform policymakers about the potential effect of uncertainty and variability in the cost items on the cost-effectiveness for several endpoints. Moreover, this could help to design future studies by considering what sample size is required to find a statistically significant difference between these different modes.