Replication research promotes open discourse

Benjamin DK Wood 23 July 2015

The just-released International Journal of Epidemiology (IJE) suite of publications reexamining the effectiveness of deworming in Kenya demonstrates the potential impact of replication research. The headline publication is a 3ie-funded replication study. The paper has been published alongside three additional commentaries: a synopsis of a systematic review of deworming evidence, a response from the original authors, and a response from the replication researchers. The publication of these papers in a respected journal puts the role of replication squarely where we think it needs and deserves to be to promote valuable public discourse around highly relevant evaluation evidence. We’re excited about these publications for a number of reasons.

First, replication research is being published! From 3ie’s perspective, this is an accomplishment onto itself. We designed the Replication Programme to help change incentives. In order to encourage more replication papers that will improve the evidence base for policymaking, we need to convince researchers that publication outlets exist for these time-intensive replication studies. The IJE arguably has the highest impact within the field of epidemiology (see information on IJE’s impact factor here). The editors’ decision to publish these papers is a testament to the value they see in replication research. This in turn helps change publication incentives.

Second, the replication studies have sparked a larger conversation around the existing deworming evidence. We’re discovering that a significant grey area exists regarding the ability of replication researchers to recreate the originally published results. However, these replication studies cannot be simply lumped into difficult to define successful or failure categories. We find that replication studies provide researchers with a valuable space to discuss analytical decisions and the robustness of publication results. We believe that these discussions improve the science around these evaluations, which in turn enhances the quality of the evidence on which policymakers rely to spend limited development funds.

Third, these conversations are public, which allows for scrutiny of the findings and a general discussion of the research. Miguel and Kremer helpfully provide their data for replication efforts here and a replication guide for their original paper here, both of which ease the process for replication researchers to reproduce their paper. The original authors have been very open with assisting researchers interested in reanalysing their results (as an example, here’s a report on GiveWell’s replication study). But most of the subsequent replication studies of their original deworming paper don’t appear to be widely circulated or publicly posted. That has now changed with the publishing of the replication results and these commentaries on the deworming evidence in the IJE. The discourse is now open for interpretation by everyone, from researchers and policymakers to funders and implementers. Regardless of which side of the deworming debate one falls, some facts remain irrevocable. The revised tables in the original author response to the replication study correct for agreed upon errors in the original publication. This clearly demonstrates the power of replication research.

Ultimately, 3ie, through our Replication Programme, seeks to change the incentive structure around replication research. If the reanalysis process becomes standardised and journals agree to publish this type of research, there will be a genuine opportunity for researchers to provide more robust evidence for policymaking. This open discourse around replication results will help normalise the replication process. We’re hoping to see more of these open discussions in the future.