The wrong fight: moving beyond the RCT wars toward rigorous learning
A response to Todd Moss and Dean Karlan
Todd Moss and Dean Karlan are relitigating a familiar argument. The short version is that Todd Moss recently argued that the evidence-for-development agenda has run its course, while Dean Karlan disagreed.
To be more specific, both anchored on RCTs: Moss fired the opening shot, stating RCTs are a dead end, the field is methodologically captured, and none of it moves politicians. Karlan fired back, noting the straw man argument, RCTs are just 25% of published work, you still want some information before spending money, don’t you? Both pieces are worth reading. Neither quite lands where it needs to.
The problem is not that they disagree. It’s that they’re fighting over the wrong thing. Moss wants to relitigate whether the randomization revolution was worth it. Karlan wants to defend it. Meanwhile, the question that actually matters gets about two sentences between them before they move on: how do you generate credible, actionable evidence about complex development problems?
That’s the question worth sitting with.
The RCT critique is mostly right, and mostly beside the point
Moss’s sharpest point is about incentives, not methods. When a field’s career structure rewards what’s publishable over what’s policy-relevant, you get a gradual drift toward questions that fit the tool rather than questions that face an actual decision-maker. That’s happened. Nothing encapsulates this better than the Worm Wars, an evidence flashpoint 3ie contributed to by funding a replication of Miguel and Kremer's influential Kenya deworming study: twenty-plus years of methodological argument about a single study on school attendance in western Kenya, while finance ministers across the continent are asking questions that nobody’s randomizing.
Karlan is right that this doesn’t indict RCTs as such. Of course it doesn’t. The question is always whether you’ve chosen the right instrument for what you’re trying to learn. RCTs are well-suited to a real and important class of problems: program-level questions about specific, isolatable treatments where you have enough statistical power, a willing implementing partner, and an outcome you can measure in a reasonable timeframe. That’s a lot of health, social protection, education, microcredit, and agriculture work. It’s not a lot else. The questions that actually determine whether countries develop, how institutions function, and why policies change or don't, including industrial policy, governance reform, and structural transformation, don't readily fit the experimental paradigm.
To be clear: we are not arguing for ditching RCTs or the evidence base they have built. That evidence is real, it is valuable, and it has improved lives. The argument is about what complements it. Where both of them go wrong is in treating this as a binary. Moss implies the whole enterprise should be scaled back. Karlan implies that anyone frustrated with RCTs just doesn’t understand that different methods answer different questions (as if naming that truism were enough). Neither asks: so what do we actually do about the questions RCTs can’t answer?
Attribution is not the only standard for rigor
Todd’s political point – that an experiment-heavy evidence agenda is counterproductive – is important and we’ll come to that, but let’s start with the methods first. The deepest methodological assumption in this debate, mostly unexamined, is that rigorous evidence means causal attribution. RCTs deliver clean attribution; therefore, the debate is about when attribution is possible and when it isn’t.
That’s too narrow. A lot of what matters in development (governance reform, coalition dynamics, institutional capacity, policy uptake) operates through causal chains that are long, multi-actor, and deeply context-dependent. You’re not going to randomize those. But you can still ask, rigorously, whether an intervention contributed to an observed change, through what mechanisms, and why it worked or didn’t in a given setting. Contribution analysis and process tracing are not softer methods. They have explicit evidentiary standards. They use Bayesian updating to reduce subjectivity in making causal claims. 3ie applied this approach to show where evidence has indeed mattered in policy formation, validating 146 instances of evidence use across 81 research projects (including – stay with me here – many RCTs), with documented confidence levels and independent jury review. That’s not anecdote. It’s a structured, replicable way of asking “did this matter, and how?” in situations where a control group is not an option.
The same logic applies upstream. Theory-based evaluation, which means taking seriously the causal model that connects an intervention to its expected outcomes, testing the assumptions embedded in that model, and revising them against evidence, lets you learn from programs even when you can’t run an experiment. It’s particularly well-suited to the big, foundational questions Moss cares about: how industrial policy actually played out, why a governance reform succeeded in one context and failed in another, what conditions made a coalition for change possible. These are evaluable questions. They just require different methods.
The what versus the why and for whom
Even where RCTs are appropriate, they’re often not sufficient. An average treatment effect across a sample tells you whether something worked on average. It doesn’t tell you for whom, under what conditions, through what mechanism, or whether the effect would replicate in a different context with a different implementing partner. Those aren’t secondary questions. For a minister trying to decide whether to scale a program, they’re often the only questions that matter.
3ie’s evaluation portfolio has been built around exactly this gap. The organizing question isn’t just “what works” but “what works, for whom, why, and at what cost.” That means embedding process evaluations within impact evaluations, using mixed methods to understand the mechanisms behind the numbers, reviewing bodies of evidence systematically in systematic reviews that identify context-specific moderating factors, and treating the theory of change as a live hypothesis rather than a box-ticking exercise at proposal stage. None of that is in tension with rigor. It’s an expansion of what rigor means.
For complex, multi-sector problems, where feedback loops, emergent dynamics, and non-linear relationships matter, a single-outcome experimental design tells you whether something worked in that context, not why, and not whether it will work elsewhere. As 3ie's own work on complexity-responsive evaluation has documented, ignoring complexity risks misestimating, misattributing, and misunderstanding real-world impacts. The WASH Benefits trials are instructive: three large, well-funded RCTs found no effect of water, sanitation, and hygiene interventions on child stunting, a result that only makes sense once you understand that stunting is driven by interacting pathways (diet, infection, care practices, environmental recontamination) that no single-arm intervention can shift alone. A null result is not the same as an uninformative one, but without the systems understanding, it is very hard to know what to do next.
The evidence-to-policy gap is a related but distinct problem
Moss’s political argument (that evidence didn’t save USAID and won’t) is essentially correct, and Karlan concedes it. But it’s a non sequitur in a debate about how to generate good evidence. Here, Dean is on target: USAID didn’t collapse because it had too many or too few RCTs. It collapsed because of a political decision made by people who had no interest in what any study said about anything. No methodological adjustment would have changed that.
The evidence-to-policy gap is real, but it’s a different problem, and conflating it with the methods debate lets both sides off the hook. Yes, evidence availability is not sufficient for policy change. Political will, institutional capacity, champion networks, and decision-making cycles all mediate whether a finding actually reaches a decision.
The 3ie-led EIPM Conceptual Framework, developed for FCDO’s Research Commissioning Centre, maps these pathways in detail: capabilities, relationships and networks, structures and processes, and evidence culture as the four routes through which evidence actually shapes policy. That’s useful not because it replaces the question of what methods to use, but because it tells you what else has to be in place for the evidence to land.
3ie’s own contribution tracing work found that the strongest predictors of evidence use had little to do with study design. What mattered was whether researchers had engaged with decision-makers in time for the decision, whether there were champions inside the implementing agency, and whether the organization had formal processes for drawing on evidence. Quality was a threshold condition, not the main driver. That finding should shape how we think about research investment: not just what methods to use, but what relationships and institutional conditions have to accompany the research.
What neither of them mentions
The most glaring omission in this debate is the policy and institutional reform agenda. Moss argues, correctly, that the big questions (growth, industrialization, structural transformation) can’t be randomized. He then throws up his hands and calls for oral history. That’s a false choice. 3ie and the Millennium Challenge Corporation developed the PIR Methods Menu precisely to address this: an open-access tool with over fifty methods for evaluating policy and institutional reform, organized by program phase, mode of inquiry, and research question. Realist evaluation. Contribution analysis. Process tracing. Institutional ethnography. Outcome harvesting. Structured case comparison. These are not methodological also-rans. They are appropriate tools for the kinds of complex, politically-embedded questions that Moss says the field is ignoring, and they carry evidentiary standards that serious analysts can hold themselves to.
The PIR menu exists because the field has already recognized that “we can’t randomize governance” is the beginning of the methods conversation, not the end of it. Neither Moss nor Karlan seems to know it exists. That’s a problem, but it’s a solvable one.
The other omission is synthesis. Individual studies, whatever their design, rarely shift policy on their own. What moves program decisions and budget allocations is the accumulated weight of evidence across contexts, populations, and implementing conditions. Systematic reviews and evidence gap maps are how that accumulation gets structured and made navigable. For example, 3ie's Development Evidence Portal (DEP), with over 20,000 impact evaluations and systematic reviews in one searchable platform, is the most direct illustration of what that infrastructure looks like in practice. The field underinvests in synthesis relative to primary research, which means a lot of knowledge that exists in principle doesn’t exist in any usable form for the people who need it.
On evidence culture
Underlying the political argument in both pieces is a real and underappreciated problem: even good evidence, well-designed and well-communicated, often doesn’t get used. That’s not mainly a methods failure. It’s an organizational and cultural one.
The TRIPS Framework identifies five levers that institutions can pull to change this: training staff to produce and use evidence throughout the program cycle; making the right resources and tools accessible; creating incentives that reward evidence-seeking rather than punishing it; embedding evidence use in formal decision processes; and consistent signals from leadership that this actually matters. Those five things (Training, Resources, Incentives, Processes, Signals) are what distinguish organizations that actually use evidence from ones that talk about it. Launched in 2023, major development institutions are signing the Global Evidence Commitment, coordinated by 3ie, to work on exactly this: FCDO, IDB, ADB, CAF, KfW, GIZ, MCC, Norad, Deval
What a more useful conversation would look like
If you take the Moss-Karlan debate seriously as a provocation rather than a debate to win, the useful question it surfaces is: what would a genuinely pluralist, policy-relevant evidence ecosystem look like, and what would it take to build one?
Research questions should be driven by what decisions actually face policymakers, not by what a given method can accommodate. Theories of change should be treated as a live set of hypotheses to be iteratively tested and refined. Mixed methods shouldn’t be a label attached to studies that include a few focus groups; it should mean genuinely integrating quantitative and qualitative evidence to understand mechanisms, not just measure outcomes. Evaluation capacity needs to be built where policy is actually made, in government agencies and regional institutions, not just in northern universities and international NGOs. More research of the kind we do with FCDO needs to enlighten us on the best ways to broker evidence into decision-making. And we need more investment in syntheses and evidence infrastructure: making the evidence that already exists navigable and usable.
None of that requires abandoning RCTs. It requires treating them as one instrument in a much larger toolkit, matched to the questions they can actually answer, and complemented by methods that can answer the rest.
The RCT debate has been running for at least a decade and it keeps arriving at the same impasse because both sides are debating method when the real question is epistemological: what counts as rigorous knowledge about how development works, who gets to decide, and how does it connect to decisions that matter? That’s harder than arguing about randomization. It’s also where the work is.