How can a rethink of lessons from field experiments inform future research in transparency, participation and accountability?

In April, during our Washington Evidence Week, 3ie co-hosted with the World Bank’s Independent Evaluation Group a day-long public conference on Citizen Engagement and Accountable Government.  It was an inspiring and very well-attended event on an extremely important issue of our times (links to the conference videos are at the end of the blog). The event gave 3ie an opportunity to share some of the lessons we have learned after completing a major evidence programme on transparency and accountability in seven countries. 

The conference also made us consider the value of using our 3ie blog to extend the audience for the topic and start a conversation around a selected presentation or remarks. We have invited one of the panellists from the ‘Reality Check’ session, Jonathan Fox, to share a version of his remarks as a blog. Jonathan focuses his research on the relationship between citizen participation, transparency and accountability. We hope that this blog helps us continue the spirited discussion that was started at the conference. — Emmanuel Jimenez, executive director, 3ie 

Strategists in the field of transparency, participation and accountability are rethinking overly optimistic theories of change. How can a reassessment of existing studies help to inform this needed reboot? What do ‘mixed results’ from field experiments with transparency and accountability interventions tell us about where the field should go next? 

Classic field experiments to improve governance and service delivery often rely on administering modest doses of information to stimulate small-scale citizen action (in contrast to studies based on natural experiments, which can unfold at larger scales). Growing recognition that information is not a magic bullet is leading to efforts to understand why. This blog builds on a recent academic review of 48 studies involving information-led inventions, which concludes that the power to act on information matters. This suggests bringing power to the center of the analysis, rather than leaving it to be discovered as a finding.

To offer a reality check to inform the ‘what works’ question in the field of transparency, participation and accountability (TPA), this blog offers three propositions for discussion:

1. We should not be surprised when ‘low dose’ interventions lead to uninspiring impacts. Let’s keep in mind that when a medicine is administered in a very small dose, to little effect, we still don’t know whether the medicine could work. Many field experiments in the TPA field test tool-led, ‘light touch’ interventions. That is, they are brief, information-led, and localized - sometimes involving a small number of community ‘interface’ meetings. They tend to rely on information that is not clearly actionable, in contrast to a ‘targeted transparency’ approach.
   
   The ‘low dose’, exclusively local approach risks addressing symptoms rather than underlying causes of governance failures, which may also be ‘upstream’, at higher levels of governance (e.g. medicine stock outs, frontline worker absenteeism, or missing textbooks). Moreover, few field experiments address the ‘fear factor’, which for good reason inhibits collective action that might question authority.  
    
2. Some field experiments are based on ‘fuzzy proxies’, which leads to fuzzy results. Experiments that use very indirect indicators of participation or accountability may fall short of actually measuring what they say they measure. This leads to slippage between links in the causal chain that we postulate connects transparency, participation and accountability. To mention just a couple of examples of fuzzy proxies from influential studies, official informational posters on the walls of clinics are not indicators of community monitoring. Attendance at meetings says little about accountability, in the absence of assessments of deliberative processes. An influential study even assumed that the ostensibly perceived potential for collective action was functionally equivalent to the actual exercise of voice. 
   
   Plus, some field experiments are fuzzy because they blur the difference between a failure to generate participatory oversight actions – and the failure of actual participatory actions to improve service delivery. The difference between these two problems is elided when study titles implicitly conflate nominal formal structures for participatory citizen oversight with actual accountability efforts. Moreover, indirect survey questions about hypothetical scenarios have been used as substitutes for empirical evidence that bottom-up accountability really happened – to then suggest that accountability efforts were ineffective at improving service delivery. Few experiments draw on the substantial existing literature on how to measure participation and empowerment. The T4D study stands out, insofar as it attends to the nuances by documenting the nature of participation.
    
3. Large-scale evaluations have assessed only a narrow range of theories of change in the TPA field. The dominant experimental approach in the field of transparency, participation and accountability addresses only one of many possible theories of change – the ‘low dose’ inventions mentioned above. If one applies a political economy approach, this suggests a ‘crowding out’ effect that neglects the broader, more diverse mix of innovations that ‘action strategists’ are experimenting with in practice. For example, multi-pronged, multi-level strategies combine multiple tactics, which allow the whole to become greater than the sum of the parts, may have more potential to encourage pro-accountability power shifts. Plus, most field experiments are one-off, which excludes change strategies based on adaptive learning.

Examples of practitioner-led change strategies that are under-researched include: 

• Vertical integration: Multilevel civil society coalitions can pursue full scale monitoring of multiple links in service delivery supply chains, like G-Watch’s Textbook Count in the Philippines or SEND Ghana’s monitoring of school feeding programs.  
• Sandwich strategies for state-society synergy: Partnerships between grassroots watchdogs and official oversight agencies create the potential for voice and teeth to be mutually-reinforcing – in part by providing official legitimacy for citizen oversight - like in Telangana, India, where hundreds of facilitators convene thousands of public hearings to audit government social programs, or in Puno, Peru, where the regional ombuds agency certified indigenous women health clinic monitors organized by Forosalud and supported by CARE. 
• Legal empowerment:  Networks of community paralegals supported by members of the vast Namati network try to bolster social accountability by squeezing bits of justice out of unjust systems. 
• Intensive grassroots civic leadership training: Investment in training large numbers of community leaders can make it possible for the most oppressed to exercise voice in official invited spaces, as in the case of Pemberdayaan Perempuan Kepala Keluarga, Indonesia’s Women-Headed Family Empowerment movement. 
• Follow the money: The International Budget Partnership’s multidimensional fiscal governance reform agendas, including their SPARK program, pursue constituency-based engagement with the accountability ecosystem. 

These diverse change strategies, informed by power analysis and the search for state-society synergy, are certainly hard to measure. But they may be more impactful than ‘low dose’ experiments.

These three propositions drawn from rethinking the experimental literature suggest the following takeaways.

Broad-based, constituency-led change initiatives may be more promising than locally-bounded, tool-led approaches. Grounded campaigns may have more leverage than externally-designed interventions.

Take the fear factor into account. This suggests proactive measures to bolster the enabling environment for collective action by reducing both actual and perceived risks associated with voice.

Being strategic involves listening to strategists. This suggests co-designing change initiatives with actual change agents, who may well know more about how power is distributed and exercised than most scholars.

To sum up, this is a good time to reassess both theories of change and how they are tested. Building in the power analysis up front is a good place to start.

With input from Joy Aceron, Angela Bailey, Daniel Berliner, Julia Fischer-Mackey and Brendan Halloran.