Urban waste pickers face several intertwined challenges such as low incomes, limited education, poor health, social marginalization, and domestic violence. In Bengaluru, India, 3ie is currently evaluating an initiative called Saamuhika Shakti, which adopts an innovative ‘Collective Impact’ approach to improving the lives of informal waste pickers. The initiative, just like the problems it looks to address, is complex. Theories of change are helping us navigate this complexity as we evaluate if and how multiple interventions working in tandem make a difference. Here, we discuss the ways in which we are using theories of change in our evaluation of the Saamuhika Shakti initiative.

Initiated and funded by the H&M Foundation, the initiative includes a diverse basket of multi-sectoral interventions by seven implementation partners seeking to achieve collective impact. The/Nudge Institute acts as the backbone organization supporting and coordinating partners’ efforts. The initiative partners are BBC Media Action, CARE, Hasiru Dala, Sambhav Foundation, Save the Children, Social Alpha and WaterAid.

One of the first steps we took in evaluating this unique initiative was to develop theories of change for the five projects where partners were working with waste picker households. A theory of change or ToC is a diagrammatic representation of the various activities planned as part of an intervention, their expected outputs and outcomes, the causal pathways that connect activities to outcomes, and the underlying assumptions. It is a visualization of the logic underlying a program.

Figure 1 

Since we wanted to map the planned intervention activities and inputs, outputs, and desired intermediate and final outcomes across the initiative, we had several rounds of brainstorming with the partners. While in-person discussions would have been ideal, the COVID-19 pandemic moved our discussions online. Through calls with CARE, Hasiru Dala, Sambhav Foundation, Save the Children and WaterAid, as well as The/Nudge Institute, we were able to complete the theory of change diagrams and use that understanding for our evaluation (see the detailed steps in Figure 2).

Getting on the same page in designing the evaluation

Through this exercise, we deepened our understanding of the individual interventions that make up the Saamuhika Shakti initiative. Sometimes, the evaluator’s mental map of a project may differ from the implementer’s vision. With the theories mapped out, we verified and reconciled our interpretation of how a partner’s intervention is set up (the various activities they would undertake) and the intended outcomes.

Figure 2: An iterative approach to mapping the theories of change

Tracing the impact of COVID-19

Crucially, both for us and the implementers, the theories of change exercise helped to pinpoint which parts of the initiative were hit hardest by the COVID-19 pandemic and would need to adapt. The exercise also helped clarify which intervention components were added after the pandemic struck. The iteratively-developed theories of change, thus, helped us communicate effectively.

Zeroing in on the questions to ask

The theory of change diagrams and the process of developing them helped in other ways too. By being on the same page as the implementers, we were able to put together an initiative-level theory of change combining common elements from across projects. This initiative-level meta-theory of change is key to our evaluation.

Since the mapping exercise helped capture the outputs and outcomes accurately, it also helped clarify which parts could be evaluated. Going beyond project proposals and descriptions, the exercise helped dive into the partners’ thinking of how the different elements were linked and which assumptions and links would be critical to test. As part of the upcoming process evaluation, we will use qualitative and monitoring data to test whether these underlying assumptions are realized or not. The identified intermediate and final outcomes in the theories of change have already informed the questions we needed to include as part of our baseline survey.

If you would like to know more about developing useful theories of change or assessing the evaluability of your program, contact our evaluation specialists at info@3ieimpact.org.