We recently completed a new systematic review, ‘Strengthening women’s empowerment and gender equality in fragile contexts towards peaceful and inclusive societies’. This systematic review, also called the Gender SR, examined 14 gender-specific and gender-transformative interventions focusing on women’s empowerment in fragile contexts. Unlike most SRs, the team added qualitative synthesis to complement quantitative meta-analyses that look at effects of interventions on specific outcomes. While quantitative analysis will tell you what works and what doesn't, qualitative analysis allows you to know why it works and why it doesn’t. In the interest of promoting more mixed-methods synthesis, we want to discuss why and how we did it.
As a complementary dimension to quantitative synthesis, qualitative findings provide better scope for researchers to analyze contextual clues and have a more complete picture of the contextual impacts of programming on individuals and communities. More specifically, the experiences and views of participants and other stakeholders tend to provide useful insight into the reasons for effectiveness of interventions or lack thereof. This is a richer base from which to inform program design and pathways for successful implementation.
To give an example from the Gender SR, the quantitative analysis indicated that asset transfer interventions did not have a significant effect on outcomes such as improved self-image and confidence. Asset transfers also failed to reduce intimate partner violence (IPV) or improve the quality of relationships women had with household and community members. Findings from the qualitative synthesis provided some insights on why asset transfers didn’t work in these domains.
For example, social norms, existing gender roles and rigid restrictive beliefs were shown to preserve men’s dominance. Several studies found that assets were often held by the women only in name, and decisions about these assets were made by husbands. So, regardless of women’s income contribution, they ultimately had limited say in household and community decision-making. To improve effectiveness, a program could include interventions to address contextual and social norms or prioritize asset transfer interventions that may lead to more significant results on achievement outcomes.
Sourcing and synthesizing qualitative findings
How did we do this qualitative synthesis? Our approach to findings-linked qualitative evidence was rigorous, systematic and informed by quantitative evidence. We looked for and included qualitative studies only for the programs that had quantitative evidence which met the inclusion criteria for our review. Our synthesis incorporated three main stages:
- Search: We used four main sources to trace documents of interest: Google, Google Scholar, implementer and funder websites. Using a simple search of each program name and the country in which it was implemented, we were able to locate qualitative, descriptive , quantitative, and process evaluation studies. We also collected project documents.
- Appraisal: Studies underwent a diligent quality appraisal process. Of nearly 310 papers found, 90 studies were selected for final inclusion.
- Synthesis: We analyzed emerging themes from the qualitative evidence by intervention types, but synthesis was feasible for only eight out of our 14 intervention categories due to a lack of available qualitative evidence on the other intervention groups. By deriving descriptive themes from our inductive coding, we combined these into relevant analytical themes.
Using EPPI Reviewer 4, we carried out “line-by-line” coding of texts from each study to associate all relevant text with emerging themes. After generating a set of thematic codes in EPPI, we developed a set of descriptive themes through analyzing relationships between the themes and aggregating themes that were conceptually similar. For instance, in reviewing the effects of asset transfers, two codes captured as “decisions are made by husband” and “husband and family members do not contribute to domestic chores” were combined into one overall descriptive theme of “gender and cultural norms”. In the last stage, the descriptive themes were translated into analytical codes. Essentially, this meant providing more detailed explanations using the content of the primary studies to better understand how different variables potentially or explicitly affected the impacts of the interventions. We generated inductive codes, descriptive themes and the eventual analytical themes so they could help us analyze the interplay of context, design, implementation and population characteristics with program effects.
How you could integrate qualitative findings in your future work
Outcomes such as empowerment and gender equality are not easily quantified. Using qualitative findings can help make sense of statistical data and paint a clearer picture of effects on these outcomes in different contexts. Implementing agencies will be able to better use this data to better apply knowledge to an asset transfer program in Malawi or a program to reduce IPV in Niger. The experience of the Gender SR exemplifies this. However, even as we call for more frequent mixedmethods synthesis, there are a few things to consider. First, a separate search for qualitative evidence instead of tying it to the search for quantitative evidence may have yielded a richer evidence base. Second, the qualitative synthesis was also time and resource intensive. Systematic reviews that choose a mixed-methods approach must factor that in at the planning stage. Lastly, mixed-methods synthesis is only possible through the use of rigorous mixed-methods evaluations contributing to the production of high-quality quantitative and qualitative evidence.
The protocol for the Gender SR can be found here. The SR report will be launched at the upcoming edition of 3ie’s Evidence Dialogues webinar on 28 October 2021 (register to participate) and published on our website soon.
This is a very relevant piece for the impact evaluation narrative. In my experience as a field researcher primarily working in a quantitative and RCT-led environment, I have come to appreciate and see the benefit of qualitative research and mixed-methods studies. They can really help go into the depth and uncover nuances and complexities hidden in the rich context of the issue being studied, which the quantitative exercise of breadth is not equipped to yield. To be able to create a validated quantitative measure of women’s agency, along with the quantitative survey, a mixed methods study I worked on used a parallel qualitative investigation to fine-tune the creation of this quantitative measure. We were able to uncover the details of the women’s experience of a hard-to-measure concept like ‘agency’ and create a stronger tool as a result.