Collaborations key to improved impact evaluation designs

Suchit Nanda_Majority World_children_india

Do funding agencies distort impact evaluations? A session organised by BetterEvaluation on choosing and using evaluation methods, at the recent South Asian Conclave of Evaluators in Kathmandu, focused on this issue. Participants were quite candid about funding agencies dictating terms to researchers. “The terms of reference often define the log frame of evaluation (i.e. the approach to designing, executing and assessing projects) and grants are awarded on the basis of budgets that applicants submit. It’s a bidding process — all about executing the funding agency’s design at a minimal cost,” said a researcher.

But participants stated that 3ie’s approach to funding impact evaluations is refreshingly different. Instead of a priori structuring of impact evaluation design, 3ie focuses on active engagement with researchers and implementing agencies to develop an appropriate study design for the evaluation. This approach was demonstrated by an on-going 3ie-supported study to assess the impact of media campaigns on the incidence of early child marriage.

Child marriage continues to be practised in India, even though it is a punishable crime. The implementing agency, Breakthrough, a global human rights organization uses interventions like mass media, community mobilization and training of community members, developed during an earlier campaign on domestic violence. Breakthrough approached the government to partner them on the child marriage campaign, but they were asked to demonstrate the effectiveness of their strategies. The organisationthen approached 3ie to assist in the evaluation of the programme. Through an open bidding process, Catalyst Management Services from Bangalore were selected to design and undertake an impact evaluation, funded through 3ie’s Policy Window.

Extensive discussions were held between 3ie, the research team and the implementing agency, to arrive at a study design that would encompass both qualitative and quantitative aspects. “3ie worked with us not merely as funders but as partners on the study design,” said Urvashi Wattal, project coordinator, Catalyst Management Services. The collaboration led to changes in the project design. The study, initially planned in three districts, was expanded to include three more districts. A mid-line survey was also added.

Collecting data on a sensitive issue like child marriage is challenging. The research team will use the polling booth method, which they have used in other studies. The community members will be asked to enter an enclosed area resembling a polling booth and vote on certain issues related to child marriage. The voting will be devised through colours and other means, thus accommodating the low level of literacy.

As a staff member of 3ie I was pleased to attend a session – which 3ie was not at all involved in organising in any way – which highlighted as best practice the way we work in order to respond to the policy needs of Southern governments and NGOs. More and more implementing agencies are approaching 3ie to undertake similar studies supported by our Policy Window.

Add new comment

Authors

About

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Archives

Authors

About

Evidence Matters is 3ie’s blog. It primarily features contributions from staff and board members. Guest blogs are by invitation.

3ie publishes blogs in the form received from the authors. Any errors or omissions are the sole responsibility of the authors. Views expressed are their own and do not represent the opinions of 3ie, its board of commissioners or supporters.

Archives