Embedding scaling into evaluation methodology and practice: A call to action

As it faces the current development finance crisis, the development community is looking for diversified funding sources, including domestic resources, private investment, and charitable giving. However, the crisis also demands that we focus intensively on spending the available resources effectively, and in particular that development initiatives are structured to support long-term sustainable impact at scale. This, in turn, requires aligning innovation, investment projects, and policy reform to support resilient, locally-owned, and eventually self-sustaining scaling pathways (as documented in an earlier 3ie blog). 

Evaluation has a key role in ensuring the mainstreaming of scaling in development finance.

A previous 3ie blog discussed why evaluation matters for effective scaling and how evaluation can best incorporate a scaling perspective by highlighting key questions, tools, and past lessons. In this blog, we build on that by sharing key takeaways from a recent study conducted under the auspices of the Scaling Community of Practice. The study analyzed the evaluation methodologies and practices of selected official funder organizations. We also introduce a newly-published tool that allows funder organizations to track the extent to which they mainstream scaling in their funding approaches.

Evaluation methodologies of most official funders hardly consider scaling.

For our study, we reviewed the OECD-DAC EvalNet evaluation guidelines, the OECD-DAC Peer Review methodology, and the MOPAN assessment methodology, as well as publicly available evaluation guidelines and policies of 18 large bilateral and multilateral official funder agencies. Our goal was to ascertain whether and how the evaluation methodologies currently used incorporate an explicit focus on scaling.

For the OECD-DAC and MOPAN guidelines, we find that while scaling is not entirely absent, they do not treat scaling effectively, let alone give practical advice on evaluating scaling. Since many of the evaluation guidelines of individual funder agencies are based on the OECD-DAC evaluation guidelines, this represents a missed opportunity to influence and support the evaluation units of official funder organizations and, indirectly, to strengthen the incentives for funder agencies to mainstream scaling.

Of the 18 bilateral and multilateral funders, our review concludes that ten of them do not address scaling; for another four, there is some, but only minimal coverage (ADB, GIZ, USAID, World Bank Group); while for a final four agencies scaling is a central part of the evaluation policies or guidelines, with varying degrees of guidance provided (GCF, GEF, IFAD, UNDP). This last set of cases demonstrates that incorporating scaling into evaluation guidelines is possible without a fundamental departure from existing evaluation approaches.

Evaluation practice of development funders rarely assesses how funders support scaling.

We also reviewed 17 recent evaluations for four agencies: four OECD-DAC peer reviews, three MOPAN assessments, six IFAD program and project evaluations, and four World Bank country program and project assessments. As would be expected, the IFAD evaluations have the most extensive consideration of scaling, while World Bank and OECD-DAC peer reviews pay very little, if any, attention to scaling. MOPAN assessments, somewhat surprisingly, do consider scaling to some extent, even though the MOPAN methodology provides little guidance on scaling to assessment teams.

How to improve the evaluation practice of official funders to reflect scaling effectively.

We recommend the following three steps for official evaluation methodology and practice:

1. OECD-DAC evaluation and peer review units, as well as MOPAN, should develop guidance documents that focus on scaling in project and program evaluations, providing key details on how scaling or potential for future scaling, replication, and sustainability can be effectively evaluated. This should include not just criteria or, but also guidance on how to answer them. In preparing these evaluation guidance documents, the OECD-DAC and MOPAN secretariats can draw on the aforementioned 3ie blogs and the scaling guidance document recently approved by the OECD-DAC membership.
2. The evaluation offices of individual funder agencies should review their evaluation policies and guidelines to integrate scaling more systematically and effectively. They can learn from agencies already using such guidelines (especially GCF, GEF, and IFAD).
3. Since effective scaling typically requires 10-15 years and effective partnerships among funder organizations, the evaluation agencies of official funders should collaborate in evaluating whether and how their agencies collectively have supported and effected scaling over a longer time horizon.

A new tool can help track the mainstreaming of scaling in funder organizations.

Finally, a new tool developed under the auspices of the Scaling Community of Practice allows funders (and their evaluators) to assess the extent to which they have currently integrated scaling into their organizational objectives, strategy and operations, set goals for further mainstreaming of scaling, and track the progress towards those goals. 

The tool tracks an organization’s performance with specific criteria across the following 13 key elements grouped by enabling conditions and operational factors:

Explore the tool here.