The 2015 Year of Evaluation has now come and gone. There were many noteworthy events (more than 80 conferences, workshops, seminars and the like, according to some accounts), most of which focused on the needs in developing countries. Participants included some of the best known from the evaluation community across the public or non-government sectors. However, the interesting question raised in these events was, Where was the private sector?
To some extent, its absence was understandable. Innovations in the private sector are usually judged not by the indicators that are linked in complex ways to social outcomes, but by the much simpler and measurable monetary returns for the companies. But it was also a glaring omission because the private sector is now seen as ‘fundamental’ to advancing the recently declared Sustainable Development Goals (SDGs).
Finance and Private Sector Development (FPD) has become a thematic focus for governments, along with education, health and infrastructure. The World Bank Group’s private sector arm, the International Finance Corporation (IFC) has said that its shareholders, aside from asking about the financial health of the companies they invest in, ‘have also asked for greater evidence that the institution is delivering against its ultimate mission: reducing poverty and improving lives.’ Moreover, even if private investments are not directly publicly supported, their social and environmental impacts need to be assessed if the sustainability bit of the SDGs is going to be attained.
Rigorous impact evaluations (IEs) in FPD have been relatively rare. Cameron et al’s recently published review of all international development IEs published from 2000-2012 found that IEs on FPD constituted only 8.5 per cent. In a review done in 2009, David McKenzie found that there were a growing number of evaluations of FPD, but these were spread across a few areas: microfinance, microenterprise, insurance and regulatory reform. Not surprising since these are areas where it may be easier to conduct impact evaluations. He also found that there were very few impact evaluations of private sector investments on themes that attract large amounts of funding, such as energy and infrastructure. With its focus on high-quality impact evaluations in areas such as governance, environment and infrastructure, 3ie’s new funding Development Priorities window will help in filling some of these important evidence gaps.
Another important concern may be the quality of evaluations. When the World Bank’s Independent Evaluation Group assessed its impact evaluations over the past decade, it found that the evaluations done by the IFC were of lower quality than those by its public sector. The report cited the relative lack of familiarity of IFC staff with impact evaluations and protocols to do appropriate quality assurance.
I wonder if the IE community can play a bigger role in social impact bonds. According to a recent report by the Brookings Institution, an excellent primer for me on the topic, what is key is that these investments provide a clear outcome metric that is a meaningful proxy for the longer term outcomes aligned with the social (and political interests) of those who repay the bonds. Rigorous evaluation needs to be part of this evidence base.
The risks of not doing good evaluations of social impact investments are highlighted in a New York Times article. The State of Utah in the United States just paid out over a quarter of a million dollars to Goldman Sachs as the first of many payments because the company’s investment in a pre-school programme had helped 109 ‘at risk’ kindergartners avoid special education. The outcome claimed by the firm was over double the rate of what every other study had found regarding the decline in the need for special education as a result of attending pre-school. Too good to be true? Maybe. According to the article, experts are questioning the counterfactual — many of those 109 kids may have not ended up in special education even without preschool. In the meantime, Utah’s taxpayers are having to shell out a lot of money.
It is thus not surprising that the Brookings report recommends rigorous evaluations ‘that compare outcomes for a group receiving a service with another group that does not receive a service, while also accounting for differences between the groups compared.’ The implication is that evaluations need to address the counterfactual. But of the 38 social impact bonds reviewed by that report, most relied on outcomes from administrative data. Only six out of the 38 social impact bonds compared the outcomes over time by comparing the baseline and endline. Only four of the bonds used an experimental design while four used a quasi-experimental design.
IEs should thus be part of what Robert Picciotto calls evaluation’s fifth wave: social impact evaluation. But as appealing as that would be, there are admittedly important challenges to taking them up more extensively. Ensuring internal consistency of results takes time and the assessments must be delivered on the often punishing time schedule of bond payouts. This is an area that requires more work. Perhaps the IE community can learn from the private sector itself. After all, many of the most innovative private companies in the world evaluate quickly and in fact, do so quite rigorously. (See this blog about Google and how technicians there use experimental designs to research how to change behaviour to maximise e-traffic.) While the assessment of impacts on outcomes such as children’s health and education is more complicated, thinking out of the box will help us meet this challenge.
