Learning from 7 outcome evaluations

• 

Selkie Consulting has carried out seven evaluations of CGIAR outcomes, using the same evaluation approach called Outcome Trajectory Evaluation (OTE). The evaluations were of:

1. Soil and water management research in Ethiopia
2. Climate-smart research on solar-powered irrigation in India
3. Mainstreaming of biofortification in the African Union
4. Development of a cassava seed certification system in Tanzania
5. Development of a cassava seed certification system in Rwanda
6. Control of the potato disease Purple Top in Ecuador
7. Study on HarvestPlus’ contribution to the development of national biofortification breeding programs

We were able to publish four papers as a result, listed below. Three journal articles based on the work are under review as of May 2022, including one in the American Journal of Evaluation.

An IFPRI Discussion Paper based on the biofortified crop breeding evaluation

Douthwaite, Boru; Johnson, Nancy L.; and Wyatt, Amanda. 2022. Using outcome trajectory evaluation to assess HarvestPlus’ contribution to the development of national biofortification breeding programs. IFPRI Discussion Paper 2104. Washington, DC: International Food Policy Research Institute (IFPRI). https://doi.org/10.2499/p15738coll2.134976

Abstract

While the key role that policy plays in sustainable development has long been recognized, rigorously documenting the influence of research on policy outcomes faces conceptual, empirical and even political challenges. Addressing these challenges is increasingly urgent since improving policies—broadly defined—is at the heart of the structural transformation agenda. This paper describes the use of a new evaluation method—outcomes trajectory evaluation (OTE), based on both evaluation and policy process theory—to explore the influence of HarvestPlus, a large and complex research for development program focused on improving nutrition, on a specific policy outcome, namely the establishment of crop biofortification breeding programs in national agricultural research institutes in Bangladesh, India and Rwanda. The findings support claims of significant HarvestPlus contributions to the establishment of the programs while also raising issues that need to be monitored moving forward to ensure sustainability. The paper also discusses the pros and cons of the OTE approach in terms of both methodological rigor and program learning. In particular, the fact that HarvestPlus is a long-running program allows us to reflect on how a “backward looking” approach such as OTE builds on and complements the more “forward looking,” theory of change-based approaches that informed HarvestPlus programming and evaluation during its earlier, highly-successful phases. Such a long-run perspective is rare in development evaluation and it offers important lessons for how to think about and plan for evaluation over the course of a complex agriculture research for development program. 

A WLE Legacy Series paper on how agricultural R4D achieves developmental outcomes

Douthwaite, B.; Child, K. 2021. How agricultural research for development achieves developmental outcomes: learning lessons to inform One CGIAR science and technology policy research. Colombo, Sri Lanka: International Water Management Institute (IWMI). CGIAR Research Program on Water, Land and Ecosystems (WLE). 27p. (WLE Legacy Series 2) [doi: https://doi.org/10.5337/2022.201]

Abstract

At the end of 2021, CGIAR Research Programs (CRPs) will be replaced by Initiatives housed within One CGIAR. This new modality is intended to achieve higher levels of impact at a faster rate and at reduced cost compared to the CRPs. As One CGIAR begins, there is a unique opportunity to reflect on what has worked in different contexts. In this paper, we provide findings that relate to One CGIAR’s overarching view of how it will achieve positive and measurable impacts, and for agricultural research for development (AR4D) more generally. Specifically, we draw from three related CRP evaluations to identify how different types of AR4D approaches have contributed to successful outcomes. In the final section of the paper, we present our conclusions and provide a list of recommendations for the science and technology policy of One CGIAR and possibly other integrated research for development programs.

A WLE Legacy Series paper on Impact Tracking

Child, K.; Desta, G.; Douthwaite, B.; Haileslassie, A.; van Rooyen, A.; Tamene, L.; Uhlenbrook, S. 2021. Impact tracking: a practitioner-developed approach to scaling agricultural innovation in Ethiopia. Colombo, Sri Lanka: International Water Management Institute (IWMI). CGIAR Research Program on Water, Land and Ecosystems (WLE). 28p. (WLE Legacy Series 1). [doi: https://doi.org/10.5337/2021.226]

Abstract

This paper argues for more creativity and flexibility in agricultural research for development (AR4D) scaling and impact evaluation in complex contexts. While acknowledging the importance of setting reasonable end-of-project targets and outcomes, we argue that the achievement of outcomes and impacts, particularly in complex contexts, requires adaptive management and acknowledgment that significant positive outcomes and impacts may occur after the project funding cycle is complete. The paper presents a practitioner-developed approach to scaling AR4D innovations called Impact Tracking (IT). We illustrate IT in practice by presenting three case studies from Ethiopia in which IT proved crucial to achieving impact. The paper concludes by drawing lessons from the case studies and discussing what implications IT may have for development practitioners.

International Potato Center synthesis paper on how the CGIAR contributes to policy change

Douthwaite, Boru (2020). How the CGIAR contributes to policy change: learning from four cases. International Potato Center: Lima, Peru.

Summary

The paper was a synthesis across four evaluations — came to four conclusions:

Conclusion 1 on policy outcomes and impact: The four case studies assemble sufficient evidence to establish beyond reasonable doubt that the four trajectories have produced important policy outcomes and CGIAR has made significant contributions to them. The policy outcome trajectories have the potential to change the lives of tens of thousands (PMP) to millions (biofortification and cassava seed certification). The case studies and this synthesis report have helped explain how.

Conclusion 2 on what drove the trajectories: The trajectories made progress in part through the combined efforts of groups of people who shared and were motivated by a common, broad vision. In the biofortification declaration trajectory, this grouping took the form of a network of champions selected and trained to explicitly advocate for biofortification at high-level conferences (i.e. policy windows) on improving nutrition in Africa and globally. In network terms, trajectory actors were links away from policy-makers in the AUC and champions were needed to bridge the gap. The other trajectories were aimed at national-level policy change. Here, the type of grouping is better described as a coalition of CGIAR researchers, donor representatives, project partners and key people working in the respective national agricultural research and extension systems. The coalition members provided resources to move their trajectory forward, including funding, knowledge and access to their contacts. They were close enough in network terms to key policy makers and their advisors to be able to advocate directly, without special training or an explicit strategy to do so, or even for the word ‘advocacy’ to be used at all.

Conclusion 3 on role of donors: BMGF and DFID showed the degree to which a donor can support outcome trajectories. BMGF provided uninterrupted funding for work on cassava seed systems from 2007 which is set to continue to 2024. The idea that seed certification was crucial to a clean cassava seed system was born out of the work prior to 2012, which led to the emergence of the seed certification standards trajectory in Tanzania. BMGF staff were part of the incipient coalition that launched the trajectory and moved it forward. In the AU biofortification trajectory, BMGF and DFID also provided uninterrupted funding over many years, which continues. Both donors also supported panels of African leaders who advocated for biofortification at the highest level. Their message strengthened and was boosted by the more bottom-up advocacy on the part of the CGIAR- supported network of champions. By funding work at field to global scale, both BMGF and DFID were offered a unique overview of the biofortification trajectory. This provided both donors with an opportunity to shape the work of other trajectory actors, in particular the CGIAR, so as to reduce competition, improve collaboration and help researchers better understand how their work fits in.

Conclusion 4 on the development of a broadly-applicable middle-range theory: Part of what this evaluation has developed is a broadly-applicable middle-range theory, adapted from the political science literature, to model how trajectory actors contributed to policy change in the four cases chosen for analysis (see Finding 7). Because it is broadly applicable, it can be reused to help explain CGIAR contribution to other policy changes in other contexts. Theoretically, it has the potential to work as a framework for accumulating understanding and learning about how CGIAR research influences policy changes more broadly. The learning can be used to refine and adapt the model to different types of policy change in different contexts.