Evaluating Scaling Efforts: Measuring What Matters

Dear EvalForward Members,

I would like to extend my gratitude for your valuable contributions to the discussion on "Evaluating Scaling Efforts: Measuring What Matters." Your insights and diverse perspectives have been instrumental in enriching our understanding of effective evaluation approaches for scaling efforts.

The richness in depth and range of experience reaffirms the complex nature of scaling and why it is so challenging to evaluate. We embrace this challenge in developing the method note on how to assess scaling activities in the CGIAR. 

This discussion concludes for now, but there will be future opportunities to exchange ideas and knowledge. We anticipate having a fair draft of the note in mid-late February, when we will share early insights for further engagement. A summary of the discussion will soon be available on the platform in three languages: English, French, and Spanish.

Please stay connected  for updates on upcoming events and further opportunities to engage.

Dear colleagues,

I am excited to join the EvalForward community to share my experience and lessons learned in sustainable agri-food systems and value chain development. I previously coordinated FAO’s community of practice on food loss reduction, fostering knowledge-sharing and peer collaboration to improve interventions aimed at achieving agri-food transformation and the Sustainable Development Goals (SDGs). I am an expert in sustainable agri-food systems, value chains, and climate-smart agriculture, with extensive experience in project management and strategic leadership across international organizations. My work includes managing a joint project for UN Rome-based agencies (FAO, IFAD, WFP) and advising IFAD’s Agricultural Research for Development Unit (AR4D).

I have collaborated with organizations such as CGIAR (e.g., CIAT in Latin America), large international NGOs, and institutions like the African Union Commission, working closely with governments and public-private partners.

My approach emphasizes participatory design and implementation, engaging smallholders and local communities throughout the project cycle, including evaluation and learning phases, to create inclusive, efficient, sustainable, and scalable interventions.

As a Senior Consultant for IFAD’s Agricultural Research for Development Unit (AR4D), Research and Impact Assessment Division of the Strategy and Knowledge Department, in 2023, I prepared and led IFAD team for the development of the following publications that are meant to be used widely to share experiences and lessons:

•  the consolidated final technical report of the IFAD-EU co-funded programme ‘Putting Research into Use for Nutrition, Sustainable Agriculture and Resilience’ (PRUNSAR), implemented from 2015 to 2023 by 13 projects led by eight CGIAR centers and the International Bamboo and Rattan organization (INBAR);

• the Catalogue of Innovations and the Knowledge Directory (PRUNSAR Knowledge Directory: Putting research into use for nutrition, sustainability, and resilience | Knowledge for policy (europa.eu) with Agrinatura, based on the achievements and recommendations from PRUNSAR 

• the final data on aggregated indicator values upload in the EU Online Operational System (OPSYS) (Achievements / target indicators of PRUNSAR)

The EU-IFAD funded PRUNSAR program, comprising 13 CGIAR-led projects, has produced valuable resources in 2023 and 2024 that highlight lessons, challenges, and recommendations for scaling and evaluating agricultural innovations.

PRUNSAR, a €39 million program running from 2015 to 2023, supported innovations that enhanced livelihoods, nutrition, and resilience for smallholder farmers across South-East Asia, Africa, and Latin America.

The PRUNSAR publications and knowledge products are accessible on IFAD and the EU websites. They provide useful insights, including evaluation approaches, cross-cutting indicators for tracking and aggregating results, and project-specific metrics.  They reflect collaborative efforts involving CGIAR, IFAD, national partners, and beneficiaries throughout the project cycle, ensuring robust monitoring, evaluation, and learning to inform future, impactful interventions.

Best regards,

Mireille Totobesola
LinkedIn Profile: Mireille Totobesola, PhD

Scaling in areas of food security, agriculture, rural development, and climate change presents significant challenges.  However, scaling is not a straightforward process. It involves navigating complexities like varying regional contexts, resource constraints, and the need to ensure long-term sustainability. Evaluators should develop innovative approaches to measure and assess the effectiveness of scaling efforts, making this a dynamic and evolving field. The challenge in scaling is that what works well in one region like Africa or Uganda as a country or different countries might not be as effective in another due to differences in culture, infrastructure, or available resources. Evaluators must account for these variances to ensure that scaling strategies are appropriately tailored and effective across diverse settings.  Again, sustainability, must not only achieve immediate results but also maintain its impact over time, which requires careful planning and monitoring considering the health of the population. Evaluators must first define what success looks like in the context of scaling. Participatory evaluation must involve engaging stakeholders at various levels to co-create evaluation frameworks that are responsive to local and international needs.  It’s a positive move that we must all embrace. 

Hi everyone! I have really enjoyed reading through these many responses and am keen to explore more effective and diverse scaling metrics with this community. I am a research specialist with the Accelerating Impacts of CGIAR Climate Research in Africa (AICCRA) project that aims to bride the research-to-practice divide and scale climate-smart agriculture and climate information services, through fit-for-purpose engagements and collaborative design processes at local, national and regional levels across Africa. I am also writing my PhD on ‘responsible scaling’ – examining models for integrating responsible pathways – those that are inclusive and into the AICCRA project.

The arguably flawed mantra of our agricultural research for development ecosystem is that scaling is always good and should be prioritized. We are pushed by internal institutional incentives (amount of publications, funding raised) to pursue “quick wins” or “low hanging fruit”. We are pressured by donor organizations to showcase our theories of change as linear and replicable models of success.  However, many have put this thinking into question – including the authors of the background resources provided above. Of course, we also shouldn’t assume to just stop scaling. There are great merits in leveraging the knowledge and successes developed by someone else or elsewhere to a new site or context to increase the speed of implementation and the odds of obtaining the desired outcomes. Furthermore, we also need to stop assuming that we as “innovators” or “researchers” come in with best-bet solutions, and also account for existing models and common practices, and learn from these – to be able to complement rather than replace traditional knowledge and innovation, with science-based approaches.

What we need to do for attaining sustainable systems change is step away  from unrealistic notions of success – such as alone (attributable to a single project) reduce poverty and further food security – and rather set goals (and anti-goals) early on in project design aligned with a recognized desired directionality, but set adaptable indicators to be able to better target end-users and anticipate potential negative outcomes of scaling. This also means setting up a monitoring and evaluation plan that is able to account for transformative systems change over time and space – examining not only the efficient implementation of a project’s results. While scaling and systems change are not the same, they certainly need to draw lessons from one another. The UNDP M&E Sandbox highlights that “how and for whom we measure matters greatly for our capacity to learn, our ability to adapt, and our awareness of whether we are on track with our (portfolio of) interventions that interact with complex systems.” To foster effective agricultural systems, we should thus engage with those closest to the problems at hand to collaboratively find solutions.

Scaling is complex and heterogenous contexts call for diverse and dynamic approaches, that respond to local challenges and priorities. We should anticipate potential risks early, and collaboratively define visions of success with those we are trying to ‘impact’ in order to design not singular, but pluralistic and socio-technical innovation bundles that can best respond and enable relevant and inclusive change. It is also necessary to be reflexive – and look in the mirror – as researchers and question our own biases and assumptions. These are all dimensions of responsible innovation (see Stilgoe et al., 2013), that can be transferred to the science of scaling (see Wigboldus and Brouwers, 2016). 

Check out recent publications with amazing co-authors for more! 

A systems approach for more effective and inclusive agricultural innovation for sustainable transformation: A call to action, reflection and five research commentaries | CGIAR GENDER Impact Platform

https://www.nature.com/articles/s44264-024-00044-y

Dear colleagues,

Great to connect here and good to see the engagement from various corners of the scaling community. My name is Marc Schut and I work as senior innovation portfolio management advisor with CGIAR. I am also affiliated to Wageningen University as senior innovation and scaling scientist. Happy to see that some of our work is in your recommended reading list :-)

For the past 10 years, we've been contributing to put scaling on the agenda in CGIAR and beyond. This resulted from a lack of realism of what responsible scaling is, and what it takes to achieve impact at scale. A key instrument is the Scaling Readiness approach that supports innovation teams in profiling their innovation, design and assess context-specific innovation bundles and packages, and co-create scaling strategies to overcome key bottlenecks for scaling.

Beyond Scaling Readiness, we've been trying to advance practice and science of scaling in various ways, including:

• Special Issue on Science of Scaling: https://www.sciencedirect.com/special-issue/108MNQ3QF2N 
• E-learning course on innovation and scaling: https://innovationandscaling.thinkific.com/ 
• Innovation Portfolio Management for Scaling: https://www.sciencedirect.com/science/article/pii/S0308521X2400057X 
• Annual Week of Scaling: https://sajili.io/events/2b19-cgiar%E2%80%99s-scaling-week-2024 
• Directory of Scaling Experts: https://www.cgiar.org/scalingdirectory/ 
• Scaling Fund to invest in scaling: https://www.cgiar.org/news-events/news/announcing-the-winners-of-a-first-round-of-funding-to-take-agricultural-innovations-to-scale-in-africa/ 
• CGIAR Scaling for Impact Program: https://hdl.handle.net/10568/162816 
   

Our team is very interested in expanding the work and learning from others so please do reach out and engage us in exchanges and cross-learning!
 

I am Johannes Linn and serve as Co-Chair with Larry Cooley of the Scaling Community of Practice (www.scalingcommunityofpractice.com) , which was already introduced by Julie Howard. We invite everyone interested in scaling to join our Community.

As part of a major action-research initiative on Mainstreaming Scaling in Funder Organizations (https://scalingcommunityofpractice.com/mainstreaming-initiative/), Ezgi Ilmaz and I recently posted a paper which reports on our review of evaluation methodologies and practice of official development funder agencies from a scaling perspective. The paper can be found here: https://scalingcommunityofpractice.com/wp-content/uploads/2024/12/Evaluation-Guidelines-of-Official-International-Development-Funders-December-2024-1.pdf. The abstract for the paper follows. Bottom line: With a few exceptions, scaling is not effectively treated in the evaluation practice of development funders, including in the OECD-DAC evlaution guidelines. This needs to change.

Abstract:

Development funders have an important role to play in supporting the pursuit of sustainable impact by the implementers of development programs, but this requires that they systematically integrate the objective of scaling, i.e., achieving sustainable impact at scale into their mission and their operational practices. Evaluation and evaluation guidelines are potentially important instruments for mainstreaming scaling in funder organizations. This paper provides an indicative framework for how evaluation guidelines could address scaling. It then reviews the OECD-DAC EvalNet evaluation guidelines, the OECD-DAC Peer Review methodology and the MOPAN assessment methodology, as well as publicly available evaluation guidelines for 18 large bilateral and multilateral official funder agencies, to determine to what extent and how they incorporate an explicit focus on scaling. 

For the OECD-DAC and MOPAN guidelines the paper finds that, while scaling is not entirely absent, the guidelines do not treat scaling effectively, let alone provide helpful guidance to evaluators on how to assess their agency’s approach to scaling and how to evaluate their performance on scaling for specific projects or programs or overall. Since many of the evaluation guidelines of the individual funder agencies are based on the OECD-DAC EvalNet guidelines, this represents a missed opportunity to influence and support the evaluation units of official funder organizations and, indirectly, to strengthen the incentives for funder agencies to mainstream scaling, 

The paper finds that for ten funder agencies evaluation policies or guidelines do not address scaling; for another four, there is some, but only very limited coverage; while for a final four agencies scaling is a central part of the evaluation policies or guidelines, with varying degrees of guidance provided. This last set of cases demonstrates that the inclusion of scaling in evaluation guidelines is possible without a fundamental departure from existing evaluation approaches. 

The paper then reviews 17 evaluations and assessments for four agencies: four OECD-DAC peer reviews, three MOPAN assessments, six IFAD program and project evaluations and four World Bank country program and project assessments. As would be expected, the IFAD evaluations have the most extensive consideration of scaling, while World Bank and OECD-DAC peer reviews pay very little, if any, attention to scaling. MOPAN assessments somewhat surprisingly do consider scaling to some extent, even though the MOPAN methodology provides little guidance on scaling to assessment teams.

The paper concludes with a summary of findings and recommendations for how scaling could be more systematically and effectively addressed in evaluations.

Comments on this working paper are welcome and should be addressed to Johannes F. Linn at jlinn@brookings.edu by 28 February 2025.

IICA was involved working with FAO, CTA and IFAD on developing, improving and testing a methodology now known as capitalization. Useful for purposes like evaluation, marketing, communication...is also different from systematization. We have focus on "experiences" and yes trying to scaling up those. Three elements are vital and without them, the task of scaling becomes more challenging: 

• Real participation: the experience "owners" as well as those interested in scalling must be present.
• The experience is more concrete. A project becomes to big, where differences of time and context will play against us.
• Institutions: The adoption process requires in many cases institutions willing to continue the race other leaves. 

Judging success is a matter of those who have adopt, adapt and increase the use of "a solution". 

There are plenty of materials develop under the alliance with IFAD, FAO and CTA. IICA has put a document in spanish and english for our cooperation purposes-also available publicly: https://repositorio.iica.int/handle/11324/21455 

 Happy New Year!

Dear colleagues,

The evaluation of scaling-up efforts is a crucial challenge, especially in the Guinean context where innovations and development programs play a vital role in improving food security, agriculture, and rural development. In Guinea, where resources are often limited and needs are immense, it is imperative to measure the real impact of our initiatives to ensure they truly benefit local populations.

Take the example of sustainable agriculture. In Guinea, several innovative projects aim to improve agricultural yields and promote sustainable farming practices. However, evaluating the success of these projects is not limited to measuring agricultural yields. It is essential to consider the impact on local communities, environmental sustainability, and social inclusion. Evaluators must adopt a holistic approach, integrating the perspectives of farmers, local organizations, and policymakers [1][2].

Another crucial aspect is the complexity of the systems in which these innovations are implemented. In Guinea, development projects often involve multiple partners, each with its own objectives and contexts. Evaluating scaling-up in this context requires appropriate methods to capture this complexity. Evaluators must not only measure immediate results but also understand the underlying dynamics and interactions between the different actors [3][4].

It is also crucial to ensure that the people affected by these innovations drive the effort to amplify the impact. In Guinea, this means actively involving local communities in the evaluation process. Their voices and experiences must be at the heart of our analyses to ensure that the innovations truly address their needs and aspirations [2][5].

In conclusion, evaluating scaling-up efforts in Guinea requires an integrated and participatory approach. We must go beyond traditional indicators to capture the true scope and impact of our initiatives. Together, we can develop more robust and adapted evaluation methods, ensuring that our scaling-up efforts contribute genuinely to Guinea's sustainable and inclusive development.

I invite you to share your experiences and thoughts on this critical subject. Together, we can enrich our understanding and improve our practices for a sustainable and meaningful impact.

Sincerely,
Jonas Sagno

References:
[1] https://mptf.undp.org/sites/default/files/documents/rapport_final_evaluation_finale_projet_isp_vf.pdf
[2] https://www.evalforward.org/fr/discussions/levaluation-des-actions-de-mise-lechelle-mesurer-ce-qui-compte
[3] https://gret.org/appel-doffres/evaluation-finale-du-projet-doperationnalisation-de-lapproche-one-health-a-lechelle-communale-en-guinee-forestiere/
[4] https://www.globalpartnership.org/fr/node/document/download?file=document%2Ffile%2F2020-06-06-evaluation-sommative-appui-du-pme-guinee.pdf
[5] https://unesdoc.unesco.org/ark:/48223/pf0000375769
[6] https://mptf.undp.org/sites/default/files/documents/25000/pbf-irf-104_gpi2_final_evaluation.docx
[7] https://webapps.ifad.org/members/ec/121/docs/french/EC-2023-121-W-P-3.pdf

Dear Colleagues,

I believe that an innovation, program, or policy that has been well-developed, particularly following a theory of change approach, requires very little effort to scale up. Founded on a relevant problem or real constraint, and analyzed in a participatory manner, the resulting innovation, program, or policy quickly spreads in terms of adoption and social behavior change. The impact is easily visible and more readily expanded. The effort to scale up the innovation, program, or policy can be measured at three levels:

1. At the level of adopters of the innovation or the contribution of the program or policy: How many people are adapting the innovation? What uses are being made of the innovation or this contribution of the program or policy implemented?
2. At the level of beneficiaries of the exploitation of the adopted innovation: Because the adoption of the innovation generates needs that are met by those who are able to provide them and who are therefore direct beneficiaries of the adoption of the innovation or the contribution of the program or policy implemented. In addition to the direct beneficiaries of the exploitation of the adopted innovation, there are also indirect beneficiaries and beneficiaries of induced effects that must be inventoried. Who are they? How many are there? What is their geographical distribution?
3. At the level of the impact produced by the innovation, program, or policy implemented: There are also direct beneficiaries, indirect beneficiaries of the impact. Who are they? How many are there? What is their geographical distribution?

Thank you very much.

Scaling Impact: Challenges, Insights, and Approaches to Effective Evaluation

Scaling the impact of innovations, programs, and policies in areas like food security, agriculture, rural development, and climate change is a pressing challenge. As evaluators are increasingly called upon to support these efforts, the field is evolving to meet the complexities of scaling. This document provides a detailed exploration of scaling, offering insights into professional experiences, evaluation criteria, methodologies, and the incorporation of diverse contexts and stakeholders. Drawing on guiding questions and background resources, it aims to foster a deeper understanding of scaling and its evaluation.

Understanding Scaling

Scaling refers to expanding the reach and impact of innovations, programs, or policies to achieve broader societal benefits. This process involves increasing adoption, adapting interventions to different contexts, and ensuring sustainable outcomes.

Dimensions of Scaling

• Scaling Up: Increasing the size or scope of an intervention to reach a larger population.
• Scaling Out: Expanding to new geographic or demographic areas.
• Scaling Deep: Changing underlying cultural, social, or systemic norms to sustain impact.

Professional Experiences with Scaling

Evaluators play a critical role in assessing and supporting scaling efforts. Here are key areas of involvement:

1. Evaluating Innovations to Inform Scaling Decisions

   Evaluators often assess programs, policies, or interventions to determine their scalability. This includes analyzing:

   • Effectiveness: Does the innovation achieve its intended outcomes?
   • Efficiency: Are resources used optimally to deliver results?
   • Sustainability: Can the innovation maintain its impact over time?

2. Evaluating at Different Levels or Types of Scale

   Scaling efforts occur at various levels, from local pilots to national or global implementations. Evaluators analyze:

   • Contextual Factors: How do geographic, cultural, and economic contexts affect scalability?
   • Stakeholder Readiness: Are communities, institutions, and policymakers prepared to adopt the innovation?

3. Evaluating Success in Scaling Efforts

   Success metrics vary but often include:

   • Adoption Rates: The extent to which target populations embrace the innovation.
   • Impact Metrics: Measurable improvements in outcomes like food security, crop yields, or greenhouse gas reductions.
   • Equity and Inclusion: Ensuring benefits reach marginalized or vulnerable groups.

4. Supporting Scaling Efforts

   Beyond evaluation, professionals contribute by:

   • Capacity Building: Training stakeholders to implement and sustain interventions.
   • Knowledge Sharing: Disseminating best practices and lessons learned.
   • Policy Advocacy: Influencing decision-makers to support scaling initiatives.

Lessons Learned in Evaluating Scaling Efforts

Challenges

• Complexity: Scaling often involves interacting systems with unpredictable dynamics.
• Context-Sensitivity: Interventions that work in one setting may fail in another.
• Sustainability: Maintaining impact over time requires ongoing resources and commitment.

Key Insights

• Adaptation is Key: Rigid replication rarely works; interventions must evolve to fit new contexts.
• Engage Stakeholders Early: Inclusive planning ensures relevance and buy-in.
• Monitor Continuously: Scaling is a dynamic process that requires real-time adjustments.

Unanswered Questions

• How do we measure long-term systemic change resulting from scaling?
• What frameworks best capture the interplay between scaling efforts and local ecosystems?

Criteria for Judging Scaling Efforts

Successful Scaling

• Impact: Tangible improvements in targeted outcomes.
• Equity: Fair distribution of benefits across populations.
• Sustainability: Continued effectiveness without external support.
• Adaptability: Ability to thrive in diverse environments.

Unsuccessful Scaling

• Limited Adoption: Failure to attract or retain users.
• Negative Externalities: Unintended consequences that harm communities or ecosystems.
• Resource Dependence: Over-reliance on external funding or expertise.

Methods for Evaluating Scaling Efforts

Traditional Methods

• Logical Frameworks: Outlining inputs, activities, outputs, outcomes, and impacts.
• Case Studies: Detailed analyses of scaling successes or failures.
• Surveys and Interviews: Gathering qualitative and quantitative data from stakeholders.

Innovative Approaches

• Systems Thinking: Mapping interdependencies within complex systems.
• Developmental Evaluation: Adapting evaluation processes to evolving scaling efforts.
• Participatory Methods: Engaging communities in co-designing evaluation criteria.

Gaps in Current Methods

Existing approaches often fall short in addressing:

• Dynamic Contexts: Rapidly changing environments where scaling occurs.
• Collaborative Efforts: Multiple actors with varying goals and resources.

Evaluating Post-Funding Scaling Efforts

Scaling often continues beyond the lifecycle of a funded project. Evaluators must:

• Track Long-Term Outcomes: Assess sustained impact over years or decades.
• Measure Institutionalization: Determine whether the innovation becomes embedded in policies or practices.
• Identify Ongoing Challenges: Highlight barriers to continued success.

Complexity in Collaborative Scaling

Scaling frequently involves multiple stakeholders across diverse contexts. Key considerations include:

• Coordination: Aligning goals and activities among partners.
• Cultural Sensitivity: Respecting local values and norms.
• Power Dynamics: Ensuring equitable decision-making among collaborators.

Ensuring Affected People Lead Scaling Efforts

For scaling to be effective and equitable, it must center the voices of those directly impacted. Strategies include:

• Participatory Planning: Engaging communities from the outset.
• Capacity Building: Equipping local actors to lead implementation.
• Feedback Mechanisms: Creating channels for ongoing input and course corrections.

Conclusion

Scaling impact is a multifaceted challenge requiring innovative evaluation approaches. By sharing experiences, adopting adaptive methods, and centering affected communities, evaluators can play a pivotal role in advancing scalable solutions for pressing global issues.

Further Exploration

• Scaling Impact: Innovation for the Public Good by Robert McLean and John Gargani (2019) provides foundational insights
• Stuck on Scale by Lennart Woltering (2024) examines systemic challenges in agricultural scaling
• Innovation Portfolio Management by Schut et al. (2024) offers lessons from private sector practices

As 2024 comes to an end, I thought I would share more FREE scaling resources (see below). And I want to announce that in 2025 IDRC is planning to release a free MOOC (massive open online course) on scaling impact. The release date is TBD. In the meantime...

The Scaling Impact Book, Playbook, and More Resources at IDRC
English https://idrc-crdi.ca/en/scalingscience

French https://idrc-crdi.ca/fr/misealechelle

Spanish https://idrc-crdi.ca/es/scalingscience

Video
Scaling Impact: Five Big Ideas (Gargani, 2024) 
https://www.youtube.com/watch?v=_k49xuTZyu0

Articles 
Scaling Science (Gargani & McLean, 2017) 
Stanford Social Innovation Review 
https://ssir.org/articles/entry/scaling_science

Dynamic evaluation of agricultural research for development supports innovation and responsible scaling through high-level inclusion (Gargani, Chaminuka & McLean, 2024) 
Agricultural Systems 
https://www.sciencedirect.com/science/article/pii/S0308521X24001823

Website 
scalingXchange
A Call to Action from the Global South 
https://www.scalingxchange.org/

See you all in the new year!

John

Thanks to all who contributed so far. The depth and range of experience is impressive. And it sheds light on why scaling is so difficult—it is undertaken in many ways in diverse contexts for different purposes through multiple pathways.

When Rob McLean and I wrote Scaling Impact: Innovation for the Public Good, we wanted to understand what innovators in the Global South considered successful scaling. We learned a lot. We also learned how much we still need to learn.

One area of learning is how to evaluate scaling efforts. This is distinct from evaluating whether an innovation/program/policy/etc. works (in some sense) at different scales. That is important. But as I see it, when evaluating scaling, the question is: How well has scaling been undertaken to achieve the best (in some sense) impact? 

Part of my struggle in answering this evaluative question is perspective. I believe we always want to answer it from the perspective of the people who are affected by scaling. In addition, we may want to answer it from the perspective of the innovator who guides an innovation through its life cycle/pipeline/diffusion/etc. Or we might answer it from the perspective of organizations that run programs/sell products/advance policies that promote the use and/or benefits of an innovation or bundle of innovations. Then again, we might consider the many variations of an innovation that may be set loos by a discovery (for example, AI), and how collectively their competition and complementarity create impacts that, for better or worse, are often unanticipated. And then there is the larger systems perspective in which the innovation is one of many factors. This is not a complete list.

What perspectives matter? How feasible is it to consider more than one at a time? How do we take into account that some may benefit from an innovation and others do not? Or is it simpler than this?

Hello everyone,

Reflecting on my experience, my work primarily focused on three key strands of evaluation for innovation and scaling:

1. Evaluating Systems

• Scaling requires clear benchmarks of success that reflect specific needs and contexts. For example, localized interventions may not apply universally, and some innovations might become obsolete or counterproductive over time.
• Given the multidimensional nature of scaling—spanning socio-economic, ecological, and cultural aspects—frameworks must align strategies across diverse contexts and components.
• Assessment of the value of scaling efforts by measuring potential returns, costs, and unintended consequences.
• Diffusion mechanics, such as adoption patterns modeled through tools like the Bass model, helps to understand the trajectory from early adoption and growth to eventual saturation.
• Scenario-based risk assessments and simulations are useful to identify conditions for success, anticipate challenges, and refine strategies for scaling.

2. Evaluating Networks

• Understanding stakeholder ecosystem—whether tightly knit or dispersed—can help identify early adopters and drive broader scaling strategy. 
• Addressing diverse interests and agendas across stakeholders, such as government, private sector, and local leaders, is crucial to align incentives and secure buy-in. 
• Mitigating resistance from those benefiting from the status quo is essential to overcome barriers. 
• Exploring co-creation opportunities, such as collaborative funding, strategic alliances, and public-private partnerships

3. Evaluating Adaptability

• Embedding evaluation within adaptive approaches fosters continuous learning through iterative design, testing, adaptation, and feedback loops executed in short cycles.
• Tools like the Rapid Results Approach (RRA) use structured processes to enhance focus and collaboration among delivery teams, with evaluation establishing performance metrics to drive results iteratively.

Hello everyone,

Please allow me to introduce the Scaling Community of Practice (CoP), a voluntary network of development and climate change professionals who are working to achieve impact at scale. The CoP was founded in 2015 by Larry Cooley and Johannes Linn as a platform for sharing scaling experience and learning across sectors, and has grown to over 2000 members from around the world. The CoP's mission is to integrate, distill and consolidate scaling expertise with a view to improving the quality of scaling efforts and encouraging more development and climate actors to adopt a scaling mindset and best practices.

All are welcome to join the CoP and its affiliated working groups:

Agriculture and Rural Development

Climate Change

Education

Fragile States

Health

Mainstreaming

Monitoring and Evaluation

Nutrition

The CoP has produced a number of reports which may be of interest to EvalForward colleagues. The reports can be accessed here. The 2022 report on Scaling Principles and Lessons (Kohl and Linn) may be of particular interest. It is intended to respond to the need for a concise statement of scaling principles and lessons that apply broadly across sectoral areas and are designed to help guide development practitioners. The paper presents eight scaling principles and 21 lessons that unpack those principles, grouped under six broad questions about how to address the scaling challenge.

The 2019 Scale Up Sourcebook (Cooley and Howard) focuses on scaling in the agriculture sector, specifically in international agricultural research and development programs. Chapter 6, "Tailoring Metrics, Monitoring, and Evaluation to Support Sustainable Outcomes at Scale" notes, e.g., relevant to the questions guiding this discussion:

..." relatively few research or pilot projects generate the critical information needed to go beyond proof of concept and provide a basis for assessing scalability, streamlining delivery, informing advocacy, and guiding scaling. Increasingly, however, scaling experience demonstrates that the following three overlapping but different types or tiers of information are needed: 

Tier 1 information is generated to test the efficacy of interventions, often under controlled or semi-controlled conditions. 

Tier 2 information is used to refine, simplify, and adapt interventions to real-life policy, financial, and operational considerations. 

Tier 3 information is generated during the scaling process to monitor fidelity and inform needed adjustments to intervention design and scaling strategy during the scaling process." 

It is tempting to view these tiers as a sequence of information needs over time as the focus of scaling moves from effectiveness to efficiency to expansion. However, experience suggests the need to incorporate efficiency and expansion considerations, and to test results under realistic conditions, at the earliest possible time rather than to defer these issues until proof of concept is well established. To do otherwise is to run serious risk of adding to the graveyard of “proven” but unscalable technologies." 

Further, one of the key background documents (and brand new - 2024) cited for this discussion was developed by another co-author of the CoP's Agriculture and Rural Development Working Group, Lennart Woltering, Stuck on Scale: Rethinking Scaling and Systems Change for AR4D.

Woltering says that "AR4D organizations are largely stuck in simplistic, linear, and techno-centric notions of how change happens which is manifested by the omnipresent notion of scaling innovations." He argues that a 'systems approach (instead of a linear approach) is needed instead to address the persistent complex problems such as food insecurity and poverty, and to better understand how food systems can transform to become more environmentally sustainable, contribute to food security and improve economic equity.'

My take-away is that our current M&E practices are inadequate to address even the more simplistic, linear view of scaling - let alone the  critically important systems dimensions discussed by Woltering. 

I am glad that EvalForward has convened this discussion to allow us to put our heads together to work on improving M&E related to scaling in AR4D. I look forward to reading and contributing to the discussion in the weeks ahead. And - in the meantime- a warm welcome to all to join the Scaling CoP!

Measuring What Matter – Critical Overlooked Evaluation Parameters

Based on the sub-title of this evaluation please allow me to expand on a couple critical but overlooked parameters that once included could substantially change the way we assist smallholder communities and perhaps better was to scale up their acceptance of innovations intended to assist them improve their well-being. Thes parameters may possibly be visibly seen without even interviewing farmers, or at least any interviewed data visibly confirmed.

The critical parameter is “Timing of Field Operations.” Has anyone ever considered this? How long does it take for farmers to complete basic tasks such as crop establishment, weeding, harvesting, and threshing? How does this compare with what development specialists anticipate? Why are there such delays? Typically, the expected time for crop establishment is only 2 weeks, however if carefully evaluated for manual farming, as most smallholder still do, it would be accuratly identified as stretching for over 8 weeks. Well beyond the time anticipated for optimum implementation of intermediate activities such weeding, and with well understood decline in yield potential. A simple physical observation of the fields associated with a project community would easily show this spread in crop establishment. The delay is normally casually noted by the development personnel but not evaluated and attributed to poor motivation, education on the importance for early planting, or spreading risk. I tend to dismiss all of these and add another more likely possibility. That is farmers have limited dietary calories relative to what is needed for a full day of agronomic field work. We tend to recognize farmers are poor and hungry but fail to factor hunger as a hinderance to crop management. This would be another critically overlooked parameter, with surprisingly little solid data. What limited data is available shows smallholder would be lucky to have access to 2500 kcal/day which after allowing 2000 kcal/day for basic metabolism leaves only 500 kcal/day for manual exertion. Against this limited diet the dietary requirement for a full day of agronomic field work is 4000 kcal. With only 500 kcal workday is limited to a couple diligent hours perhaps paced for a couple more hours at lower diligence. It noted to Kenyan introduce to the topic that in Kenya the casual workday is only 5 hours. If we added this to our evaluation parameters, would we get a better understanding of what smallholders are facing and provide guidance to better means for scaling up crop management and food security? However, it has to be recognized as a major hinderance to scaling agronomic acceptance before it will be included as a necessary parameter for evaluations.

Another way to quickly approach to the problem and perhaps confirm the need to address the issue, is to simply make a field visit to where it is possible to observe a variety of farms and see how many people are working in the field in the morning and again after lunch. I expect there will be a substantial decline for the afternoon observation. That was the case in Madibira, Tanzania, the 3000-ha irrigated rice project I worked on, and stimulated my interest in the dietary energy balance concern.

If dietary energy is a critical issue in scaling agriculture production, how critical will it be facilitating smallholder access to contract mechanization to reduce the drudgery for smallholder farmers and enhance the time it takes for crop establishment and allow them to concentrate of other important crop management activities? The importance of mechanization to smallholder farming can be easily seen in paddy producing Asia where some 30+ years ago farmers shifted from water buffalo to power tillers. While there is limited data on it impact, I get the impression it half the crop establishment period, increased the area individual farmer could manage, comfortable allow for double cropping irrigated rice and when small combines were added allowed for 5 rice crops every 2 years. Also, look at Egypt where contract land preparation with tractors has been the common practice for at least 40 years I can account for. Is it possible to scale up smallholder production without facilitating access to mechanization? It might be worth noting that a return visit to Madibira 5 years after the advisor effort ended, the farmers had obtained 50 Asian power tillers for use in the paddy irrigation scheme plus 4 4-wheel tractors for use on the adjacent upland areas. If the development community has limited interest in mechanization the farmers certainly do. The critical need to the financial arrangements that will allow individual owner/operators to obtain the necessary tractors to serve their communities.

Please note the referenced article I prepared reflecting on my 50+ years working with smallholder communities. The article is more interested in factual accuracy than political correctness with a major section on Dietary Energy Balance and critical need for mechanization for smallholder communities. The link is: https://agsci.colostate.edu/smallholderagriculture/wp-content/uploads/sites/77/2023/03/Reflections.pdf

Thank you  

Thanks. This is very valuable.

Hoping Hezekiah considers looking at scaling in terms of:

1.  Policy development to allow like-minded innovations, in addition to
2.  Horizontal ‘expansion’ of successful factors.

Both have advantages and disadvantages that the discussion can learn based on his experience and others’ as well.

All the best.

Dear Hezekiah,

Thank you for the feedback and follow up questions. 

Since the project's interventions focus more on influencing planning than on direct implementation, my pervious explanation (engaged stakeholders in participatory processes and fostered strong multi-stakeholder collaborations for co-designing) are essential to ensure that the approach has a real impact. While we have not yet conducted an independent evaluation for this project, the stories of change have helped us self-assess the project's impact to some extent. Stories of Change (SoC) document successful practices and challenges as a reference for replication in other contexts, while supporting evidence-based evaluation of strategy effectiveness. (will be published soon)

The metrics established to track progress include: (1) process monitoring to track the accomplishment of output-level targets, (2) measurement of progress indicators for outcome-level targets, and (3) Stories of Change to describe the project's impact on individual stakeholders or institutions.

I would expect an evaluator assessing participatory processes and stakeholder collaboration to pay attention to representation and inclusiveness of all groups, including the vulnerable, ensuring participation is substantive with meaningful input, transparency and fairness, synergies to align goals, and collaborative outcomes that reflect shared consensus in the form of outputs or action plans. 

Anyone who would like to have a look at the projects outputs and our knowledge products can access them through the following link: https://www.cifor-icraf.org/project/scaling-JA-palmoil/#home

Dear Hezekiah,

Elaborating further on my responses, and answering your questions, the following considerations are quite important to note. The Food and Agriculture Organization (FAO) employs several context-sensitive evaluation approaches to assess the success of its initiatives effectively. Here are some specific strategies that FAO has used:

1. Stakeholder Engagement: FAO actively involves stakeholders at various levels, including local communities, government bodies, and NGOs, in the evaluation process. This participatory approach helps to gather diverse perspectives, ensuring that evaluations are grounded in the realities and needs of those directly affected.
2. Contextual Analysis: Prior to evaluations, FAO conducts thorough contextual analyses to understand the socio-economic, environmental, and cultural factors influencing agricultural practices and food security in specific regions. This helps in tailoring evaluations that reflect local conditions and challenges.
3. Utilizing Mixed Methods: FAO employs a combination of qualitative and quantitative methods in its evaluations. By integrating surveys, case studies, focus group discussions, and interviews, the organization can capture a holistic view of outcomes and impacts, taking into account the complexities of local contexts.

And ensuring the success of sustainability in scaling agricultural projects is critical for long-term success and impact. Several indicators can help assess the sustainability of scaling efforts in such initiatives. Here are some key indicators to consider:

1. Economic Viability:
  - Cost-Benefit Ratio: Measure the economic returns from the project relative to the costs incurred.
  - Profit Margin: Assess the profitability of scaled operations.
  - Market Access and Value Chain Integration: Evaluate the ability of farmers to access markets, as well as their participation in value chains.

2. Environmental Impact:
  - Soil Health: Monitor soil fertility, erosion rates, and organic matter content.
  - Biodiversity: Measure the diversity of crops and wildlife in the area influenced by the project.
  - Water Usage: Assess water efficiency and the sustainability of water resources used in agricultural practices.

3. Social Equity:
  - Community Involvement: Evaluate the level of local stakeholder engagement and participation in decision-making.
  - Access to Resources: Monitor equitable access to land, credit, and inputs for all farmers, especially marginalized groups.
  - Gender Inclusion: Assess the involvement of women and marginalized groups in farming, decision-making, and benefits from the project.

4. Resilience and Adaptability:
  - Risk Management Practices: Evaluate the adoption of practices that enhance resilience to climate change, pests, and market fluctuations.
  - Diversity of Income Sources: Analyze the diversification of income streams for farmers involved in the project.

5. Technological Adoption and Innovation:
  - Use of Sustainable Practices: Measure the extent to which sustainable agricultural practices are adopted, such as organic farming, conservation agriculture, and agroforestry.
  - Innovation Transfer: Assess the effectiveness of knowledge and technology transfer mechanisms.

6. Capacity Building and Education:
  - Training and Support Programs: Evaluate the availability and effectiveness of training programs for farmers.
  - Access to Information: Monitor the availability and accessibility of agricultural research and extension services.

7. Impact on Local Communities:
  - Food Security: Measure changes in household food security levels in the target area.
  - Nutrition Improvement: Assess improvements in dietary diversity and nutrition among local populations.

Hope this is useful and kindest regards,

Serdar Bayryyev, Senior Evaluation Officer, FAO

Dear Serdar, Monica and Mike:

Thank you for your very insightful contributions.

Our focus is on the how to evaluate efforts to scale agricultural research for development (AR4D) innovations, specifically methods that lend themselves to assessing processes and performance.

Serdar: Great examples and information on criteria for evaluating success. You alluded to the importance of context-sensitive evaluations, any specific approaches FAO has used to achieve this? What indicators of sustainability of scaling efforts should one look for?

Monica: Thank you for the example of CIFOR's project. I read your reflections from a project design perspective, has the project been evaluated? A couple of questions. What should an evaluator look for while assessing participatory processes, multi-stakeholder collaborations? Could you please elaborate on the specific mechanisms and metrics that were established to track progress? Would the existence of story of change and implementation pathways be useful indicators for the scaling process?

Mike: Yes, innovation scaling is complex. I'm curious about your perspectives on the complexities inherent in scaling initiatives, please share some details. How should an evaluator treat such complexities in a timebound evaluation exercise? What kind of information should we gather to evaluate such complexities? Should we even attempt to evaluate the kind of complex systems that drive scaling for impact?

Your ideas are greatly appreciated.

Dear Hezekiah,

We have assessed various programs and interventions of FAO and its partners, aimed at enhancing food security and sustainable agriculture. These reports are available from our website: https://www.fao.org/evaluation/en/

In many instances, our evaluations informed stakeholders on whether efforts were made for scaling up the programs further, and whether scaling-up has worked or not. We have leveraged evaluation findings to provide actionable recommendations for scaling, highlighting key success factors which led to broader adoption strategies at larger scale or in other locations.
 

Here are a few illustrative cases:
 

1. Integrated Pest Management (IPM): An evaluation of the FAO's implementation of IPM in various regions showed significant improvements in pest control and crop yields. The evaluation highlighted how local farmer training and the adaptation of IPM strategies led to successful outcomes. This informed stakeholders about the potential benefits of scaling up IPM programs in other agricultural zones, particularly in countries facing similar pest challenges. The key success factors identified included participatory approaches involving local communities and leveraging traditional knowledge.
 

2. The Improved Global Governance for Food Security initiative: This program aimed to enhance cooperation among stakeholders in food security. Evaluations revealed that enhanced governance structures were instrumental in enabling better stakeholder engagement and resource allocation. The findings recommended scaling up governance training at local levels as a best practice that could be adopted in other regions. The evaluation emphasized the importance of establishing clear communication channels and collaboration frameworks, which were pivotal in achieving positive food security outcomes.
 

3. Climate-Smart Agriculture (CSA): Evaluations of FAO's CSA programs demonstrated how integrating climate resilience into farming practices benefited farmers in various countries. The assessments pointed out that scaling these interventions led to improved productivity and sustainability. Key success factors included the provision of technical assistance, access to climate information, and the establishment of demonstration plots. As a result, recommendations were made to expand similar CSA initiatives to other vulnerable agricultural areas.
 

4. Food and Nutrition Security Program in Ethiopia: The evaluation of this multifaceted program indicated significant improvements in dietary diversity and access to food among targeted households. It highlighted the importance of community participation and localized content in training modules, which were critical for the successful implementation of the initiative. The recommendations called for expanding the program further into other regions of Ethiopia and beyond, based on the positive outcomes observed, thereby underscoring the crucial role of context-specific interventions.
 

5. Support for Smallholder Farmers: An evaluation focused on FAO's initiatives to support smallholder farmers through access to resources and markets found that these programs had a high success rate in improving food security and income levels. The successful models were identified for potential scaling up in different contexts across Africa and Asia. The report emphasized the importance of building local partnerships and ongoing support mechanisms to ensure long-term sustainability and effectiveness.
 

These evaluations, among others, illustrate how FAO and its partners use assessments to derive actionable insights that facilitate the scaling up of programs aimed at enhancing food security and promoting sustainable agriculture practices globally. The findings provide a roadmap for replication and adaptation in various contexts, ultimately contributing to broader efforts in achieving food security objectives.

Lessons Learned:
One key lesson has been the importance of context-sensitive evaluations that consider local conditions and stakeholder dynamics. Furthermore, engaging with local communities throughout the evaluation process enhances ownership and relevance of scaling initiatives. The criteria that could be used for assessing successful scaling could include the following: 
- Evidence of increased access or adoption of innovations.
- Positive changes in target outcomes (e.g., improved livelihoods, enhanced food security).
- Sustainability of scaling efforts over time.

I look forward to further discussions and shared learning on this important topic!
 

Reflecting on my experience with CIFOR's project, Scaling Jurisdictional Approaches in the Palm Oil Sector, which focused on enhancing the readiness of four major palm oil-producing regencies in Indonesia. Scaling in this context involves expanding successful approaches and innovations across jurisdictions while ensuring they are adapted to local contexts.

During the implementation phase, we engaged stakeholders in participatory processes, fostered strong multi-stakeholder collaborations for co-designing jurisdiction-specific Theory of Change, Theory of Action, and Monitoring and Evaluation Framework and scaling effective solutions as scalable recommendation adaptable to regional action plan at jurisdictions level and embedded policies such as the Regional and National Action Plans for Sustainable Palm Oil (RAD KSB and RAN KSB) to ensure long-term sustainability. 

To measure and evaluate scaling efforts, we established robust mechanisms with specific metrics to track progress. We ensured policy integration by aligning efforts with the RAD KSB and RAN KSB to support broader sustainability objectives and conducted systematic documentation, crafted the story of change and identified viable pathways for the implementation of Jurisdictional Programs, fostering learning and adaptability throughout the scaling process.

The urgent need to scale impact in critical sectors such as food security, agriculture, rural development, and climate change presents both challenges and opportunities. As evaluators, we are increasingly tasked with delivering rigorous insights to guide and support these scaling efforts.

Drawing from my professional experience, I have been deeply involved in scaling a community-based agricultural extension program and evaluating the impact of a climate-smart agriculture initiative across multiple regions. These experiences have provided me with valuable perspectives on the complexities inherent in scaling initiatives and underscored the critical role of robust evaluation in driving sustainable outcomes.