John [user:field_middlename] Hoven

John Hoven

independent
United States of America

Process tracing (a leading method of qualitative causal inference) is designed to investigate cause-and-effect in one-of-a-kind situations where unknown unknowns are important factors. My personal focus is applying this to design, implement, and evaluate development and peacebuilding projects.

My contributions

  • How are mixed methods used in programme evaluation?

    Discussion

    • John Hoven

      independent
      United States of America

      Qualitative methods are often descriptive. Has anyone used qualitative causal inference?


      The goal of qualitative causal inference is to prove cause-and-effect, either looking back into the past, or forward into the future. My sense is that this approach relies heavily on unscripted interviews, where undiscovered issues are revealed by follow-up questions. (What do you mean? Can you give an example?)

  • Reporting and supporting evaluation use and influence

    Discussion

    • John Hoven

      independent
      United States of America

      Olivier Cossee says, “evaluators need to propose reasonable solutions to the problems they raise… the hard part is to propose something better, in a constructive manner.”

      A reasonable solution is, at least implicitly, a Theory of Change. It should be explicit: a tentative goal, a first step toward the goal, and some intermediate steps. The hardest part is that first step. Taking that first step should answer the question, “Might this be worth considering?”

      John Hoven

  • Can we use an evidence-based, evolving Theory of Change to achieve "local learning” during project design?

    Discussion

    • John Hoven

      independent
      United States of America

      Dear all,

      Let me close this discussion topic with some reflections on: What have we learned? What should we learn next?

      Two things we have learned are:

      1. Large-scale projects can be customized to local desires and capabilities. CGIAR does this, to ensure that farmers adopt their agricultural research.

      2. An evolving ToC can be done with hardly any money or skill. (See attachment.) This allows locals to be full partners in a village-level development project. 

      Three things to learn next:

      1. Donors can offer contracts that do not specify actions, outcomes, or indicators of success. (See Grandori et al. and Reuer.) That would remove a major obstacle for local learning. Businesses do this when contracting for innovation, because the actions and desired outcomes aren’t known when the contract is signed.

      2. Nested ToCs let you zoom in to see more detail, like internet maps. Close-up ToCs feature a specific village, product market, or social group. They evolve rapidly during the start of a project (every week or two, not every 6 months or a year). Zoom out to see a ToC with less detail (actors are categories rather than named groups). This categorical ToC can become a first guess ToC for a new project (e.g., Figure 2, page 5 in the Community-driven development evaluation by IFAD).

      3. An evolving ToC gets revised when evidence strongly disproves one step in the chain of cause-and-effect, or it is confirmed when evidence strongly validates the step. That evidentiary proof delivers the accountability that donors require.

      Thank you all for your contributions. Please feel free to email me with additional thoughts.

      John Hoven

      jhoven@gmail.com

      Evolving ToC Feb 2021.pdf

    • John Hoven

      independent
      United States of America

      It seems that both at individual and organizational level there are attempts to ensure the ToC is not “cast in stone” as mentioned by Jackie Yptong, and that there are some good examples of using ToC for learning like in CGIAR. 

      I thank all of you for your contributions and describing how you or your organization are using ToCs.

      I would like to go a little further and ask you if you know actors / donors that would be ready to start projects at local level with no assumptions and to develop a ToC as they develop their understanding of what is needed on the ground? 

      In this case the project would not start with pre-defined outputs but only with a general / broad outcome, the causal pathways to reach it not defined yet. 

      Below my feedback to Erdoo and Janvier and follow up questions for their consideration.

      Erdoo Karen Jay-Yina says that CGIAR agricultural research programs are learning to use Theories of Change more effectively. Their ToCs focus on “the mechanisms of change by which the new agricultural product gets adopted by a farmer. Can farmers use new technologies? Do they even want to? TOCs should identify the mechanisms of change based on evidence and testable hypotheses. Stakeholder farmers should be involved from the outset of the research.” (ISPC 2012 pp. 14, 23, 7, 25) Erdoo says, “When the underpinning ToC and the evidence are revisited, captured and tracked coherently, then process tracing or contribution analysis of particular causal pathways is made easier.”

      My follow-up questions for Erdoo:

      Does CGIAR use process tracing to make evidence-based predictions? Have you encountered others in the development / peacebuilding community that are using ToCs as a learning tool?

      Janvier Mwitrehe cites two reviews of USAID’s use of ToC. In Tanzania, “USAID/Tanzania did not anticipate the need to revisit the foundational Theory of Change. However, after its second year, it became clear that the original Theory of Change and the reality of implementation were not aligned. Some of the activities were not implementable, due to changes in the local context. The original Theory of Change was a binding constraint to the Activity’s successful implementation.” A 2019 review of TOC as an Adaptive Management Tool confirms USAID’s use of ToC as a contractual binding constraint: “The main purpose of a TOC review is to ensure alignment of the TOC with the goals you are trying to address. Factors that might prompt a special review of a theory of change include failure to influence the next level outcome as expected, previously unknown causal pathways, and significant changes in the political or environmental conditions of the local context.”

      I think these two examples show that for USAID, a ToC is a contractual binding constraint rather than a tool for learning. My new follow-up question is this: Janvier, are you aware of any discussion within USAID of the need to adapt to a local context, rather than just adapting to changes in the context?

      John Hoven

    • John Hoven

      independent
      United States of America

      What I love about this forum is that it brings out such a broad array of perspectives. Let me summarize briefly, and suggest a way forward on each perspective.

       

      The issue is using an evidence-based, evolving ToC to design a localized project.

       

      Jean Providence Nzabonimpa offered some compelling reasons to embrace the concept: “There are important factors unknown at the design stage of development interventions… Keeping the ToC intact throughout the life of a project assumes most of its underlying assumptions and logical chain are known in advance and remain constant. This is rarely the case… Assume X outputs lead to Y outcomes. Later on one discovers that A and B factors are also, and more significantly, contributing to Y… A project which discovers new evidence should incorporate it into the learning journey.”

      Follow-up question for Jean Providence: Can you describe a specific project that illustrates your point? Do you know anyone who might use an evidence-based, evolving ToC to design a localized project?

       

      Serdar Bayryyev highlights “community-driven development” projects, which focus on social capital and empowerment. A case study review of these projects used a theory of change based on the assumption that a participatory implementation process supports people-centered development processes.

      Follow-up question for Serdar: Have you seen a community-drive development project that used an evidence-based, evolving ToC to design a localized project? Do you know anyone who might give this a try?

       

      Janvier Mwitirehe says that “we operate in fast evolving environment that need to be considered.” This can be done through USAID’s “collaborating, learning and adapting” (CLA) framework, which says that “critical assumptions central to a TOC must be periodically tested – which is a central feature of assumption-based planning.”

      Follow-up question for Janvier: Suppose a project environment is poorly understood, but not rapidly evolving during the first 6 months that the project staff interacts with local people. Will the CLA framework help the project staff design a ToC based on local evidence rather than assumptions? Is USAID receptive to using an evidence-based, evolving ToC to design a localized project?

       

      Carlos Tarazona, Senior Evaluation Officer FAO, says that “In the FAO Evaluation Office we have used Theory of Change (ToC) … for evaluation purposes only.”

      Follow-up question for Carlos: Have you seen an evidence-based, evolving ToC used for real-time evaluation? If someone wanted to use an evidence-based, evolving ToC to design a localized project, could they get helpful advice from an expert in real-time evaluation?

       

      John Hoven