In the first of a new series of knowledge-sharing sessions open to members of the Community, participants shared how they have adapted their evaluation practice to cope with restrictions imposed by the pandemic.
The extensive and rapid spread of Covid-19 has shifted much evaluation work to remote modalities and raised new challenges and uncertainties for both managers and evaluation teams [1].
Renate Roels, Evaluation Manager at FAO, opened the discussion by talking about sharing the unexpected issues she faced and how she addressed them, including managing teams and expectations remotely, as well as building trust.
Below are the key areas of concern and practical solutions identified by participants during the discussion.
Remote data collection
Participants found themselves using their creativity to adjust to working remotely, looking for new ways to collect data from the field and to reach out to beneficiaries living in remote areas. This way of working certainly requires us to pay more attention to how we engage stakeholders, to planning, and to finding the interactive tools most suited to a given team and context.
Compared to face-to-face exchanges, online meetings need to be planned. Evaluators need to make sure that technology and connection issues don’t get in the way of information gathering and a participatory approach.
In the words of one participant: “The main challenge in the project I was working on was that the beneficiaries are from the remotest part of the country and they had no access to communication tools/devices let alone internet, some of them didn't have mobile phones to communicate.”
It was suggested that going back to old-school phone interviews may be a more direct and easier way to reach out to people than email interaction. People may have a phone, especially in local communities, even if they are not connected to the Internet. The instant-messaging function was also mentioned as a commonly-used tool with a great deal of potential, when its use is well organized. “Surveys can be easily shared via WhatsApp,” one person commented.
Participants recommended other technologies which offer practical tools for remote data collection and sharing. They include: KoBo www.kobotoolbox.org and ONA Data https://ona.io/home for data collection, while Mural www.mural.co and Lino http://en.linoit.com are online canvases or bulletin boards for digital collaboration (anonymous inputs) or facilitated meetings, which can be used in combination with a preferred video-conferencing platform, like the popular Zoom.
“KoBo is great - you can set up questionnaires for the enumerators, and they can add photos and GPS coordinates. When they are in the signal, it goes to the cloud. The advantage is that the data is collected immediately - doesn't rely on enumerators typing it up later. So the first night you can check the answers and ensure that everyone has understood the questions properly. Then you can change the question if there is a problem - you don't have to wait to the end of the survey,” one participant said.
Other tools mentioned were: Evalmapper, www.alnap.org/help-library/evaluation-map, a digital map made available by a network dedicated to improving the performance of humanitarian action through shared learning (ALNAP). It is designed to support the search for evaluation evidence, using information from previous evaluations. Geographic Information Systems (GIS), the free and open-source Open Foris software and the Collect Earth toolset www.openforis.org allow users to analyse satellite images for a variety of purposes, including monitoring agricultural land and urban areas.
Building trust
Another key challenge in remote evaluation is building trust while communicating across a computer screen with team members and counterparts. Missing those informalities that happen before or after a meeting, while sharing a meal, or on a bus ride, inevitably generates a more sterile environment.
One way to address this challenge is to create “safe spaces,” for instance, by making it clear that conversations on Zoom or other tools are not recorded.
However, most felt that there is no replacement for a full, in-person engagement. As a participant highlighted: “I still think we will miss out on the casual meetings in the field. Not just the organized focus groups - but chatting on the side with community members or talking with local government representatives in the car.”
Another way to tackle the distance which Renate shared with the group was to virtually “visit” and get to know the project area using Google Street View (www.google.com/streetview). The visual representation of the surroundings of the project helped her to understand the project environment and opened the way to a deeper level of interaction, based on trust.
Dealing with health risks
Sending a consultant to the field and organizing focus groups or meetings carries the risk of Covid transmission. Some participants shared their anxiety and expressed worry at seeing photos from field visits where people did not always respect social distancing or wear masks.
One proposed solution was to provide simple and easy-to-use guidelines to consultants and community leaders but this may not always work: “In our Nepal project we insist on our own staff using masks and other strict measures, like distancing, etc. We try to have meetings outside wherever possible. But many community members don't see the point.”
Participants mentioned the importance of having a regular and open dialogue with team members and local consultants to voice their concerns and share their difficulties in the field. They said team members should be ready to change plans and shift to fully remote modalities at any time.
Risk of overburden
Some participants mentioned feeling guilty about approaching people with requests for necessary data and information. They had to interact with tough situations in the field, people overwhelmed by work, worries, pain, psychological stress, and virus-related restrictions and risks. Stress and underperformance due to psychological or physical problems and changes in teams due to the pandemic affected evaluation work. Once again, regular communication with team members was suggested.
Planning and adjusting
Proper planning can go a long way to limiting the impact and risks of such challenges. What information is really needed? Can it be found elsewhere? In other words: “Choose the best way to collect what you need, it may not be necessarily an interview, or an email.”
In this sense, triangulation can become extremely challenging. One way around this is to be honest about the limitations and use different terminology, for instance, talking about “examples” instead of “best practices”.
One way forward suggested was for the evaluation manager or team leader to act as a sounding board, checking in with team members frequently and not leaving them alone.
New opportunities
In reflecting on how work has changed over 2020, it became clear that opportunities to improve have emerged. For instance, expanding the recruitment and responsibilities of local consultants promotes strengthening evaluation capacities at the national level.
Online meetings have also widened the spectrum of attendees, making it possible to invite stakeholders from across regions, while reducing travel-related costs, and lowering our carbon footprint. Said one participant: “The COVID-19 experience enabled us to access a caliber of interviewees that we might not have had access to.”
---
The EvalForward talks aim to convene informal discussions among members for peer-to-peer support and to voice their challenges, concerns and lessons in an open and friendly manner.
Please reach out if you wish to propose the next session.
[1] see also the EvalForward blog series Evaluations in the time of Covid 19
Norah De Falco
IFADThe Independent Office of Evaluation of IFAD (IOE) has recently released two products detailing its practical experience with conducting evaluations under the travel, health and social restrictions established internationally to fight the spread of Covid-19. These restrictions had a severe impact on IOE’s work, preventing international missions and field data collection – signature elements of its evaluation approach.
The learning note on the evaluation of the “Coastal Climate-Resilient Infrastructure Project” (CCRIP) in Bangladesh focuses on how IOE conducted its first fully remote evaluation, discussing the adaptations made to its traditional evaluation approach – including the leveraging of national consultants and modern technologies – and detailing the lessons learned. The video on the upcoming CCRIP evaluation builds on the above learning note and features interviews with Roberto La Rovere and Sally Smith, International Consultants, and Fabrizio Felloni, Interim Officer-in-Charge, IOE.
We hope that these products will provide fruitful evaluation lessons during the pandemic and beyond.
For further information please write to evaluation@ifad.org.
Pamela Dianne White
Senior Manager Cowater International Finland OyThis was a very enjoyable and useful session - thanks!