RE: How to evaluate science, technology and innovation in a development context? | Eval Forward

There are several innovative tools to leverage on bibliometric data for insights beyond citation metrics, which can give more depth to the evaluation of science - as pointed in other contributions, social network analysis techniques can be applied to identify networks of scientific collaboration or of funding institutions on a given topic. 

Furthermore, recognising that traditional and novel metrics (such as altmetrics) tend to focus on the significance of research within the scientific community and fail to unpack reach beyond researcher networks, we have recently published a paper in which we used a digital methods approach to evaluate the "soft" forms of influence of climate science in the policy space. Our study proposed a framework to assess the broader influence of the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) through online media representations. 

By considering online networks and narratives as evidence of influence, we repurposed publicly available digital artifacts to assess CCAFS' reach among stakeholder at various levels. We assessed the dynamics of information diffusion, interaction and discourse amplification as representations of how CCAFS supported policymaking at various levels of engagement, as opposed to simply examining ‘formal’ policy outputs reported in monitoring mechanisms, or academic impact metrics.

Focusing on the importance of translating climate science into actionable policy, Google Trends, hyperlink analysis, network analysis, and text mining were applied in an integrated framework to assess the centrality and influence of CCAFS in the climate science-policy interface. Our approach points to the possibility of leveraging on data-driven methods to assess interactions and estimate influence. As such, the development of comprehensive evaluation frameworks by climate research programs requires the establishment of indicators that capture influence from a holistic perspective, not only based on traditional indicators, but also in relation to messaging, visibility, knowledge exchange and engagement.

Reference: 

Carneiro B., Resce G., Läderach, P., Schapendonk, F., Pacillo, G. (2022) What is the importance of climate research? An innovative web-based approach to assess the influence and reach of climate research programs, Environmental Science & Policy, 133, https://doi.org/10.1016/j.envsci.2022.03.018