1. What do you think are the challenges in evaluating quality of science and research?
In the case of CGIAR, perhaps consolidating and adopting a standardised approach to measure QoS beyond the single appraisal by each CGIAR centre. This could help measure better the outcomes, assess effectiveness, improve data quality, identify gaps, and aggregate data across centres.
a. What evaluation criteria have you used or are best to evaluate interventions at the nexus of science, research, innovation and development? Why?
Within MEL, we have used in the past a combination of quantitative and qualitative methods. For instance, bibliometrics (e.g., indexing the Web of Science Core Collections, percentage of articles in Open Access, ranking of journals in quartiles, Altmetrics) for published journal articles and Outcome Impact Case Reports (OICRs) to describe the contribution of CGIAR research to outcomes and impact.
We have also recently embarked on a study that uses social network analysis (SNA) and impact network analysis (INA) to investigate the research collaboration networks of two CGIAR Research Programs (CRP). Networks are generated based on journal articles published over the course of four years and their metadata is used to explore aspects ranging from team structures to the evolution of collaborations between organisations.
In general, a mix of quantitative and qualitative methods could be the most useful strategy, allowing for a combination of different approaches and metrics to measure the impacts of interventions.
b. Could a designated quality of science (QoS) evaluation criterion help capture the scientific aspects used in research and development?
Not a single but a combination of different criteria might be the best. We have learned from the work of Rünzel, Sarfatti, Negroustoueva (2021) the usefulness of using relevance, scientific credibility, legitimacy, and effectiveness within the framework for evaluating the Quality of Research for development.
RE: How to evaluate science, technology and innovation in a development context?
1. What do you think are the challenges in evaluating quality of science and research?
In the case of CGIAR, perhaps consolidating and adopting a standardised approach to measure QoS beyond the single appraisal by each CGIAR centre. This could help measure better the outcomes, assess effectiveness, improve data quality, identify gaps, and aggregate data across centres.
a. What evaluation criteria have you used or are best to evaluate interventions at the nexus of science, research, innovation and development? Why?
Within MEL, we have used in the past a combination of quantitative and qualitative methods. For instance, bibliometrics (e.g., indexing the Web of Science Core Collections, percentage of articles in Open Access, ranking of journals in quartiles, Altmetrics) for published journal articles and Outcome Impact Case Reports (OICRs) to describe the contribution of CGIAR research to outcomes and impact.
We have also recently embarked on a study that uses social network analysis (SNA) and impact network analysis (INA) to investigate the research collaboration networks of two CGIAR Research Programs (CRP). Networks are generated based on journal articles published over the course of four years and their metadata is used to explore aspects ranging from team structures to the evolution of collaborations between organisations.
In general, a mix of quantitative and qualitative methods could be the most useful strategy, allowing for a combination of different approaches and metrics to measure the impacts of interventions.
b. Could a designated quality of science (QoS) evaluation criterion help capture the scientific aspects used in research and development?
Not a single but a combination of different criteria might be the best. We have learned from the work of Rünzel, Sarfatti, Negroustoueva (2021) the usefulness of using relevance, scientific credibility, legitimacy, and effectiveness within the framework for evaluating the Quality of Research for development.
Rünzel, M., Sarfatti, P. & Negroustoueva (2021), Evaluating quality of science in CGIAR research programs: Use of bibliometrics