The need to evaluate science, technology and innovation in a development context
The significance of science, innovation and research in supporting global efforts towards more sustainable and climate-friendly development is growing. There is an urgent need for relevant science, quality research and innovations that are ground-breaking, as the world is experiencing new and unprecedented challenges and crises. Evaluation of the quality of evaluation and research is essential in determining the usefulness of and effectiveness of science, innovation and research activities. Evaluation findings should help decision-makers in determining important priority areas for further investigation and facilitate the decision-making on allocating resources for future research activities.
Key limitations
The evaluation of science and research is, however, quite complicated, and facing numerous methodological challenges. For example, assessment of the relevance and significance of scientific or research products is mostly based on the use of bibliometric methods. This is a quantitative method that may in fact produce solid evidence-based findings, yet its use is constrained by major limitations. For example, not all science, innovation and research products are included and properly recorded in the bibliographic databases, or not even published, hence not all products can be assessed.
The bibliometric methods often are based on calculating average number of citations, which also presents the basis for some biases. For example, there is sometimes an overly exaggerated attention to a specific author, who is known for previous work in a specific field or is affiliated with institutions that have strong political or financial support. In terms of citations, some authors may also deliberately exclude certain reference materials from their publications. Henceforth, whenever the bibliometric data analysis is used, it should be used with caution and should be combined with the use of other methods for validity purposes.
The other major limitation is that in today’s complex world of science and innovation, there are various standards or criteria of assessing quality of research, science and innovation in various parts of the world, and various parts of science and innovation.
Assessment of science and research products may also be biased due to the differences in political affiliation, beliefs, culturally or religiously-based perceptions of those who undertake these assessments or evaluations.
Key considerations
As this stream of evaluation function is still evolving, there are few key considerations that need to be taken into account in undertaking relevant evaluation, or in developing appropriate evaluation tools and methods.
Assessing relevance/significance of science and research.
Assessment of relevance or significance of science, innovation and research products need to take due consideration to the context in which these products are to be used. What works in one context may not be suited for the other, and what constitutes innovation and ground-breaking science varies substantially depending on the intended use or users.
Assessing effectiveness (or quality)
In assessing the effectiveness of the research and scientific analysis, the key is to assess “influence” of these activities, or extent to which the science, innovation and research products have influenced the policies, approaches or processes.
Assessing the degree of “networking”, i.e. the degree to which the researchers and scientific institutions have interacted with all relevant stakeholders, including those that may have had a “negative” or opposing stance to the subject research theme/topic.
Assessing Transformational nature
In today’s world, perhaps, the most important criteria for assessing the relevance, use and efficiency of science, innovation and research activities, is whether these activities cause truly transformational change, or at least trigger important policy discourse on moving towards such transformational change.
The above-written are suggestions for consideration, that are aimed at stimulating further feedback into this important discussion.
RE: How to evaluate science, technology and innovation in a development context?
The need to evaluate science, technology and innovation in a development context
The significance of science, innovation and research in supporting global efforts towards more sustainable and climate-friendly development is growing. There is an urgent need for relevant science, quality research and innovations that are ground-breaking, as the world is experiencing new and unprecedented challenges and crises. Evaluation of the quality of evaluation and research is essential in determining the usefulness of and effectiveness of science, innovation and research activities. Evaluation findings should help decision-makers in determining important priority areas for further investigation and facilitate the decision-making on allocating resources for future research activities.
Key limitations
The evaluation of science and research is, however, quite complicated, and facing numerous methodological challenges. For example, assessment of the relevance and significance of scientific or research products is mostly based on the use of bibliometric methods. This is a quantitative method that may in fact produce solid evidence-based findings, yet its use is constrained by major limitations. For example, not all science, innovation and research products are included and properly recorded in the bibliographic databases, or not even published, hence not all products can be assessed.
The bibliometric methods often are based on calculating average number of citations, which also presents the basis for some biases. For example, there is sometimes an overly exaggerated attention to a specific author, who is known for previous work in a specific field or is affiliated with institutions that have strong political or financial support. In terms of citations, some authors may also deliberately exclude certain reference materials from their publications. Henceforth, whenever the bibliometric data analysis is used, it should be used with caution and should be combined with the use of other methods for validity purposes.
The other major limitation is that in today’s complex world of science and innovation, there are various standards or criteria of assessing quality of research, science and innovation in various parts of the world, and various parts of science and innovation.
Assessment of science and research products may also be biased due to the differences in political affiliation, beliefs, culturally or religiously-based perceptions of those who undertake these assessments or evaluations.
Key considerations
As this stream of evaluation function is still evolving, there are few key considerations that need to be taken into account in undertaking relevant evaluation, or in developing appropriate evaluation tools and methods.
Assessing relevance/significance of science and research.
Assessment of relevance or significance of science, innovation and research products need to take due consideration to the context in which these products are to be used. What works in one context may not be suited for the other, and what constitutes innovation and ground-breaking science varies substantially depending on the intended use or users.
Assessing effectiveness (or quality)
In assessing the effectiveness of the research and scientific analysis, the key is to assess “influence” of these activities, or extent to which the science, innovation and research products have influenced the policies, approaches or processes.
Assessing the degree of “networking”, i.e. the degree to which the researchers and scientific institutions have interacted with all relevant stakeholders, including those that may have had a “negative” or opposing stance to the subject research theme/topic.
Assessing Transformational nature
In today’s world, perhaps, the most important criteria for assessing the relevance, use and efficiency of science, innovation and research activities, is whether these activities cause truly transformational change, or at least trigger important policy discourse on moving towards such transformational change.
The above-written are suggestions for consideration, that are aimed at stimulating further feedback into this important discussion.
Kindest regards,
Serdar Bayryyev
Senior Evaluation Officer