RE: How to evaluate science, technology and innovation in a development context? | Eval Forward

I have participated in two Research Excellent framework exercises in the UK (2014 and 2021) which assess the research excellence of many diff disciplines and institutions. Alongside assessing research outputs, it also evaluates impact case studies and the research environment producing a blended score with outputs having the biggest weighting but impact growing in weighting from 2014-2021. 

It is mainly through peer-review but does include bibliometrics (undertaken centrally and provided to you so all done in the same way). Bibliometrics can be "used" to enhance a score as opposed to bringing a score down.

Interestingly the same researchers, institutions score well across the board although one does get "pinnacles" in some specific research areas and/or higher impact etc. Thus, those scoring large amounts of 3 and 4 * (international excellence and world leading) often have high scoring impact and/or research environments too as you might expect.

The above benefits from a substantive period prior to REF of the RAE and lots of information known about baselining/normalising. For example, in ag/food/vet (my panel), we had information on typical citations etc for the sub-topic etc and thus could see if above/below the norm alongside our own peer-review.

More can be found here: for 2021 and here for the 2014 exercise which has been analysed by many and there is quite a science on REF and informing best ways of doing it moving forward.

It has an incentive both financially and for reputation which means "its a big deal" for UK universities whom spend lots of time and money on it. 

I do think the breadth of coverage and consideration of rigour, originality and novelty etc for outputs and reach/significance and impact for Impact case studies and then a range of metrics on research culture for research environment offers a broad view of excellence and the fact its judged for the topic and in relation to others undertaking work on that topic etc. means it is very thorough and resilient - however, it takes a lot of time and money to do it. There remains a debate as to how much could be done using algorithms and not peer-review in the more STEM subjects.