This is the last blog post on the Evaluative Inquiry, the new approach to research evaluation that CWTS has been developing since 2017, following one on broadening the concept of academic value, evaluating research in context and mixing methods.

In evaluations of any kind people often distinguish between summative and formative evaluations (see for example the classic Evaluator’s Handbook from 1987). Summative evaluations are carried out to ensure accountability for past work. They test whether investments have led to the desired outcomes in order to formulate statements about the effectiveness of policy instruments and to support decisions about the allocation of funding. Formative evaluations are, on the contrary, primarily concerned with learning for improvement: they assess processes and mechanisms in order to formulate recommendations to improve them.

Originally, the focus of the evaluation of research was on accountability. In order to justify investments in public research governments wanted to know the returns on these investments and university managers were looking for data and insights to inform the allocation of money among different faculties and research groups. Yet, evaluation reports implicitly also produced relevant lessons for the research units under evaluation, either in the form of recommendations, or as observations or interpretations of their activities that fed back into the group’s discussions about research directions, organizational matters or human resources.

While learning and accountability are different purposes and distinguishing between these is important, we do not see a contradiction between accountability and learning in the practice of research evaluation. Research evaluations can perfectly serve both purposes at the same time and, as said, implicitly often do. The evaluative inquiry explicitly works towards both purposes. It would be a waste of energy and resources to organize evaluations solely for accountability and not use them as a learning opportunity.

The latest rendition of the Dutch Strategic Evaluation Protocol (2021-2027) fortunately expanded the space for learning, focusing on goals and strategy rather than metric assessment. Academic research units are now evaluated in terms of research quality, societal relevance, and viability with special attention for Open Science, PhD policy and training, academic culture and HR policy. The groups under evaluation themselves decide which indicators they want to be evaluated on. Moreover, research evaluation is a crucial part of quality assurance, which is now a topic of yearly conversations with the board of the institute. All these elements of the current SEP protocol clearly promote self-reflection and learning as part of a strategy towards viable, relevant and well-recognized and used research.

The Evaluative Inquiry most directly feeds into the first phase of the evaluation process when the academic unit needs to prepare its self-evaluation document. Insights are helpful much beyond, however, most notably for research strategy and organizational planning. Carrying out a self-evaluation under the new SEP can be quite demanding for research groups. Firstly, they have to make their aims and strategy explicit. As research aims and strategy of many research groups are implicit, making this explicit for a SEP evaluation requires work that an external party can help with. Secondly, the unit is expected to reflect in a coherent, narrative way on how it actually performs and organizes its research to achieve its strategic aims. Our experience is that many groups struggle with this. Describing performance in the form of a narrative is not what academic environments are used to. Especially the integration of research quality and societal relevance in one text can be challenging. Lastly, as organizations of any kind, academic units have cultures and are collectives of people with different opinions. Bringing in external research evaluation specialists helps to facilitate the putting together of the self-evaluation document as well as the larger conversation around the missions and ambitions of the collective.

The Evaluative Inquiry supports academic units in crystalizing goals, missions, visions and strategy, taking stock of the diversity of output, making the multitude of stakeholder relations visible, and listing staff opinions about the academic organization. Building on a combination of scientometric and qualitative analyses this work helps to make visible whether what people say what they do (their missions and strategies), is in fact what they do (their output and their collaborations), and whether their academic organizations are aligned with this (SWOT). The Evaluative Inquiry helps to navigate the academic unit through this inspiring and sometimes thorny process towards a solid self-evaluation document as well as better informed research strategy and organizational planning more generally.