“Fund and forget”, that is how Jordi Molas-Gallart from the research centre Ingenio (CSIC-UPV) in València teasingly characterized the way large national research funders often operate. It’s also a point of departure for thinking about new ways of research evaluation. The cornerstones of many forms of research evaluation are peer review and bibliometric analysis. Interestingly, peer review as practiced by evaluation committees, on closer inspection typically consists of judgement based on intimate entanglement of peer review and information based bibliometric indicators, to the point that the two are often hard to separate.

What research evaluation currently looks like is relatively clear, the road ahead is a little less so. But there is a shift visible towards formative and participatory forms of evaluation. One key assumption in those new approaches is that the traditional distinction between research evaluators, the organisations they are evaluating and societal stakeholders can no longer hold: intensive interactions between the three can help to make evaluations more relevant, more productive and more effective in terms of learning. Co-production models stress the need for contribution from stakeholders throughout the whole evaluation process, with the aim of creating synergy between the various people and groups involved. And ideally the use of a co-production model results in stakeholders experiencing a shared responsibility for the outcomes.

How stakeholder involvement or ‘co-production’ works out in practice was the topic of an online workshop, organised on June 4^th by CWTS, as part of its ‘Responsible Evaluation’ and ‘Engagement and RRI’ thematic hubs. The close to thirty participants were drafted from CWTS and various (research) organisations involved in or thinking about research evaluation.

The workshop started with three brief presentations about ‘new’ forms of evaluation. Pierre-Benoit Joly (INRA) introduced a novel method for real-time assessment of research impact, building on the ASIRPA tool that has become mainstream in the French agricultural research institute INRA. Jordi Molas-Gallart (Ingenio, CSIC-UPV) introduced a formative approach to the evaluation of transformative innovation policy. Tjitske Holtrop (CWTS) introduced the main principles of Evaluative Inquiry, a multi-method approach to analyse the interaction between knowledge production and societal practices. In breakout rooms participants focused on the (possible) role of societal stakeholders in evaluation, which brought about discussions in many directions.

More questions were raised than answered, as expected in a field that is contested and quickly developing. For example, there seems to be a blurring of boundaries between formative evaluators and co-creation partners. And if the distance between evaluators and research performers gets smaller, does that leave enough room for ‘independent’ critical feedback? Or should we define an inner and outer circle of stakeholders that are differently involved in research evaluation? If a research organisation has only a single large stakeholder, which also is involved in evaluating that organisation, questions of academic freedom or conflicts of interest may arise.

Or should we even go beyond involving stakeholders as additions to evaluation, and instead thinking about mutual benefits by the continuous inclusion of relevant stakeholders with the research? Then a co-production type of evaluation may become real-time, instead of ex post or ex ante. In such a configuration, one can even wonder what the difference is between an evaluator and a partner in knowledge co-production.

Some research funders are experimenting with ex ante involvement of stakeholders in reviewing research proposals, but this is not widespread and review is still dominated by the scientific community.

We hope that this initial workshop can function as a stepping stone towards a broader dialogue about the opportunities and limitations of formative and participatory research evaluation. The feedback we received from participants suggests that there is a need for more debate and exchange.

As a direct follow-up we consider to organize workshops on more specific themes in this area, such as:

• How to deal with relationships between scientific and societal value in institutional research evaluation?
• What are the opportunities of formative and participatory methods in natural sciences versus social sciences and humanities?
• What is the potential of real-time impact assessment for research funding programmes?

The original blog post included a paragraph about the Dutch Strategy Evaluation Protocol (SEP), but that was removed because it unintendedly suggested that the SEP does not encourage involvement of stakeholders.