Building knowledge infrastructure to support research assessment reforms: Dispatches from the Dutch Recognition and Rewards Festival 2022

Building knowledge infrastructure to support research assessment reforms: Dispatches from the Dutch Recognition and Rewards Festival 2022

In this blog post we reflect on knowledge infrastructures currently emerging to support organizations that are pursuing research assessment reforms, and call on sociologists of science and research evaluation researchers to study and contribute to these unfolding developments.

How can academic research systems enable more diverse profiles and career paths for academics? How can research assessment better recognize quality, content, creativity, and social relevance of research? Can academics be better rewarded for individual and team-based contributions, where appropriate? These were some of the questions at stake at the second annual Recognition and Rewards Programme event in the Netherlands on 4 February 2022, moved online due to coronavirus.

Combining elements of bottom-up and top-down coordination, the Recognition and Rewards initiative is a coalition of research funders, universities, university medical centers, and public research institutes that aim to reform and diversify academic career and research quality assessment. Since the publication of the position paper, Room for everyone’s talent: towards a new balance in the recognition and rewards of academics in 2019, Dutch Universities have pursued local action plans and some have begun implementing changes in pursuit of Recognition and Rewards’ vision.

As a nationally coordinated effort, Recognition and Rewards is one of the first of its kind (see also initiatives in Norway and Finland), providing a fascinating natural experiment on what happens when large-scale reforms of a research system are introduced. Being one of the first into unchartered territory though means the Dutch research system has some ‘first mover disadvantages’, as those investing in reforms lack existing stocks of knowledge and experience to inform their change efforts. Alongside bottom-up changes being implemented locally, the Recognition and Rewards programme is seeking to build shared infrastructure to guide and monitor individual changes and draw out common lessons. Events such as the annual festival are one such example of infrastructure, as they provide an important moment to pause and reflect on challenges and more generally to forge communities of practice that share knowledge on common challenges that cut across organizations.

The Festival itself took place over a day, and consisted of roundtable discussions by senior stakeholders and parallel stream workshops put on by different research system actors. Our own team hosted a session on the challenges of implementing complex changes in organizations, using fictional vignettes of university leaders introducing narrative CVs and portfolio assessment tools. Clearly such interventions cannot ‘resolve’ complex issues around which different meanings and conflicts of interest emerge. Occasions such as online workshops, can though provide opportunities for common challenges to be surfaced, discussed, and acknowledged, thereby supporting learning and networking among different actors in the research system as they approach their respective change efforts.

Broadening out, with its Recognition and Rewards programme of reforms, Dutch Universities have a unique opportunity not only to be at the forefront of assessment reforms, but also to cultivate communities of practice and build shared knowledge about implementing, scaling and sustaining reforms from which others can benefit. Building this knowledge is not easy, as it involves dedicating potentially scarce resources to evaluating and monitoring interventions and being open to sharing findings with one’s perceived competitors and opening oneself up to scrutiny from the wider world. But these are costs that should be worth bearing.

Alongside the Dutch efforts, there are other signs of emerging cross-national knowledge infrastructure to inform and guide academic institutions pursuing assessment reforms. The Recognition and Reward Festival itself took place on the same day as the Paris Open Science European Conference (OSEC) 2022 focusing on research assessment reforms in the European Union. 160 organizations have signed up to declare interest in the European Commission’s action plan to implement research assessment reforms. CWTS, Leiden University, where the three of us work, is currently partnering with the San Francisco Declaration on Research Assessment (DORA) on three year project funded by Arcadia, a charitable fund of Lisbet Rausing and Peter Baldwin, to build community resources to support assessment reform practitioners internationally, including building a toolkit and an online dashboard to display and visualize information around novel assessment activities in the context of hiring, promotion and tenure decisions. The dashboard will complement an existing raft of resources that the DORA website has established in recent times, including case studies, blogs and tools to develop change management. Professional research management associations like International Network of Research Management Societies (INORMS) Research Evaluation Group (REG) have also been actively developing resources to guide organizational change, including the SCOPE Framework.

Much is happening. Yet, two overlapping research communities that have been surprisingly quiet on research assessment reforms are the fields of sociology of science and the multi-disciplinary social science field of research evaluation. This is alarming, as a rigorous, theoretically informed academic knowledge base about novel assessment tools and practices is urgently needed. Participating in the Recognition and Rewards Festival it struck us these are just the kinds of collective engagements in which such a knowledge base should be playing a valuable role. Given quite dramatic changes are underway in core problem areas of their field – research assessment - our message to colleagues and peers in the sociology of science and research evaluation is clear: it’s time for us to step up.