Various reform movements are working hard to improve the research system, focusing on issues ranging from research assessment, diversity and inclusion, and research culture to scientific publishing, open science, reproducibility, research integrity, and several others. The agendas of the different reform movements are interconnected and partly overlapping, but sometimes they are also in tension. And while the different movements have the potential to reinforce each other, they also compete for scarce resources and for the attention of researchers and policy makers.

Being involved myself in two reform movements, the research assessment reform movement and the scientific publishing reform movement, I feel increasingly concerned about the limited coordination and collaboration between these movements. Although the two movements generally seem to be sympathetic to each other, they engage with each other only in superficial ways. This lack of coordination, I believe, is not only a missed opportunity; it is a threat to the success of both movements.

Publishing reform requires assessment reform

Broadly speaking, the publishing reform movement advocates for more open, inclusive, and efficient approaches to scientific publishing, and for scientific publishing to be controlled by research communities themselves rather than by external actors such as commercial publishers. As shown on the left side of Figure 1, this movement promotes ideas such as diamond open access, preprinting, modular publishing, open peer review, open data, and open code. It opposes the status quo in scientific publishing, which is strongly focused on publishing in prestigious high-impact journals that are supposed to publish only the highest quality and most important research. Publishing in such journals usually requires going through time-consuming non-transparent peer review processes. These journals typically also charge hefty fees for open access publishing.

It is widely recognized in the publishing reform movement that reform in scientific publishing is possible only if research assessment practices are reformed as well. Large-scale reform in scientific publishing will not happen as long as assessment practices keep incentivizing researchers to publish in prestigious journals and do not reward researchers for adopting alternative publishing approaches.

Assessment reform requires publishing reform

As shown on the right side of Figure 1, the assessment reform movement promotes approaches for assessing researchers and research units (e.g., research groups, departments, and institutions) that are aligned to the specific context and strategy of a researcher or research unit, that value all relevant contributions, and that rely primarily on qualitative expert judgment, with a supporting role for quantitative indicators. An important tenet of the assessment reform movement is the rejection of the use of journal impact factors and other journal-based information to assess researchers and research units. This principle goes back to the San Francisco Declaration on Research Assessment (DORA), published in 2013, which states that research needs to be assessed “on its own merits rather than on the basis of the journal in which the research is published”. Building on DORA, the Agreement on Reforming Research Assessment, published in 2022 and serving as the basis for the Coalition for Advancing Research Assessment (CoARA), includes a commitment to move away “from using metrics like the Journal Impact Factor” and to abandon “assessing outputs based on metrics relating to publication venue”. Despite some discussion, the idea that journal-based information should play no role in research assessments has become a foundational principle for the assessment reform movement.

Although the assessment reform movement rejects assessing research based on the journal in which it is published, the movement does not see it as its role to challenge the status quo in scientific publishing in more fundamental ways. It seems to accept, at least reluctantly, that in most research fields scientific publishing is done primarily in journals, that most journals use non-transparent closed peer review to decide what to publish, and that researchers aim to publish their work in journals that are as prestigious as possible, because these journals are supposed to perform the most rigorous peer review and to have the largest readership.

Accepting the status quo in scientific publishing while rejecting the use of journal-based information in research assessments leads to a schizophrenic situation for researchers. On the one hand, when publishing their research, researchers live in a world in which they are expected to opt for prestigious journals that are believed to perform robust quality control and to play an important role in bringing the research to the attention of the relevant research communities. On the other hand, when researchers are assessed, for instance in hiring and promotion processes, they find themselves in a world in which everyone is expected to disregard journal-based information and to pretend it does not matter where research is published.

This schizophrenic situation explains why it is so challenging for the assessment reform movement to eliminate the use of journal-based information in research assessments. Abandoning the use of this information is at odds with the status quo in scientific publishing and therefore is possible only if this status quo itself is abandoned. To eliminate the use of journal-based information in research assessments, a fundamental reform in scientific publishing is needed.

Bringing the two reform movements together

The research assessment reform movement needs to develop a clear perspective on reform in scientific publishing. Rejecting the use of journal impact factors and other journal-based information in research assessments is not sufficient. Research assessment and scientific publishing are interwoven in complex ways, which means the agendas of the assessment and publishing reform movements need to be aligned as closely as possible.

For instance, if the assessment reform movement favors qualitative expert judgment over reductionistic quantitative indicators, shouldn’t it then also favor open peer review of research articles over non-transparent closed peer review? In closed peer review, an in-depth evaluation of the strengths and weaknesses of an article is reduced to a dichotomous outcome: the article either is or is not published in a journal. When an author of the article is subsequently assessed, for instance in a hiring or promotion process, only the final dichotomous outcome of the closed peer review of the article can be considered. This pushes evaluators in the direction of reductionistic quantitative assessment approaches, for instance based on journal impact factors. Open peer review, on the other hand, offers much deeper insights into the strengths and weaknesses of an article, and these insights can feature into a nuanced qualitative assessment of an author.

The assessment reform movement also argues that all relevant contributions, not only peer-reviewed journal articles, should be valued in research assessments. A non-peer-reviewed article on a preprint server or a non-peer-reviewed data set in a repository can be as relevant as a peer-reviewed article in a journal, and these contributions should be valued accordingly. Likewise, performing peer review can be a relevant contribution in itself and should be valued as such. Importantly, valuing these contributions is possible only if publishing practices are reformed and preprinting, open data, and open peer review become standard ways of working.

Scientific publishing and research assessment are also in competition for an extremely scarce resource: the expertise of researchers who are willing to serve as peer reviewers in the evaluation of research outputs, researchers, and research units. Peer review is already overburdened, and the ambition of the assessment reform movement to increase the use of peer review in research assessments poses the risk of overburdening peer review even more. However, this risk can be mitigated through close coordination between the assessment and publishing reform movements. For instance, the publishing reform movement promotes preprinting, open peer review, and the publish-review-curate model, all of which help to reduce the pressure on peer review (e.g., by allowing reviews to be reused and by enabling articles to be published without peer review). This offers a way to free up the much-needed capacity to realize the ambition of the assessment reform movement to rely more strongly on expert judgment in research assessments.

In conclusion, while it is widely acknowledged that scientific publishing reform will fail if it is not supported by reform of research assessment practices, the opposite also applies: assessment reform will fail without reform in scientific publishing. Assessment reform and publishing reform therefore need to go hand in hand. Close collaboration between the two reform movements is crucial.

Ludo Waltman studies assessment and publishing reform in various projects at CWTS and in the Research on Research Institute (RoRI). As Open Science Ambassador of Leiden University, he promotes assessment and publishing reform at the university. Ludo is Editor-in-Chief of the MetaROR (MetaResearch Open Review) publish-review-curate platform. He also serves as President of ASAPbio.
The author is grateful to Katie Corker, Thomas Franssen, and Lizzie Gadd for helpful feedback on an earlier draft of this blog post.
DOI: 10.59350/2pxgd-tj976 (export/download/cite this blog post)