Twenty years ago, Fiona Godlee, at that time Editorial Director for Medicine at open access publisher BioMed Central, wrote a sharp critique of the traditional system of closed pre-publication peer review, arguing that the system needs to be opened and drawing attention to the opportunities offered by “preprint servers combined with open commentary” to realize this openness. In subsequent years, Godlee contributed to increasing openness in peer review as Editor-in-Chief of the British Medical Journal, a journal with a long-standing commitment to open peer review.

To what extent have ideas on open peer review developed by Godlee and others been realized over the past two decades? There is no straightforward answer to this question, since the availability of systematic data on peer review practices is limited. In this blog post, we use data from Crossref to offer some partial insights into the growing popularity of open peer review.

Open peer review - Journal articles

Two years ago, Dietmar Wolfram and colleagues revealed a strong growth in recent years in the number of journals supporting some form of open peer review. Over 600 journals offered open peer review in 2019. These journals published the reports of reviewers, the identities of reviewers, or both.

Figure 1 shows a similar growth in the number of peer review records in Crossref that are linked to a journal article. This number increased from fewer than 10,000 in 2018 to more than 60,000 in 2020 and 2021. Each of these peer review records represents a review report of a reviewer, a decision letter of an editor, or a response letter of an author. Many journal articles are linked to multiple peer review records in Crossref. In 2021, over 13,000 journal articles had a link to one or more peer review records.

As shown in Figure 2, Publons is by far the largest contributor of peer review records in Crossref, accounting for two-third of all records. A large majority of these records are linked to journal articles published by Wiley. Indeed, Wiley has made a considerable effort to promote open peer review (referred to as transparent peer review by Wiley). Other important contributors of peer review records in Crossref are PeerJ and eLife.

Figure 2. Peer review records in Crossref linked to a journal article: Breakdown by platform (left); Breakdown by publisher for Publons peer review records (right).

Importantly, there are several publishers that publish review reports but don’t register a Crossref DOI for these reports. This is for instance the case for BMJ, EMBO, MDPI, PLOS, and Springer Nature. There is no straightforward way to determine the number of review reports published by these publishers, although some figures can be obtained from information provided by the publishers. MDPI for instance reports that in 2020 it published review reports for over 34,000 articles. Recent statistics for PLOS show that PLOS published review reports for more than 7,500 articles in 2021. For Springer Nature recent statistics don’t seem to be available, but the journal Nature reports to have published review reports for almost 450 articles in 2021. While precise figures aren’t available, it is clear that the total number of articles in 2021 for which review reports were published without a DOI is much larger than the 13,000 articles for which review reports were published with a DOI. Hence, a large majority of the published review reports don’t have a persistent identifier and don’t have openly available metadata. This severely limits the value of these reports.

Open peer review may refer not only to publication of review reports, but also to publication of identities of reviewers. The pros and cons of disclosing reviewer identities have been discussed extensively. For peer review records in Crossref classified as review report (instead of decision letter or author response), we found that the identity of the reviewer is disclosed in 39% of the cases.

Open peer review - Preprints

A recent development is the publication of review reports for preprints instead of journal articles. A variety of platforms and initiatives offer a range of different approaches to peer review of preprints.

Figure 3 shows the growth in the number of preprint peer review records in Crossref. While the number of records is much smaller than for journal articles, the growth is impressive, from 20 records in 2019 to 733 records in 2021. Three platforms, Qeios, MIT Press Rapid Reviews COVID-19, and ScienceOpen, account for almost all preprint peer review records in Crossref. A small number of records originate from Publons, and a few records represent review reports published by one of us using the PubPub platform.

Figure 3. Peer review records in Crossref linked to a preprint: Growth in the number of records (left); Breakdown by platform (right).

Many review reports for preprints don’t have a Crossref DOI. These reports aren’t included in the statistics presented in Figure 3. This is for instance the case for review reports from PREreview, a platform for preprint peer review that uses DOIs from DataCite instead of Crossref. PREreview currently contains more than 250 review reports (excluding so-called rapid reviews). Other review reports for preprints don’t have a DOI at all. This applies both to review reports published directly on a preprint server and to review reports published on various preprint peer review platforms. The latter reports may be visible in Sciety, a platform that aggregates review reports and other evaluations of preprints from a variety of sources.

Open peer review - Copernicus and F1000

Copernicus and F1000 are special cases. Copernicus offers an integrated platform that publishes both journal articles and preprints as well as the associated review reports. Likewise, F1000 provides a platform that publishes multiple versions of an article, including the review reports for each version. Because of their special nature, we present statistics for Copernicus and F1000 separately from the statistics reported above. Peer review records for Copernicus and F1000 aren’t included in Figures 1, 2, and 3.

Figure 4 shows the growth in the number of peer review records for Copernicus and F1000. The statistics for F1000 were obtained from DataCite instead of Crossref, because F1000 uses DataCite to register DOIs for review reports. Copernicus is a major contributor of peer review records in Crossref, especially in earlier years. F1000 is a pioneer in registering DOIs for review reports, starting in 2012. The number of peer review records for F1000 is much smaller than for Copernicus, but it has shown a steady growth over the past decade.

Figure 4. Growth in the number of peer review records in Crossref for journal articles and preprints published by Copernicus (left) and in the number of peer review records in DataCite for articles published by F1000 (right).

Recommendations

The statistics presented in this blog post show an impressive growth in the adoption of open peer review, especially in recent years. Nevertheless, the transition advocated twenty years ago by Fiona Godlee, from the “flawed system of closed prepublication peer review” to a system of “preprint servers combined with open commentary”, is still in a very early stage, with preprint peer review starting to take off only very recently.

We see improvement of peer review as a joint responsibility of all stakeholders involved. Each stakeholder needs to make a contribution. We recommend that:

• Authors and reviewers explore the interesting new opportunities offered by preprint peer review platforms.

• Scientific publishers and preprint peer review platforms register DOIs for open review reports and include links in the metadata of these reports to the corresponding journal articles and preprints.

• Bibliographic databases provide data not only for journal articles, but also for preprints and for open review reports linked to journal articles and preprints.

• Research funders and institutions consider using this data in research assessments in order to give appropriate recognition to the work done by authors and reviewers.

We end this post by drawing special attention to an opportunity individual researchers have to contribute to the growth of open peer review. The Publish Your Reviews initiative, launched today by ASAPbio (and co-organized by one of us), calls on researchers to publish their review reports alongside the preprint version of an article. This offers an easy way to promote openness in peer review. Researchers are invited to express their support for the initiative by signing a pledge.

This blog post is largely based on a presentation given by the authors in the CWTS research seminar on June 10, 2022. The slides of this presentation can be found here.

Photo credits header image: Katerina Pavlyuchkova