Peer review: Time well spent?

Peer review is a time-consuming activity. In 2020 I received 73 invitations to review a new article submitted to a scientific journal. On average, reviewing an article, both the original version and possibly also subsequent revised versions, probably takes me about seven hours in total. I am in the fortunate situation of being able to invest a significant amount of time in peer review. Of the 73 invitations that I received in 2020, I accepted 24, resulting in an estimated time investment of 24 × 7 = 168 hours in peer review of journal articles.

Is the time invested in peer review well spent? To some extent I believe it is. As a reviewer I think I usually manage to help authors improve their work. In most cases, I am probably also able to provide useful advice to the editors of a journal helping them decide whether to publish an article or not.

Nevertheless, despite the value my reviews may have for authors and editors, I often do not feel satisfied with the way my reviews are used. A review is typically read by just a few people: one or more journal editors, the authors of the article under review, and perhaps also other reviewers of the article. Importantly, readers of an article typically do not have access to the reviews of the article, even though the reviews may give them a lot of valuable information. Reviews are likely to offer readers an insightful perspective on the strengths and weaknesses of an article, and on issues on which authors and reviewers may not agree.

By making reviews available only to the small group of individuals directly involved in the peer review process of an article, readers of the article are denied the opportunity to benefit from the information provided by the reviews. Given the significant efforts made by many reviewers to provide detailed comments on the articles they review, this is a major waste of scientific labor. For me personally this is why I tend to feel dissatisfied with the way my reviews are used, making me question whether the time I invest in peer review is really well spent.

Making peer review more useful

An important step toward making peer review more useful is to publish reviews alongside accepted articles, either with or without revealing the identity of the reviewers. This form of transparent or open peer review was pioneered by publishers like BMJ, BMC, EMBO, and eLife. It is gradually being adopted more widely, for instance by some Wiley and Springer Nature journals, and also by Quantitative Science Studies, of which I am Editor-in-Chief. A recent study shows that the number of journals offering some form of transparent or open peer review has increased rapidly over the last few years (see Figure 1).

Ongoing developments in scholarly publishing offer additional opportunities to make better use of the efforts of reviewers. Nowadays many articles are already available online on a preprint server (or in an institutional repository) long before they appear in a journal. For these articles, peer review can be made more useful by publishing reviews as soon as they are available instead of postponing this until the article appears in a journal (which may take months or even years, or which may not happen at all). Immediate publication of reviews provides valuable information to readers of the preprint version of an article. Publishers could facilitate immediate publication of reviews, as is done by F1000, but reviewers can also take care of this themselves by posting their reviews online.

Reconsidering my own approach to peer review

To make my own peer review activity more useful, I have reconsidered my way of working as a reviewer, partly also by taking inspiration from others (see for instance the approach to peer review taken by James Fraser at UC San Francisco). From now on, when choosing which review invitations to accept and which ones to decline, I will give priority to journals that offer transparent peer review. In addition, when I finish a review of an article that is available as a preprint, I will immediately publish my review online, so that readers of the preprint can benefit from it. I will also prioritize reviewing such articles over reviewing articles that are not available as a preprint.

In the last two months, I published my first two reviews online. They are available here and here. I used the PubPub platform for this, but various other platforms can be used as well. Importantly, PubPub facilitates registering a DOI for a review and making the metadata of the review available through Crossref. This metadata includes a link between the review and the preprint version of the article under review. This link enables preprint servers and discovery tools to inform their users about the availability of a review for a preprint.

One of the articles that I reviewed had a preprint on Research Square. For this preprint I posted a brief comment on the preprint platform to draw attention to my review. The other article had a preprint on SSRN, which does not provide the possibility to post comments. Unfortunately, this means that many readers of the preprint probably will not be aware of my review.

Toward a broader initiative

Some of the above ideas to make peer review more useful were discussed in a session that I co-organized with Cooper Smout (founder of Free Our Knowledge) and James Fraser in ASAPbio’s recent #FeedbackASAP workshop. As a result of this session, we are preparing a broader initiative to campaign for publishing reviews. If you want to provide input or show your interest, do not hesitate to post a comment on GitHub or to contact me directly.

More information about the peer review project of the Research on Research Institute (RoRI) can be found here.