The adoption of open science practices has become a prominent topic of study for the science studies community. However, the research practices of the community itself are still quite traditional. While open access publishing, preprinting, open peer review, open data sharing, and other open science practices are gradually becoming more common in the science studies community, the adoption of these practices is still at a relatively low level.

Given the community’s deep understanding of the research system, we think we should be able to do a better job. As organizers of this year’s Science, Technology and Innovation Indicators conference (STI 2023), we therefore introduce two open science experiments: We adopt a new publication and peer review process, fully aligned with state-of-the-art open science practices, and we invite authors of contributions submitted to the conference to reflect on their own open science practices.

Experiment 1: Opening the publication and peer review process

In earlier editions of the STI conference, contributions submitted to the conference were reviewed in a traditional closed peer review process. Contributions accepted for presentation at the conference were published in the conference proceedings while those not accepted for presentation were not published.

For the STI 2023 conference, we are going to experiment with an open ‘publish, then review’ model as an alternative to the closed ‘review, then publish’ approach. The publication and peer review process will be organized as follows:

• The conference will use a submission and publication platform provided by Orvium. All contributions submitted to the conference will immediately be published as a preprint on the platform. Contributions will be openly accessible under a CC-BY license. Authors will retain their copyright. Each contribution will have its own DOI.

• The conference will organize an open peer review process. For each contribution submitted to the conference, the reviews will be published on the Orvium platform and will be linked to the preprint version of the contribution. Reviewers will be encouraged to disclose their identity, but they may also choose to remain anonymous. Authors will be invited to update their contribution based on the feedback provided by reviewers. The updated contribution will also be published on the Orvium platform.

• As conference organizers, we will use the peer review results to select contributions for presentation at the conference. All contributions will remain available on the Orvium platform, both the contributions selected for presentation and those not selected.

Further information about the publication and peer review process of the STI 2023 conference can be found in the call for papers. The ‘publish, then review’ model that we are going to use at the conference is inspired by platforms such as F1000Research, eLife, and Peer Community In, which combine preprinting and open peer review.

Expected benefits

Compared with the publication and peer review process in earlier editions of the STI conference, we see a number of benefits in our new approach:

• Accelerating the dissemination of new scientific knowledge. The immediate publication of conference contributions as preprints will speed up the dissemination of new scientific knowledge. Interested researchers and practitioners will have access to the latest scientific findings without waiting for the conference to take place.

• Increasing the value of peer review. In the traditional closed peer review model used in earlier editions of the STI conference, reviews were made available only to the authors of a conference contribution and to the conference organizers. In our new open peer review model, reviews will also be made available to the readers of a conference contribution, helping readers to develop a more informed understanding of the strengths and weaknesses of a contribution, also the ones not selected for presentation. This will increase the value of the reviews.

• Giving more recognition to authors. The immediate publication of conference contributions as preprints will enable authors to get feedback and credit more rapidly.

• Giving more recognition to reviewers. In the traditional closed peer review model used in earlier editions of the STI conference, reviewers hardly got any credit for their work. In our new open peer review model, reviewers who choose to disclose their identity will get public recognition.

Potential concerns

We recognize that the publication and peer review process of the STI 2023 conference may also raise concerns. A common objection against preprinting is that preprints may present inaccurate results because they are published before peer review. While results presented in preprints may indeed be inaccurate, the same applies to results reported in peer-reviewed articles, since peer review usually does not resolve all inaccuracies in an article. We also note that the reviews that will be published alongside the preprinted conference contributions will help readers to assess the quality of a contribution. Another concern about preprinting is that journals might be reluctant to publish articles that have already been published as a preprint. However, very few journals still have such a policy.

A common concern about open peer review is that criticism provided by reviewers may be incorrect or even offensive. As conference organizers, we call on reviewers to give constructive and respectful feedback, for instance by following the FAST (focused, appropriate, specific, transparent) principles. We reserve the right to moderate reviews that we regard as disrespectful. If the authors of a conference contribution consider criticism provided by a reviewer to be incorrect, they will have the possibility to publish a response in which they explain why they disagree with the reviewer. Authors will also be able to update their conference contribution to address problems identified by reviewers.

We appreciate that special consideration needs to be given to the interests of PhD students and other researchers who find themselves in vulnerable positions. These researchers may be disproportionately affected by the drawbacks of the way publication and peer review processes are organized. In a closed ‘review, then publish’ model, peer review may be biased against these researchers, lowering their chances of getting their work published. In an open ‘publish, then review’ model, these researchers may feel uncomfortable both about their own work being critiqued publicly and about publicly critiquing the work of others, in particular the work of more senior colleagues. In the evaluation of our new approach to publication and peer review (see below), we will pay special attention to the experiences of PhD students and other researchers in vulnerable positions.

Experiment 2: Reflecting on open science practices

As organizers of the STI 2023 conference, we strongly encourage authors of contributions submitted to the conference to adopt open science practices in their work. At the same time, we recognize that there may be barriers to the adoption of such practices, including for instance reliance on proprietary data sources, legal or ethical concerns, and lack of experience with open science practices. Rather than introducing formal open science requirements, we therefore take a more experimental approach. We invite authors of conference contributions to explicitly reflect on their own open science practices.

Each contribution submitted to the conference is expected to include a short section in which the authors reflect on the use of open science practices in the research presented in their contribution. Authors may for instance discuss the openness of the data used in their research. If the data is openly available, the authors can explain how the data can be obtained. If the data is not openly available, the authors can explain why they do not use openly available data or why they are unable to make their data openly available. Openness of software and source codes can be discussed in a similar way. Authors may also discuss whether a research plan was made openly available at the start of their research (‘preregistration’) or whether any intermediate results of the research have already been published, for instance in a preprint.

We hope this experiment will increase the awareness and adoption of open science practices in the science studies community. In addition, the experiment may also help organizers of future conferences to better understand how the adoption of open science practices can be facilitated and promoted.

Evaluating the experiments

The above two experiments will hopefully provide an additional motivation to colleagues in the science studies community to submit their work to our conference. We are eager to see how the experiments will work out. After the conference, we will invite conference participants to fill in a survey to evaluate the experiments. The outcomes of the evaluation will be reported in a blog post. In the meantime, if you have any questions about the experiments, or any feedback you would like to share, don’t hesitate to contact us.

Looking forward to meeting you at STI 2023!