Authors

Alex Rushforth and Gunnar Sivertsen

Abstract

Recent years have seen considerable growth of reform movements aiming to arrest perceived dysfunctions across science. A prominent problem that has formed the focus of activism in academia has been research assessment practices. Multi-actor initiatives such as the Declaration on Research Assessment (DORA) and the Coalition on Advancing Research Assessment (CoARA) have served as prominent backbone organizations for this movement, coordinating collective action among diverse research stakeholders. As governance structures within science, a notable characteristic is their reliance upon ‘soft governance’ and ‘self regulation’, rather than incentives or coercion. In this paper we take stock of how effective these example initiatives have been in coordinating and mobilizing collective action on their respective problems. We introduce a framework to compare the initiatives, apply it to these two cases, and draw conclusions about the respective achievements, challenges, and limitations of these structures.

Authors

Antoine Schoen, Patricia Laurens, Gaston Heimeriks, Martina Neuländtner, Thomas Scherngell, Alfredo Yegros, and Philippe Larédo

Abstract

This paper looks at how scientific collaboration linking universities and large international firms fosters local invention. The first step aims at identifying the universities engaged in collaboration with large corporate entities belonging to the industrial sectors of pharmaceuticals, chemicals and biotechnology. After having identified the corpus of corporate publications authored by these enterprises we analyse in this corpus the universities engaged in this corporate scientific collaboration allocating the co-authoring institutions in the territories hosting the academic authors. The extent and nature of the universities’ collaboration with large firms is then analysed using the SAR model in regard to the intensity of local inventive intensity in the metropolitan area hosting the universities. The results show that corporate publications have a clear and significant effect on inventive outcome and production.

Authors

Eleonora Dagiene, Vincent Larivière, and Ludo Waltman

Abstract

Lithuania’s research assessment prioritises Web of Science publications, aiming for international impact. This paper examines the policy, introduced in 2012, of excluding articles from journals with low impact factor or suspected citation inflation. We analyse how this suspension policy affected domestic journals and researchers’ behaviour. The results show that Lithuanian researchers’ publications in foreign outlets increased, but concerns remain about the chosen outlets’ quality and potential continuing manipulation of Web of Science metrics. These are initial findings of a broader study exploring policy dynamics and unexpected consequences of quantitative research assessment.

Authors

Huilin Ge, Clara Calero Medina, and Rodrigo Costas

Abstract

Academic mobility is a phenomenon in today’s academic landscape that facilitates global collaboration and knowledge transfer. Despite the benefits of academic mobility, concerns about brain drain persist, and fears of inequalities in mobility dynamics start to emerge. Our research explores the relationship between mobile researchers and the collaboration ties with their country of origin, revealing a remarkable trend: mobile academics maintain varying degrees of collaboration with their origin countries after moving. Scholars migrating from higher-income countries to lower-income ones exhibit a strong inclination to maintain ties with their highincome origins, but the reverse is not observed. These findings suggest unequal benefits of mobility regarding the collaboration ties and highlight the necessity for implementing policies and initiatives geared towards nurturing more beneficial international research collaboration and harnessing the contributions of mobile scientists.

Authors

Jonathan Dudek, Jeroen van Honk, Isabel Basson, Carole de Bordes, Ismael Rafols, and Rodrigo Costas

Abstract

Most African countries have low domestic research funding and receive a significant portion of their funding from foreign sources. Consequently, it is of special interest to understand funding flows and their influence on African research. The Dimensions (Digital Science) database brings together information from publications and projects funded by more than 500 funding agencies from more than 40 countries. By combining grant (project) data and funding acknowledgments in publications, we analyse funding flows to African countries: who are the main funders? which countries and organizations do they support? what issues do they prioritize? The study will also explore the possibilities and limitations of using Dimensions funding-related data for the analysis of funding flows between global regions.

Authors

Judit Varga, Wolfgang Kaltenbrunner, and Helen Buckley Woods

Abstract

This short paper presents preliminary findings from a research project investigating the uptake and use of so-called narrative CVs in review panels for research funding decisions. Our analysis draws on empirical material collected through interviews and participant observation in two subsequent panel meetings of a funding program for early career researchers of the Dutch national research council NWO. As expected, we find that the recent introduction of a narrative CV in this program has not instantaneously transformed evaluative practices. Rather, the emphasis on substantive narrative accounts of applicants’ achievements, combined with the backgrounding of quantifiable evidence such as publication counts and metrics, leads reviewers to gradually problematize and reconsider previously dominant definitions of quality. Our analysis showcases exemplary situations in which such reflection becomes apparent, and it highlights different ways in which reviewers try to resolve newly underdetermined evaluative situations.

Authors

Leyan Wu, Akrati Saxena, and Vincent Traag

Abstract

Recent decades witnessed a steady increase in international scientific collaboration, alongside debates on nationalist tendencies in scientific practices and global disparities in citation flows. Geographical proximity has been shown to influence collaboration and citation rates, yet how this affects global patterns remains unclear. Addressing this gap, we examine the interplay of geography and country-level effects in collaboration and citations using OpenAlex data. Analysing 1.6 million publications with 6.8 million collaborations, and 3.7 million citations, we unveil distinct distance dependencies and country-level effects. Collaborations between countries decline slower with distance compared to intra-country collaborations. Surprisingly, citations between countries display lower distance dependence above a certain threshold, while intra-country citations decrease notably. These findings suggest that geographical factors alone do not determine inter-country collaborations and citations, hinting at underlying country-level effects that we will explore further in follow-up research. This study advances our understanding of the complex dynamics shaping international scientific engagement.

Authors

Marc Luwel and Nees Jan van Eck

Abstract

The explosion of cases of peer review fraud and activities of paper mills pose a systemic threat to the integrity of the scientific publishing process. By linking the Retraction Watch and Web of Science databases, metadata of retracted publications are analysed. In the period 2007-2021, the use of non-institutional email addresses by these publications’ reprint authors increased by more than 20%, mainly due to an even larger increase from authors with a postal address in India and especially China. Among the journals with the most retracted publications, the use of non-institutional email addresses by reprint authors differs greatly: from 7 to 99%. It is also surprising that only 36% of the mainly Chinese reprint authors of the mass retracted publications in Hindawi journals used a non-institutional email address. These results indicate that caution is recommended and not to automatically red-flag publications as fraudulent when non-institutional email addresses are used.

Authors

Nelius Boshoff, Similo Ngwenya, Amani J. Uisso, Susanne Koch, Rodrigo Costas, and Jonathan Dudek

Abstract

Inspired by a search for epistemic justice, the study examines the representation of Tanzanian forest science in five databases: Dimensions, OpenAlex, Scopus, Web of Science and a manually compiled set of articles in Tanzanian journals. It also ascertains whether profiles of three epistemic elements (alignment of research to national research priorities, national region studied, and forest type studied) differ between the databases. Initial findings of the ongoing research indicate that OpenAlex has the highest coverage of Tanzanian forest science articles and that it incorporates all the forest science articles of the other databases (excluding those in the Tanzanian dataset). In fact, 87% of articles in Tanzanian journals do not appear in the other databases’ forest science collections. This is a form of epistemic injustice as the invisibility of large parts of forest science produced by Tanzanian authors collectively marginalises them and ultimately prevents them from fully participating in scientific meaning-making.

Authors

Patricia Alonso-Álvarez and Nees Jan van Eck

Abstract

Unlike traditional proprietary data sources like Scopus and the Web of Science (WoS), OpenAlex emphasises its comprehensiveness, claiming an extra coverage of humanities, non-English languages, and the Global South. Strengthening diversity and inclusivity in science is crucial for ethical and practical reasons. This paper analyses OpenAlex’s coverage and metadata availability of African-based publications. For this purpose, OpenAlex is compared with Scopus, WoS, and African Journals Online (AJOL). We first compare the coverage of African publications in OpenAlex against that of WoS, Scopus, and AJOL. We then assess and compare the available metadata for OpenAlex, Scopus, and WoS publications. The analysis shows that OpenAlex offers the most extensive publication coverage. Regarding metadata, OpenAlex offers a high coverage of publication and author information. It performs worse regarding affiliations, references, and funder information. Importantly, the results also show that metadata availability in OpenAlex is better for publications also indexed in Scopus and WoS.

Authors

Thed van Leeuwen, Lieuwe Kool, and Ingrid Wijk

Abstract

In this study we aim at some unintended consequences of science policy regarding open access publishing by Dutch academics. Various actors in the system have advocated different forms of open access publishing, and our study will unravel how publishing occurs in various forms of open access publishing (Gold versus Hybrid), and for two types of scholarly publishing, the general output of the Netherlands and the output for which authors located in the Netherlands are corresponding authors. A special focus will be on any underlying field differences across the Dutch scholarly landscape. The study shows that contrary to national and international efforts, the main focus of Dutch academics is on Hybrid open access publishing.

Authors

Tommaso Ciarli, Hugo Confraria, Ed Noyons, and Ismael Rafols

Abstract

This paper examines the characteristics and impact of research related to the Sustainable Development Goals (SDGs), aiming to understand specific features of SDG-related research compared to non- SDG related research. The findings reveal that SDG-research research, especially on SDGs related to people and society (such as poverty alleviation, education, gender equality, and justice) are more likely to be cited in policy documents, indicating their relevance for public use, and to adopt features of open science, such as open access and interdisciplinary. However, these publications are less likely to mention funding and are less likely to be cited in academic publications compared to non-SDG-related research, suggesting a disconnect between potentially more impactful research and academic recognition. Results reveal gaps in research funding priorities while emphasising the importance of considering the societal relevance of research alongside traditional excellence criteria.

Authors

Thomas Klebel and Vincent Traag

Abstract

Sound causal inference is crucial for advancing the study of science, but many publications in science studies lack appropriate methods to substantiate their causal claims. Using the example of Open Science, our poster provides an overview of structural causal models and how they allow researchers to make their causal assumptions transparent, providing a foundation for causal inference in quantitative science studies.

Authors

Qianqian Xie 

Abstract

Our work assesses the effectiveness of CSC program through the research performance of CSC recipients and compares it to that of their home and host countries’ counterparts. We contextualized CSC awardees’ performance benchmarking against their home and host countries’ counterparts, by using normalized mean citation score (MNCS) and the proportion of highly cited papers (PP(top 10%)) and the proportion of open access papers (PP_OA). Our work reveals CSC-funded papers overall consistently demonstrate superior scientific research and open access performance compared to Chinese researchers of the same period. Additionally, within Physical Sciences and Engineering and Mathematics & Computer Science, CSC awardees exceled than their hosting countries with high MNCS and PP (top 10%), but in some countries CSC awardees (e.g. the US, Canada, Australia et al) have lower open access rates. Conversely, in Biomedical and Health Sciences, CSC awardees show inferior performance but a higher open access percentage compared to their counterparts.

Authors

Wei Quan, Huilin Ge, and Rodrigo Costas

Abstract

This study presents the overview of Chinese scholars’ international mobility based on bibliometrics data from Dimension. We identify 2,446,785 individual researchers affiliated with China from the 2000-2022 period. Based on those data, we find that 2,381,665 researchers are not moving in their academic careers, and 65,120 researchers have international mobile footprints. Furthermore, after distinguishing China as an origin country and destination country, we find that China has been losing researchers since 2000, with the United States being the main destination for those who move internationally. What’s more, we find that when scholars move, they generally do not change their research fields on a large scale, but some do change their fields, particularly in the social sciences. Our study illustrates the landscape of Chinese researchers’ international mobility and highlights the flow patterns.

Authors

Wenceslao Arroyo-Machado, Rodrigo Costas, and Adrián A. Díaz-Faes

Abstract

This research-in-progress paper examines the alignment between public interest, as evidenced by Wikipedia page views, and the distribution of academic resources across various health conditions. Utilising data from Wikipedia, Wikidata, and OpenAlex, the study reveals both relevant geographical correlations and notable gaps in how diseases are addressed in academic research compared to their visibility in social media. Moreover, discrepancies in content quality on Wikipedia pages indicate potential biases in the global research agenda. These findings underscore the importance of considering both social and academic metrics to address misalignments and advocate for a more equitable distribution of research resources in the biomedical sciences.

Authors

Wolfgang Kaltenbrunner, Felicitas Hesselmann, Serge Horbach

Abstract

While a lot of STI research focuses on innovations in scholarly publishing as a research object, our own scholarly communication practices often rely on more traditional (closed) journal - based pre - publication peer review. In this special session, we aim to discuss to what extent this is an intentional choice, and in what ways we might benefit from the various affordances of open peer review - in particular the prospect of reusing open review reports across various outlets or contexts. The discussion will be prepared by four short pitches. First, some of the organiz ers of the preceding STI conference will report about experiences with open peer review in the 2023 STI iteration in Leiden (Ludo Waltman). A second pitch will collect the existing evidence on implications of open peer review for diversity and participatio n (Serge Horbach). A third pitch will reflect what open and portable review reports mean for the workings of the peer review economy in a publishing context (Wolfgang Kaltenbrunner). Finally, there will be a pitch on the possibilities of automation in peer review and its implications for openness (Felicitas Hesselmann).

The subsequent discussion will focus on the following questions:

• What are STI attendants’ motivations to (not) engage in open review formats?
• What are the most important opportunities and challenges related to implementing open and portable peer reviews?
• How do open and portable reviews alter the development of research arguments and the engagement with audiences?
• How compatible are various publication cultures with open reviews?

Authors

Alex Rushforth, Janne Pölönen (org), Neil Jacobs, Louise Bezuidenhout, Maria Pietila, Clifford Tatum, Francesca Di Donato

Abstract

Over the past five years, the reform movements for open science and research assessment have expanded significantly. Increasingly, their agendas overlap , with champions of open science seeing current assessment practices as incentivizing behaviours that hinder wider accommodation of open science activities among researchers (UNESCO, 2021, Nosek, 2019, Spector, 2022) . Reform of academic career assessment systems is often earmarked as a site through which to incentivize academics to more fully embrace open ways of working (e.g. EC, 2017, Mustajoki et al., 2021, Pietilä et al., 2023, Méndez and Sánchez - Núñez, 2023) . It is of course easy to say that researchers in general should be ‘recognized’ and ‘rewarded’ for open science (in its various incarnations) . But what might this look like in practice, what has actually been tried out so far, how might researchers in their roles as evaluators and producers of knowledge respond to such steering efforts, and what might be the consequences for career evaluations and research practices in different research fields and academic institutional settings?

This special session brings together a range of science studies researchers and practitioners working on the emerging intersections between open science and academic career research assessment reforms , to present, discuss, and debate current and future practices . Each speaker will be invited to reflect upon the promises, achievements, and challenges faced in turning this broad promise into workable practices.

Authors

Stephan Gauch, Ludo Waltman, Stephan Stahlschmidt, Rodrigo Costas, Martijn Visser, Nees Jan van Eck, Martin Reinhart, Laura Rothfritz, Heinz Pampel, Jacqueline Sachse, Clemens Blümel, Max Leckert, Marcus John, Bianca Kramer, Cameron Neylon

Abstract

There is a growing interest in the scientometric community in open research information (ORI), that is, scientometric data sources that are open. Data in these sources can be freely accessed and shared. Examples of open data sources include global multidisciplinary infrastructures (e.g. Crossref, OpenAlex, OpenAIRE, OpenCitations, an ORCID), discipline - specific infrastructures (e.g., PubMed), and regionally focused infrastructu res (e.g., SciELO and Redalyc).

The aim of this special session is to provide an overview of the current state and potential futures of the landscape of open research information. This will include the quality (e.g., accuracy and completeness) of the data available from open scientometric data sources, the use and adoption of open scientometric data sources, and developments regarding governance, cooperation, and sustainability of open scientometric data source. To get started, there will be four five - minut e talks aimed at offering a diversity of perspectives on the state of the open research information landscape. These talks will be given by the organizers of the special session, highlighting the various perspectives on openness of research information in the organizing team. One of the talks will cover the Barcelona Declaration on Open Research Information. The special session then explores potential futures of ORI through participatory foresight methods engaging present stakeholders, such as bibliometrici ans, policymakers, research evaluators, and the academic community.

After the conference, the scenarios and results will be consolidated, discussed with experts, and disseminated through explainer videos to engage a wider audience. The findings will inform the development of a research agenda for Open Research Information.

Authors

Stefanie Haustein, Jeremy Y. Ng, Dimity Stephen, Alexander Schniedermann, Stephan Stahlschmidt, David Moher, Ludo Waltman, Wolfgang Glänzel

Abstract

Over the past decade, there has been a significant increase in the number of bibliometric analyses published in the peer - reviewed literature, particularly outside of library and information science where bibliometric s is traditionally situated. While diversity has always been a strength, it also poses challenges in ensuring quality. The variability in reporting quality we currently see may, at least partially, be grounded in the lack of guidance on how to conduct and report bibliometric studies by experts from within the field. This special session will engage the community in a discussion about quality and reporting of bibliometric analyses. Our session relates to the conference theme of “Into the great wide open?” in that the level of openness in the bibliometrics community – our methods and analytical tools, the application of our work, our disciplinary backgrounds, and increasingly more often our data sources – have given rise to a challenge in ensuring the quality of our work and the integrity of the field. The session will combine an introductory panel and an open fishbowl conversation. Five chairs will be arranged in the center of the room, four of which will be occupied initially by our panellists . A fifth chair will be left empty for audience members to participate in the conversation , forcing a panellist to leave the fishbowl and invite a dynamic discussion. This session will be an opportunity for participants to provide their input and discuss current issues in the context of open science with a focus on transparency and reproducibility, rigor, robustness and reliability of bibliometric analyses.

Authors

Alex Rushforth, Nosisa Dube, Steven Hill, Moumita Koley, Cameron Neylon, James Wilsdon

Abstract

The trans-national reform movement for responsible research assessment is growing considerably. As part of this, there is increasing interest in monitoring and evaluating research cultures and environments (RCE) within national, program and institutional assessment processes. Moving from broad aspirations to workable, transparent and legitimate evaluation practices is not however straightforward. Meeting these demands will necessitate relevant, trusted and reliable indicators, that can provide the robustness of evidence needed for monitoring and evaluation. Rapidly unfolding developments around RCE indicators thus merit urgent attention from experts within the STI community. The aim of this special session is to bring together researchers and practitioners to debate urgent questions over emerging RCE indicators, draw out important lessons, and compare experiences a cross research systems.

The format will consist of short presentations from invited experts from the Research on Research Institute’s AgoRRA RCE workstream and audience Q&A. Speakers will share insider accounts of the latest policy developments, debates and controversies in their respective national research systems, and to draw out technical and social issues RCE developments are throwing up. These include methodological questions over indicator design and data needs, potential disagreements over what to evaluate, resourcing, and opposition by some research system stakeholders.

Authors

Gunnar Sivertsen, Ludo Waltman, Lin Zhang, André Brasil

Abstract

Background
In its proposal for a new strategy, "Towards responsible publishing" , cOAlition S summarizes: “In the five years that have elapsed since the publication of the Plan S principles, the move toward full and immediate Open Access (OA) has become global and irreversible.” However, “this has been delivered through business models – such as Read and Publish agreements and APCs – which are highly inequitable”. The coalition now envisages a “scholar-led” and “community-based” scholarly communication system “without author-facing charges”. Furthermore, the general OA ambition of Plan S is “extended to include all scholarly outputs, such as preprints and peer review reports” because “the current practice of pre-publication peer review needlessly delays the sharing of research outputs”.

Session Goals
The aim of the session is to discuss the new focuses and priorities of Coalition S based on experiences so far – including evidence provided by the areas of research contributing to the STI conferences. Three topics will be prepared for short introductions to start discussions with the audience:

1. Empirical examination of the OA transformation so far: The discussion will focus on the empirical evidence pertaining to the transition into Open Access publishing in recent years. A core question will be how the transition aligns with the traditional research policy criteria of research quality and affordability.
2. Challenges with changing the market of scientific publishing: The discussion will focus on new initiatives for scholarly publishing, their foundation and implementation issues, in particular the vision of a “scholar-led” and “community-based” scholarly communication system “without author-facing charges”. To what degree are the trends in the present market of scientific publishing aligned with these principles, and how can they be fully aligned?
3. Exploration of change trajectories for editorial practices: What are the experiences so far with alternatives to pre-publication peer review, and how do they align with the plurality of existing community-based efforts, including disciplinary differences?

Authors

Sergio Salles-Filho, Evandro Coggo Cristofoletti, Rodrigo Costas, Ed Noyons, Biegzat Murat, and Cassidy Sugimoto

Abstract

In today's rapidly evolving social landscape, which encompasses critical issues such as climate change, poverty, and technological advancements, the intersection of policy and science holds paramount significance. Understanding how research informs and shapes policy decisions has become increasingly crucial, sparking discussions on methodologies for assessing impact. While tools like Overton and Altmetric.com have gained popularity, they do not provide neither enough coverture, nor open data, presenting two significant challenges: establishing open - access sources for policy documents and bridging the coverage gaps between the Global South and the Global North. The special session aims to address two specific interests: (i) exploring the differences and synergies between open and proprietary data sources of policy documents, and (ii) examining the disparities in coverage between the Global South and Global North regarding policy documents and related scholarly outputs within these data sources. The proposal aims to foster debate on potential methodologies for leveraging open sources of policy documents and for identifying and analyzing inequalities in accessing research impact on policy. Furthermore, the session seeks to explore strategies for overcoming these inequalities and enhancing the significance of open data sources in the process.

The session's dynamics will involve brief speeches from invited guests, approximately 10 minutes each, forming a debate panel. Audience participation will be encouraged through commenting or making questions on the topic.