Whether in publications, seminar Q&As, or on Twitter, scientists tend to disagree. Although many topics have broad consensus—human-caused climate change and the link between smoking and cancer being but two—even the most settled knowledge was at once the subject of debate. The history of science is littered with such examples. The Galileo Affair, Einstein and Bohr’s debates over quantum theory, and the uproar over Alfred Wegner’s theory of tectonic drift are but few examples of disagreements with great consequence. Even now, debates continue over topics ranging from the value of the hubble constant to the evolutionary origins and role of dance.

When it comes to disagreement in science, one thing is obvious: it’s everywhere. Yet, the true extent of disagreement is still not well known. Countless theories posit the central role of disagreement to epistemic progress, yet empirical evidence is often limited to case studies, ethnographies, and historical examples, which while immensely valuable, can’t be generalized across all of science. For decades though, this was the only kind of data available. Fortunately, the increasing availability of scientific full-text has allowed newfound progress, making it possible to study disagreement quantitatively across all of science.

A new approach to study disagreement

In our study published in eLife, we develop a novel approach to identify disagreements from citation sentences. We define disagreement in two ways. The first is paper-level, the most clear-cut case of direct and explicit disagreement from one paper to another over any aspect of its findings, interpretations, or design. The second is community-level, such that a paper signals controversy and debate within the broader field. These two types of disagreement are exemplified in the following cases:

• Paper-level: “We find that coffee does not cause cancer, contrary to the finding of <ref> that coffee does cause cancer.”
• Community-level: “There remains controversy in the scientific literature over whether or not coffee is associated with an increased risk of cancer <refs>.”

By considering both of these types, we are able to capture the full extent of scientific disagreement. Working from our definition, we develop a novel, cue-word based strategy for identifying citation sentences as cases of disagreement. For example, words like “challenged” and “conflict” can signal disagreement, especially when accompanied by words relating to objects of disagreement (e.g., results of methods). We devise 65 queries, and through extensive manual validation, select only the best to form a single disagreement indicator. This final indicator is precise, easy-to-scale (implemented in SQL), and wholly transparent.

A complex landscape of disagreement

Using our indicator, we search for disagreements across the citation sentences of nearly 4 million articles indexed in Elsevier’s ScienceDirect database. What this immediately reveals is that disagreement is highest in the social sciences, and lowest in fields like physics and computer science (Figure 1). This almost exactly mirrors Auguste Comte’s “Hierarchy of Sciences”, a theoretical model for organizing disciplines. Fields at the bottom of the hierarchy, like physics, can postulate simple and general theories, a task that becomes increasingly difficult moving up the hierarchy towards the social sciences. Yet as we dig deeper at the level of sub-fields, a more complex picture emerges.

Examining 817 sub-fields (Figure 2), we see how the “hierarchy of sciences” begins to break down. Within the Biomedical Sciences, disagreement is highest in fields studying the social aspects of healthcare. In the Life Sciences, Paleontologists end up disagreeing more akin to sociologists than to their natural science counterparts, reflecting the inherent challenges of geological records. An interactive tool published alongside our study allows anyone the opportunity to dig into our data and uncover the heterogeneity of disagreement.

So while it’s true that disagreement is everywhere, we demonstrate that it is not uniformly distributed. Simplistic models, such as the hierarchy of sciences, cannot explain the complex landscape of disagreement observed in our study. Instead, the roots of disagreement are multi-faceted. Individual scientists disagree over differences in approach, interpretation, and background. Their likelihood to disagree is, in turn, the result of the kinds of evidence, methodology, and culture that varies across disciplines. Accounting for these many factors is essential for a better understanding of disagreement.

Setting a path forward

Our study offers a glimpse into where disagreement happens in science, but it also sets a stage for addressing questions such as what causes disagreements? and why do they matter? We make some progress on each, finding evidence of social factors at play (e.g., disagreement focused on recent, rather than past papers) and the implications towards impact (e.g., papers that disagree receive more citations). These results are preliminary, but they provide the foundation for addressing more and more consequential questions about scientific disagreement.

One of the key questions is the normative one: is disagreement good for science? Existing theories argue that disagreement forces researchers to collect new evidence and reckon with differences in worldview, pushing knowledge forward. But disagreements can also erupt over simple miscommunications, causing progress to stall.

Whether or not disagreement is necessary for advancement, it is clear that it’s ubiquitous, touching nearly every aspect of scientific activity. As scientometricians, it is necessary that we better understand disagreement, not only to build stronger theories of scientific progress, but also to inform policies and practices that take into account the role of disagreement in science.

Ironically, a science of scientific disagreement requires consensus—clear definitions and methodologies are needed to make progress. Our study sets up a foundation, one that can be collectively built upon to address this challenge. The method we introduce is robust, transparent, and open to anyone to improve; to support further progress, our code and data have been published alongside our study. Of course, disagreements over our approach are welcome—they may even lead to stronger and more impactful science.