The world of research evaluation is changing. In particular, we observe a growing need in research organizations for interactive, formative and tailor-made evaluation services. In response to this need, a team of CWTS colleagues has developed a new approach in collaboration with Ad Prins that we call the Evaluative Inquiry. Over the past few years this approach has been put to practice in several projects mainly assisting research groups and institutes with putting together the self-evaluation document of the Standard Evaluation Protocol (SEP). On the basis of these experiences, and most notably the projects with the Protestant Theological University and with the University for Humanistic Studies, we want to describe the four underpinnings of this Evaluative Inquiry in a series of blog posts: open-ended concept of research value; contextualization; mixed-method approach; and a focus on both accounting and learning. This first post focuses on our concept of value.

Both protestants and humanists approached CWTS to ask for help with their self-evaluation. These small universities with diverse workloads and publication cultures combine characteristics of the broad and diverse academic Social Science and Humanities (SSH) domain and have distinct philosophical and spiritual missions. It was a challenge to provide strong proof of their societal relevance and scientific excellence which was demanded by the formal SEP protocol. Their questions about benchmarking, visibility through citation patterns, and “lists” of productivity revealed a worry that their books and carefully crafted single-authored papers weren’t going to demonstrate the vital characteristics of their work.

In recent years societal relevance was added to the evaluative palette of the SEP protocol, adding performances of academics in the public debate or collaborations with societal stakeholders to the understanding of impact. However, do sermons that staff members of the Protestant Theological University (PThU) occasionally preach count towards impact? And what can be said of the multiple advisory roles in institutions, with research informants, in political parties or the cultural sector that theologians and humanists have, that are often informal and not traceable to reports, media performances or otherwise? These relations are said to be so local and small-scale that they are almost invisible, so what is their impact, and how to count it?

The problem of the value of academic research, then, for our SSH clients lies in the fact that they understand it in the realm of performance, of impact that can be shown in metrics. However, this reduces the concept of value to measurable achievements and does not do justice to the multiple possible contributions that researchers make both to science and society. Moreover, it understands value as the vehicle for accountability, showing one’s worth according to an external standard of excellence. The Evaluative Inquiry (EI) is, instead, interested in value as the connection between the institution’s mission or research themes and their reception and use by others in the world around the institution. As the EI’s first principle, this focus on “value trajectories” opens up the concept of academic value and moves it away from a focus on citation scores to finding dense and vital activity around research themes and ambitions. This implies that EI does not necessarily adopt the distinction between scientific and societal value of knowledge that has become commonplace in research evaluation. The Evaluative Inquiry draws both on metrics and on qualitative analyses as ways or making these trajectories visible.

In the first phase of the Evaluative Inquiry with the theologians and humanists we invited them to think about their value as something that goes beyond what can be expressed in performance metrics and the inherent anxiety about how they compare to others that goes along with it. In these exploratory conversations the EI team has spent quite a bit of time explaining this and the benefits of attuning research ambitions, research strengths, and research organizations. For the theologians this brought up the effects they felt of the declining appeal of theology in society and the downsizing of a thriving theological academy to a handful of institutions scrambling for students and research funding. Over time one of the ambitions that ensued from our conversations and analyses was to showcase the multidisciplinarity and spiritual orientations of the PThU while balancing the diverse demands of a small organization on its employees with needs and professional ambitions of the latter.

To conclude, what is of value and to be valued is not fixed. When clients choose to work with this method, the Evaluative Inquiry takes the time for probing and exploring to find out where their value is and what it looks like. Making visible organizational specifics and ambitions are not meant to better situate performance and productivity metrics, but to expand what can be considered as good academic value. Good academic value, then, goes beyond writing articles and counting citations to also include maintaining organizations and building conversations. This is not to say that we deny the importance and use of performance metrics and accountability; it is a way to expand the stories we consider to be achievements and want to account for.

Read part II of the series here.