The learnings have been presented and published in different shapes and forms, book chapters, working papers, reports, many deliverables, publications, and presentations. Here we present the breadth and depth of the work by linking it to the SUPER MoRRI blogpost series and turning them into an overarching story in two episodes.

In 2014, CWTS got involved in the MoRRI project funded by the European Commission (EC) to follow up on their new policy of Responsible Research and Innovation (2013). In the policy context, the concept of Responsibility is attributed to anticipatory, reflexive, responsive, and inclusive attitudes in research and innovation processes and was operationalized in six key principles: gender equality, public engagement, ethics, open access, science literacy and science education, and governance. MoRRI followed the key principles of the EC, building upon earlier indicators, and developing new ones, on a national level. Data collection across Europe on the actual situation was hampered by the low number of responses, and it was felt that such an approach didn’t do justice to the diversity in policy, geographical and organisational aspects. This was all taken up by the MoRRI successor, SUPER MoRRI.

SUPER MoRRI is one of the EC flagship projects that brings together the EC policies on Responsible Research and Innovation, funded through the H2020 Science with and for Society (SwafS) program. It does so in different ways: continuing the MoRRI train through a significant data collection exercise, doing broad conceptual work through in-depth case studies, and connecting with European and global partners. But first, the consortium felt that they should take the time and go back to the drawing table to reflect on several aspects.

Reflections on what monitoring and evaluation of RRI is

In the first blogpost of the SUPER MoRRI series, Wouter van der Klippe discusses the transition from MoRRI to the new SUPER MoRRI project. When we stop focusing on the outcomes of measurements and their accuracy and instead consider the possibilities that monitoring provides, evaluation is set free from producing indicators. Instead, evaluation can be an opportunity to create a space for articulating what is valued and why. Thus, SUPER MoRRI is creating a transformative opportunity that may better equip us to develop more open potential RRI-enriched futures, as diverse and inclusive as were initially hoped for.

But when we talk about monitoring RRI, is there a consensus on the purpose or even the content of RRI? Roger Strand distinguishes between three policy narratives for RRI in his blogpost. They are portrayed in a slightly caricatured way and exaggerate the differences. The first policy narrative is: “How do we regain control over the runaway train of science and technology before it totally destroys our world?” Some hope that RRI will enable society to speak back to science and help shape research agendas and ultimately research trajectories so that they will lead to outputs and outcomes that are beneficial to people and the planet. The second, opposite narrative is: “How do we educate, reassure and calm down the ungrateful public and make them trust us, trust science again?” RRI, and especially the so-called RRI “keys” of the EC are better fit for the purpose of providing accountability and legitimacy to calm down the citizens and argue that everything is in order. The third policy narrative sees the world of science, technology, and society as a set of entangled networks that are increasingly in need of mutual collaboration and communication. Here RRI is seen as a way to improve collaboration across silos and promote thinking outside of the box. Therefore, it is important to be aware of such plurality of desires for RRI when working for SUPER MORRI.

In the third reflection, Paula Otero Hermida describes the difference between monitoring science and monitoring innovation. In science, there is a wide set of indicators and periodic monitoring efforts on e.g. gender aspects. As the main reference, there are the She Figures reports (European Commission) as well as indicators in the main RRI monitoring initiatives at the European level. However, when looking at the monitoring of innovation, it seems that the current paradigm ignores who (and in which conditions) innovates, but rather focuses on the firm’s environment and systemic variables. It is hence important to realise which data are not available: there are no data on the human factor in the innovation surveys. That means we can find data regarding innovation’s investment and firms, but there are limited data available on the sociodemographic traits, labor conditions, or other factors regarding the actors that are innovating themselves. Therefore, there are no data on gender in the case of the OECD, EUROSTAT nor in the innovation survey carried out by the Spanish National Institute of Statistics. When data is not collected or available, it is important to be aware of gender and other diversity aspects, and include it in the monitoring framework to make it explicit.

SUPER MoRRI’s principles

When we do monitoring and evaluation of RRI, we would like ourselves to be responsible too! That is reflected in the strategic plan that emerged from the internal conversations. Richard Woolley describes our principles in the SUPER MoRRI strategic plan. The consortium’s conceptual view on RRI takes engagement among diverse constellations of actors as the key driver of enhanced responsibility in both research and innovation and would increase consideration of relevant environmental and societal uncertainties. The monitoring framework will provide interested stakeholders with resources that can help them to plan and progress toward more responsible practices and strategies. This will be done on the basis of a ‘responsible quantification’ approach in which data and information are presented, and made interpretable, in appropriate ways. A related conceptual innovation is what we call ‘credible contextualisation’. This is the idea that any indicators we develop should, first, pass through a co-creation phase with potential users, and second, be accompanied by guidance on the degree of interpretive ‘stickiness’ of the indicator.

Three years into the project, the two principles still stand and are applied as further elaborated by Richard Woolley in a blogpost. Transforming these principles into a framework presenting SUPER MoRRI indicators in three components: an indicator fiche; a description of potential interpretive models; and complementary information to support user understanding and interpretation of the indicator. In the case of new RRI indicators, the interpretive model will explain the rationale for the creation of the indicator and how it is perceived to support RRI. For indicators that are time-series – or have the potential for future replication to create time-series – the model describes what a change in the indicator can be reasonably understood to mean. Further information is provided to try and ensure the credible contextualisation of the indicator. Involving users in the development phase of the indicator also helps to guide the design and production of these supporting elements as work progresses.

Putting the monitoring framework into practice

How to operationalize then the data collection in a responsible way? Moving away from surveys and analysis at the country level, instead, we focus on the organisational level. In a blog post, Massimo Graae Losinno explains how SUPER MoRRI has engaged a panel of Country Correspondents. The monitoring activities cover all EU countries, Norway, and the UK. For each of these 29 countries, a Country Correspondent contributed to data collection and reporting. Country Correspondents were chosen on the basis of their expertise in social science methods, their knowledge of the national research and innovation system in their country of residence, and their familiarity with RRI and monitoring. The Country Correspondents are responsible for conducting interviews and document analyses in relation to studies examining 55 Research Funding Organisations (RFOs, 2 per country) and 142 Research Performing Organisations (RPOs, 4-8 per country) in their respective countries. With this approach we build upon local knowledge and qualitative methods across diverse political, cultural, and social settings, as well as diverse research and innovation environments.

The substantial qualitative data collection effort of the Country Correspondents is complemented by the repeated collection of secondary data that were presented in MoRRI as well, and hence provide timeseries. In the first monitoring report (i.e. D2.2), secondary data on R&I expenditure, patent applications, labour market participation (gender equality), bibliometric data on open access, and Eurobarometer data are presented in a responsible way. The second and third monitoring reports are available shortly from the SUPER MoRRI website. Here the results of the RFO and RPO studies are included, which provide for the first time extensive organizational level data.

From the beginning, SUPER MoRRI aimed to create impact by connecting to other funded SwafS-projects on RRI-related topics. And the wish from the EC to take a global perspective on responsibility too, has been implemented through the Global Satellite partners and the annual events we organized. This will be described in the second blog post on SUPER MoRRI’s European ecosystem and global network.