Developing new indicators to assess RRI uptake in research and innovation may be controversial

Starting as a primarily academic debate, Responsible Research and Innovation (RRI) has increasingly gained significance both for science, technology and innovation policy and research and innovation practices, in the past few years. However, efforts to make RRI more mainstream across the European Research Area have, so far, been modestly successful. Without adequate indicators providing evidence about the effects of RRI, a broad roll-out of the concept will, arguably, not receive sufficient and sustained political backing. This explains why the demand for practical approaches and instruments–which facilitate the uptake of RRI–is becoming more and more urgent. In particular, there is rising demand to measure the effects of RRI.

In this context, the European Commission is funding a large study on ‘Monitoring the Evolution and Benefits of RRI’ (MoRRI), which will run until 2018. It maps out and assesses existing metrics capturing RRI. It also aims at developing new indicators specifically designed to monitor RRI-related impacts. As such, the project partially draws on and develops further the work of the Expert Group on Policy Indicators for RRI.

The MoRRI approach

Scientific communities are still in the process of defining the meaning of the term “responsibility” in research and innovation. Meanwhile, the European Commission’s current definition of RRI has operated through its thematic elements, or “RRI keys”; namely public engagement, open access, gender, ethics and science education.

The MoRRI project is required to build on these five key RRI dimensions, complemented by the horizontal dimension of governance. To develop a solid conceptual framework capable of addressing the complex nature of RRI, the MoRRI project builds on ideas based on the scientific literature pertaining to evaluation.

It introduces the ‘intervention logic model’ as a starting point for monitoring various types of impacts and benefits of RRI. This model considers that a series of issues or problems that need to be addressed that characterise the complex policy problems. It is built on a set of inputs that are applied to a series of activities, which generate outputs. These, in turn, lead to outcomes or the resolution of the problems.

This logic can be understood as ‘theory of change’, which is based on assumptions as to why an intervention will generate certain outcomes and impacts. The elaboration of such a theory can strengthen the case for attributing observed changes associated with an action. The identification of key contextual factors that could influence the intervention is crucial in this model.

Guided by the intervention logic model, the indicators have been developed following a multi-stage process. First, the project team established an initial inventory of 98 potential indicators. These stem from an assessment of international and EU-funded projects and draw on an extensive review of relevant literature and available data sources.

The team then narrowed these indicators down to 6 for each of the RRI Keys. To do so, the MoRRI project examined whether the indicators include EU-28 coverage, the possibility for sustained, long-term data collection, the balance of targeted societal actors, the mix of quantitative and qualitative data and aspects such as content validity and reliability.

The final indicators resulted from a combination of an extensive assessment, synthesis and considerations about robustness, richness and RRI relevance. Out of the total of 36 indicators, 13 exploit secondary data, while the remaining 23 require primary data collection. Multiple methods will, therefore, be necessary to collect the data.

Critical reflection on RRI’s benefits

Despite the growing pressure to generate indicators in policy contexts, the establishment of new metrics remains contested and open to misunderstandings. Particularly, when these metrics are about concepts that are highly complex themselves, such as RRI. Although the wider use of quantitative indicators can support the transition to more open, accountable and outward-looking research and innovation systems, inappropriate indicators could easily create perverse incentives and unintended consequences. Monitoring exercises and appropriate indicators can support processes of transformation towards more open science.

The use of indicators is fine as long as it is made clear what aspect of RRI they do not cover or what they may hide. Hence, ‘RRI scoreboard’ with a too narrow set of RRI indicators could have potential pitfalls that would need to be taken into account.

The current RRI debate in Europe stresses that research and innovation activities need to become more responsive to societal challenges and concerns. The indicators that are suggested for the monitoring of RRI in EU member states by the MoRRI project assume such rationality. But we should be aware that other rationalities are equally present. It is worth remembering that the MoRRI indicators, however useful they may be, only measure part of the reality of research and innovation.

Ralf Lindner

Ralf is a Senior Researcher at the Fraunhofer Institute for Systems and Innovation Research ISI, Germany, and the coordinator of and RRI project called Res-AGorA.

This opinion piece has been made possible thanks to the joint work of the MoRRI research team, including Ingeborg Meijer, Niels Mejlgaard, Richard Woolley, Ismael Rafols, Erich Griessler, Angela Wroblewski, Susanne Buehrer, Jack Stilgoe, Lena Tsipouri, Nikos Maroulis, Viola Peter, and Ralf Lindner.

Featured image credit: Jennifer Burke via Unsplash

Go back to the Special Issue: RRI Implementation