Concerns have recently been raised about the social impact and ethical considerations of research. In health research, studies have identified a high level of dissatisfaction about how research is being assessed and how its effects are defined mostly in academic or economic terms [1–3]. In more detail, the traditional assessment of health research based on academic outputs is, arguably, linked with (i) a lack of translation of health research into healthcare policy and practice [4, 5] and (ii) an ineffective use of scarce resources in both research and health systems [6]. Therefore, measuring research impact beyond traditional measures is considered crucial [7, 8]. In the search for new measures, ethical issues [9] and the needs of patients and society [10] are given prominence.
Previous research has stated the need for a paradigm shift in measuring the impacts of health research toward more collaborative and multi-dimensional forms of assessment, such as engaging a wide array of stakeholders [11] and considering multi-cultural environments [12] to collectively define impact assessment systems [10, 13]. To this end, the European Commission [14] has proposed the “Responsible Research and Innovation” (RRI) approach to making research more inclusive, responsive and sustainable. The key aims are promoting public engagement, gender equality, ethics, science education and open access in research governance [15]. Overall, RRI means that a wide variety of stakeholders work together to align both research processes and outcomes with the values, needs and expectations of society [16–18].
This paper presents a framework with a tool that facilitates the operationalization of the RRI approach. The Collective Research Impact Framework (CRIF) was developed in the MULTI-ACT project in line with the European RRI action to go beyond the traditional forms of research assessment and to increase the impact of health research on patients and society. A specific focus was on brain diseases, which have been causing a high proportion of disabilities and deaths [19]. Therefore, the effects of brain diseases are significant for the healthcare system, increasing the heavy burden for society, the economy, patients and their families [20, 21]. The premise of MULTI-ACT lies in the observation that sustainable research needs a transformational mission and collective impact measurement [13]. Instead of single solutions provided by individual organizations, reaching an impact on patients and society requires collective action and a shared mission defined by all the relevant stakeholders. This innovative co-accountability strategy of MULTI-ACT calls for novel frameworks and impact measurement systems that respond to different demands of stakeholders and enable the alignment of a plurality of interests toward a shared mission (see [22]).
Despite the growing importance attached to RRI and creating innovations that can better address population health needs, there are currently no tools that can help to evaluate RRI requirements in the health sector. A considerable number of studies have focused on developing frameworks for implementing RRI (see e.g. [16, 23, 24]), but there is a lack of tools allowing the assessment of whether and how research and innovations are actually following the RRI approach, aligned with the needs of society or considering the patient as the core element of the impact measurement. Hence, to address these needs and challenges, the CRIF and a specific impact measurement system were developed to assess the impacts of multi-stakeholder initiatives by emphasizing multi-dimensionality and patient engagement.
The CRIF is a responsive model that allows the participation of multiple stakeholders involved in health research and innovation. The framework takes into account the diversity of interests of stakeholders as well as reflecting the variety of measurement aspects related to health research impacts. Multi-dimensionality is based on a broad understanding of the return on investment aligned with the needs and values of different stakeholders, such as patients and their caregivers, rather than purely focused on short-term financial outcomes. In aligning divergent priorities of different stakeholders, the measurement system enables the co-accountability of all relevant stakeholders to progress toward the shared mission for the benefit the whole of society.
In a time of sustainability challenges that involve transformative missions such as those in the health field, the future RRI requires new multi-stakeholder and multi-disciplinary models of cooperation that guarantee a long-term return on investment, not only economic. One innovative feature of the CRIF versus existing frameworks is that it considers the mission-related dimension as one explicit driver for accountability. In doing so, it introduces the dimension of the efficacy of a research initiative interpreted and evaluated as to its capacity to fulfill the shared mission. Efficacy (along with the other impact dimensions detailed hereafter) acts as a pivotal element to promote research programs and projects having an impact on patients and society. Around this core revolves the development of high-quality health research (excellence), which has to be aligned with the mission success of health research (efficacy), the co-participation of all the stakeholders who are directly or indirectly participating in the field (social), while enabling financial sustainability (economic). The fifth dimension (patient-reported outcomes) is transversal, to be applied across the other four dimensions. It covers investigating the impacts on patients and people affected by a disease and highlighting the active engagement of stakeholders during the whole research process. Figure 1 shows the five accountability dimensions of the CRIF that reflect the interrelated perspectives of impact measurement: excellence, efficacy, economic, social, and patient-reported outcomes.
As an integral part of this framework, the MULTI-ACT Master Scorecard (MSC) was designed to foster the collective evaluation of impacts. So far, most of the conventional research measurement systems for assessing the impact on people or health have not been effective, as they lack shared impact measures or a supporting infrastructure to allow for the true alignment of efforts and accountability of results. This has discouraged the true commitment of the various stakeholders to be co-accountable and thus narrowed the impact measurement of health research. To overcome these issues, the MSC was developed as a practical tool that can be used for collaborative decision-making in assessing the return on investment that best reflects the mission, relevant claims and issues for each stakeholder, including benefits for patients and society. The MSC can be applied at the beginning or during the development of a research initiative to engage multiple stakeholders in collectively defining the impact indicators toward a given mission.
This study describes the development process of the MSC, which, consistently with RRI, was multi-stakeholder in nature, combining expertise in different fields with literature reviews, individual interviews and focus groups with experts, patient organizations and practitioners. The contribution of this paper is twofold. First, the study attends to the call of going beyond conventional metrics for assessing the impact of research [3, 10] and implementing RRI [16, 25–26]. Second, by doing this, the article presents the MSC with the multi-dimensional impact indicators to assist future multi-stakeholder initiatives in the field of health research to measure their impact, particularly their collaborative actions aligned with the RRI approach.
The remainder of the paper is organized as follows. The next section lays out the methods and the key developmental phases of the MSC. In section 3, the results of the analysis and thus the advanced MSC with its functionalities and multi-dimensional impact indicators are presented. The paper concludes with a discussion of the potential value and the usability of this kind of collaborative measurement tool for implementing and assessing the RRI actions of future multi-stakeholder initiatives.