The detailed results of each phase are available in additional files 1 to 3. This results section will focus on the collective action plan that is the culmination of the elements that emerged from the three phases of the project. These elements have been grouped around four main axes that structure the action plan. These axes will be presented, as well as the 14 more specific actions proposed to support these axes. Figure 2 provides a summary of the action plan.
AXIS 1 – Raise awareness of the importance of evidence-informed decision-making among stakeholders in the health and social services network
1.1 Supporting evidence-informed decision-making
Promoting an organizational culture in the health and social services network that values evidence-based decision-making was the course of action considered most important in the survey. Almost 93% of participants considered this course to be very or extremely important. Proposals on how to operationalize this approach included implementing communication strategies to stress the importance of evidence-informed decisions and raising awareness among users of the various evidence syntheses. To achieve this, the proposal was to make them more aware of the range of types of synthesis methods and to explain that the choice of method depends on the nature of the decision-making needs and of the context. At the workshop it was stressed that, although more methodological shortcuts might be acceptable in times of crisis given the urgent need for the response, RES should not become the norm in all contexts. Participants in the deliberative workshop also stressed the importance, in reports, of being transparent about methodological trade-offs and their impacts so as to better inform decision-making. They also noted the need to sensitize decision-makers to the importance of independence and impartiality in the production of syntheses.
1.2 Promoting RES services
This course of action was recommended by participants in the open-ended questions of the survey and discussed during the workshop. The proposed means were to set up teams in organizations that would be dedicated to coordinating or producing RES in times of emergency and to make the services of teams and organizations that support decision-makers in the health and social services system and the general public more widely known.
In addition, in the initiatives found in the literature, it is possible to identify many teams that were set up to produce different products for evidence syntheses (n = 47). Four broad categories of products can be identified: 1) literature monitoring, to keep abreast of the latest scientific evidence and track progress in developments and research trends; 2) summaries of scientific evidence, which allow the results of studies or syntheses to be made available in a short, popularized format; (3) rapid evidence synthesis, to integrate research findings from studies into the body of evidence available on a topic; various synthesized products were developed, such as rapid responses, rapid guidelines, clinical guidelines, and practice points; and the methods used for the syntheses were variable; and finally, 4) the continuous updating of rapid evidence syntheses to take into account the rapid evolution of scientific evidence. The terms “living” or “rolling” are used to characterize these products.
1.3 Ensuring sufficient resources for evidence synthesis
It is important to ensure that decision-support teams and organizations have the means and resources needed to carry out their mission, especially in times of crisis, and that their mission be considered essential. During the workshop, it was noted that several people with tasks related to evidence synthesis (HTA professionals, information specialists) were relieved from their usual duties during the pandemic in Quebec, when they could have contributed to supporting decision-making in a time of crisis.
Given the short time allotted for producing syntheses, one facilitating measure mentioned in the literature is to enlist a team with expertise in evidence syntheses, with members who have previously worked together [11, 32, 41–45]. There is a need for staff who are trained and dedicated, have established relationships of trust, and are able to work in a variety of contexts. Moreover, due to the magnitude of the task during the pandemic, some authors also cautioned that the workload was excessive and unsustainable over the long term. They called for a reduction in the pace of work to avoid staff burnout [46, 47] and for investment in infrastructure to enable rapid deployment of highly qualified staff [48].
AXIS 2 – Promote optimal collaboration of key stakeholders in the production of RES to support decision-making
2.1 Ensuring strategic coordination province-wide
To ensure strategic coordination province-wide in the event of a health crisis, participants in the survey and the deliberative workshop recommended implementing a "crisis cell". This infrastructure would involve having a strategic committee attached to the Ministry of Health and Social Services to quickly identify evidence needs and bring together key expertise, including managers from the ministerial departments concerned (e.g., public health administration, clinical departments), managers of decision-making teams and organizations, and the Ministry of Public Security. Such coordination would also involve a tactical committee that would bring together the decision-support teams to pool resources.
Several infrastructures were quickly set up to meet urgent demands for scientific evidence during the COVID-19 pandemic. To ensure the sustainability of the structures put in place, several authors have stressed the importance of institutional commitment, adequate funding, an integrated organizational structure, a formal definition of each person’s role, and qualified personnel [49–52]. The same observations have also been made with respect to the technological tools developed that require a long-term financial investment in order to ensure that web platforms are updated and to make them accessible and useful for user communities [53].
In addition, ensuring that rapid synthesis processes can meet the needs of all regions of Quebec was considered the fourth most important course of action. As such, a provincial coordination infrastructure is also needed to ensure that local contextual and experiential data from the regions are taken into account.
2.2 Fostering a community of practice for RES
In line with the previous course of action, participants suggested strengthening the interactions among decision-support teams and organizations in order to pool efforts, promote mutual learning, and share methods and the projects they are working on in times of emergency. This would involve creating a virtual sharing platform where the teams producing evidence syntheses would share their monitoring findings, submit their productions, and announce the synthesis projects underway to better coordinate productions. The importance of strengthening collaboration within the network (both intra- and inter-organization) was suggested by both survey respondents and workshop participants, as it remains a major challenge.
Moreover, improving the sharing of RES to avoid duplication was an important course of action in the survey, as 90% of participants considered it very or extremely important. Also, during the workshop several means were proposed for improving evidence centralization in times of crisis, such as pooling information monitoring or grouping products within a single strategic location.
2.3 Developing multi-level collaborations
This course of action is aimed at encouraging and strengthening provincial, national, and international collaborations in the event of a health crisis in order to improve concerted action. Also, the importance of creating more links with research teams and engaging university researchers in decision-making processes was raised in the survey and at the workshop.
Given the global scale of the COVID-19 pandemic, many organizations in each country have been mobilized to identify, summarize, assess, and synthesize existing evidence to inform decision-making. Various efforts have been made to create networks and consortia among the evidence-producing communities to foster inter-organizational collaboration. In this context, it is essential to foster collaboration between organizations at provincial, national, and international levels [32, 42, 44, 48, 54, 55]. This is necessary to avoid duplication and overlapping of work, to prioritize issues and spread out the projects to be done, and to share methodologies and resources available and under development [6, 11, 32, 42, 56]. For example, the COVID-19 Evidence Network to support Decision-making (COVID-END) is an international network initiated at McMaster University that has brought together more than 50 organizations around the world with expertise in evidence synthesis, HTA, and the development of guidelines to support decision-making [54]. COVID-END offers a variety of services and resources for the production of evidence syntheses.
2.4 Involving stakeholders in RES
It is recommended to involve key people from the beginning of a RES project. These may include stakeholders, health professionals, members of the public, service users or patient partners, and researchers. Several ways to operationalize this course of action were proposed, such as recognizing in times of crisis the importance of different types of evidence (including experiential and contextual evidence) in syntheses, maintaining communication with partners throughout the project, encouraging interdisciplinary engagement, and having a librarian involved from the beginning.
With respect to involving patient partners, users, and members of the public in RES in times of crisis, several means were proposed during the group interview and the workshop: valuing their expertise; calling on networks already mobilized or existing communities; involving partners promptly to keep them mobilized and responsive; valuing diverse profiles and promoting representativeness; including the partners throughout the process by considering them to be full members; jointly defining expectations and providing guidelines; addressing disability situations and discussing possible accommodations; and providing compensation modalities tailored to their situation.
The literature also emphasizes the need for greater involvement of stakeholders (e.g., service users, clinicians, or managers) from the beginning to the end of the RES process [44, 48, 54, 57, 58]. This is suggested to promote better consideration of the factors that could influence the conclusions and contextualization of the results, as well as to formulate feasible and acceptable recommendations. A mechanism for feedback and communication with stakeholders should be set up from the start of the process [59]. For example, one initiative identified was created by the Cochrane Collaboration, which set up a group of stakeholders experienced in partnership engagement. The purpose of this group was to recruit people at risk of or with COVID-19, connect them with production settings, and provide guidance and resources to train synthesis producers and service users on the evidence synthesis process and partnership engagement (Cochrane’s COVID-19 consumer rapid response task group).
AXIS 3 – Use a variety of RES methodologies known to be effective in supporting decision-making
3.1 Choosing a context-relevant evidence synthesis methodology
In times of health crisis, it has been proposed that the focus should be on a proportionality of efforts approach. This is a strategy, suggested during the workshop, whose aim is to determine the right methodology to address a decision-making need based on the context and nature of the request. It has been said that more biases could be acceptable when a response to the need cannot wait. This would allow for a decompartmentalized continuum of methodologies (beyond the traditional rapid review and systematic review) and the use of a range of evidence synthesis methods. To properly determine methodological choices in RES, the stage of clarifying the request from managers and decision-makers is retained as a priority, even when the request is urgent. This stage is done to specify and validate their needs as well as to sequence the steps involved in producing the synthesis, in order to respond to the request as quickly as possible.
3.2 Adopting methodological guidelines and quality criteria
It is important to apply the recognized methodological benchmarks and quality criteria for each type of method used. These criteria must uphold integrity, transparency, rigour, and ethics. The importance of transparency and of further clarifying the limitations of the methodological adaptations and, above all, their impact on decision-making needs was repeated many times during the workshop. In this regard, it was recommended that it be made clear in the reports which findings come from preprint studies and non-peer-reviewed reports, among others. Independent double verification was an adaptation often made during the COVID-19 pandemic to speed up evidence synthesis processes. However, it was recommended that a certain way of working in pairs be consistently applied to limit potential biases despite the urgency to act.
In the literature, it is suggested that methodological guides be developed to standardize and formalize the RES process [32, 41, 48, 60–62]. During the pandemic, several organizations published methodological guides on RES, such as Cochrane, the Haute Autorité de Santé (HAS), the Institut national d’excellence en santé et en services sociaux (INESSS), and the National Institute for Health and Care Excellence (NICE). This process needs to be clear, rigorous, and transparent, but also flexible and adaptable due to the evolving nature of the literature.
3.3 Providing useful recommendations for decision-making
The importance of improving practices to make clear, specific, and operational recommendations, where applicable, was another important course of action in the survey, with 92.5% of participants rating it as very or extremely important. During the workshop, it was noted that when an RES is received at the ministry, work must be done to take its conclusions and translate them into decisions according to different scenarios. The roles and responsibilities of the evidence synthesis communities in translating data into recommendations were discussed several times. Even though this is what decision-makers need, many producers expressed their discomfort in making recommendations, because in a crisis context, the synthesis processes do not comply with the usual quality standards. Some preferred using the terms “main findings” or “positions” in their RES rather than issuing recommendations. It was suggested during the workshop that consideration be given to separating the recommendations from the evidence synthesis document or providing more contextual information before making recommendations. In anticipation of the next health crisis, it was suggested to have guidelines for integrating experiential, contextual and scientific evidence into clear, specific and operational recommendations, with a view to better supporting decision-making.
AXIS 4 - Use effective strategies to promote the dissemination, sharing, and use of RES products to support decision-making
4.1 Engaging knowledge translation specialists
It was recommended that people with expertise in communication and knowledge translation be involved from the outset of the project in order to develop a dissemination plan. During the workshop, the suggestion was made to identify people who would help facilitate exchanges between stakeholders. These people can play a translation role between the decision-makers’ needs and the decision-support teams producing the evidence syntheses. They can also begin developing a dissemination and translation plan. Key persons should be located at the Ministry of Health and Social Services to support ministerial actors in making decisions with the help of evidence synthesis products.
Along these lines, it was pointed out during the workshop that one of the keys to success during the pandemic was the close collaboration established with potential users from the beginning of the rapid syntheses. These collaborations facilitated the triangulation of information essential to decision-making (experiential, contextual, and clinical-administrative evidence), made it possible to assess the feasibility of the recommendations, and improved community adherence to recommendations. People in this intermediary role helped create the networks needed for knowledge translation.
To expedite the dissemination of results, the literature also recommends collaborating with the knowledge translation team from the beginning of the project and maintaining regular communications [11, 55, 57]. This will make it possible to prepare a knowledge translation plan early in the process and to develop appropriate strategies that can be deployed quickly to reach user communities.
4.2 Using varied and proven communication strategies
The aim of this course of action is to encourage the use of diverse communication strategies known to be effective in reaching different audiences (e.g., offering incentives to encourage people to participate in activities, writing executive summaries in plain language with key messages highlighted, ensuring the accessibility of documents, using existing communication platforms to reach target audiences [26]). In the survey, 87.5% of participants considered it important or extremely important to present the RES in a brief, easy-to-navigate, and visually appealing format. To this end, recommendations were made to take into consideration the literacy level of the target population, to adopt an inclusive writing angle, and to consider innovative strategies for reaching some populations that are more resistant to certain recommendations (e.g., collaborating with champions or leaders recognized by these populations). Also, during the workshop, participants stressed the importance of providing more training in knowledge translation and strategic writing to the teams producing rapid syntheses, as RES need to get to the essentials, clearly and concisely. Workshop participants also raised the importance of translating scientific knowledge into practical information for settings to facilitate its uptake in times of crisis (e.g., developing guides for practice settings).
To facilitate timely access to results, the literature suggests diversifying knowledge translation strategies, such as the use of email, social media, YouTube, blogs, and podcasts [11, 63]. The use of simple, succinct summaries that highlight key messages for user communities is also recommended [11]. Finally, products can be easily accessed via web platforms, such as directories [48, 49].
4.3 Supporting and media-training the experts
The recommendation was made that people who are asked to participate in media outings be supported and trained to use accessible language and to be aware of the posture they should adopt (e.g., communicating scientific facts and not an opinion). In the survey, 85% of participants deemed improving the communication of RES when they are disseminated in the media and social networks to be very or extremely important. Also, in the workshop, the importance of reviewing how public communications were conducted during the COVID-19 pandemic was raised. Explaining more clearly what is being done to deal with a situation filled with uncertainty and the fact that the situation is evolving in line with the science would allow to readjust the aim and correct the messages in step with this evolution. Transparency was recommended, as well as taking into account the impacts of such transparency on the population. This would help the public better understand the processes and what the decisions are based on. In this respect, on several occasions, participants stressed the importance of generally strengthening the capacity of organizations to take into account literacy levels, in order to adapt communications to different target audiences. In addition, when organizations’ positions diverged from those of other organizations or provinces, it was suggested that the reasons for the differences be clearly explained.
4.4 Monitoring and evaluating evidence use
It would be important to monitor the use of evidence products and fund research projects to assess the impacts of these products on decision-making processes in times of health crisis. For example, if certain recommendations are not implemented by decision-makers, practice settings, or the general population, such monitoring and evaluation processes would be useful to improve and readjust the communication and knowledge translation strategies. To do this, collaborations could be undertaken between decision-support teams, organizations, and research teams to evaluate the products’ impacts on decision-makers, health professionals, or the general public, and thus ensure better monitoring. This is in line with a recommendation highlighted at the workshop of further engaging academic researchers.
In the literature, a few studies on the evaluation of the use of RES products were identified [48, 64, 65]. For example, a mixed methods study was conducted with healthcare practitioners to assess the impact of guidelines produced by the Australian National COVID-19 Clinical Evidence Taskforce. Data were collected on guideline awareness, relevance, ease of use, trustworthiness, value, importance of updating, use, strengths, and opportunities for improvement. Results showed that more than 50% of respondents reported having used the guidelines for various purposes, such as to inform treatment decisions, compare with other guidelines, and seek new evidence [65]. In other studies, qualitative interviews with key informants were conducted to explore how and to what extent RES products were useful to stakeholders [48, 64].