A bespoke rapid evidence review process engaging stakeholders for supporting evolving and time-sensitive policy and clinical decision-making: reflection and lessons learned from the Wales Covid-19 Evidence Centre 2021-23

doi:10.21203/rs.3.rs-3878814/v1

Background

The COVID-19 pandemic presented policymakers with time-sensitive decision problems and a rapidly increasing volume of research, not all of which was robust, or relevant to local contexts. A bespoke evidence review process supporting stakeholder engagement was developed as part of the Wales COVID-19 Evidence Centre (WCEC), which could flexibly react to the needs of decision-makers, to address urgent requests within days or months as required.

Aims

To describe and appraise the WCEC review process and methods and identify key learning points.

Methods

Three types of rapid review products were used, which could accommodate the breadth of decision problems and topics covered. Stakeholder (including public) engagement was integrated from the onset and supported throughout. The methods used were tailored depending on the needs of the decision-maker, type of research question, timeframe, and volume and type of evidence. We appraised the overall process and compared the methods used with the most recent and relevant best practice guidance.

Results

The remote collaboration between research teams, establishing a clear pathway to impact upfront, and the strong stakeholder involvement embedded in the review process were considered particular strengths. Several key learning points were identified, which focused on: enhancing stakeholders’ abilities to identify focused policy-relevant research questions; the collection and storage of review protocols at a central location; tightening quality assurance process regarding study selection, data extraction and quality assessment; adequate reporting of methodological shortcuts and understanding by stakeholders; piloting of an algorithm for assigning study design descriptors, and a single quality assessment tool covering multiple study designs; and incorporate, where appropriate an assessment of the confidence in the overall body of evidence using GRADE or similar framework.

Conclusions

The review process enabled a high volume of questions that were directly relevant to policy and clinical decision making to be addressed in a timely manner using a transparent and tailored approach.

Health Policy

Evidence synthesis programme

Rapid reviews

Stakeholder involvement

COVID-19

Pandemic

Health- and care-related policy and practice decisions should be based on relevant and trustworthy research evidence, but this relies on providing policymakers and their advisors with timely and accessible evidence [1]. Effective communication and collaboration between researchers, topic experts, and decision-makers are key elements in achieving impact from research. The COVID-19 pandemic demanded new ways of working between academics, policymakers and others making health and social care practice decisions to address time-sensitive decision problems within an ever-changing environment and evidence base. Identifying and synthesising the rapidly increasing volume of available research evidence, not all of which was robust, or relevant to specific local contexts was an important challenge.

Systematic reviews represent the gold standard for informing policy and practice as they provide a comprehensive, rigorous, and transparent synthesis of the evidence. They use standardised and empirically tested methods to minimise bias and error. However, they can take years to complete. One alternative approach is a ‘rapid review’ - an abbreviated systematic review, where processes are streamlined or omitted, to produce evidence for policy and decision-makers in a timely (and resource-efficient) manner [2]. However, even rapid reviews can take six months or more to complete [3, 4], whilst policy and practice decisions were needed within days or weeks during the pandemic. Further rapid evidence review products, that either modify or use alternative methods, have been developed. Hartling et al. [5] developed a taxonomy of these products, based on the extent of synthesis conducted (Box 1), which includes four categories: evidence inventories, rapid response briefs, rapid reviews, and automated products.

Box 1 Taxonomy of rapid review products

Evidence inventories - list of available evidence with no attempt to appraise, synthesise or present conclusions or recommendations (e.g. systematic maps, scoping reviews).

Rapid response briefs - summary without formal synthesis of the best available evidence for addressing a specific question, generally based on the conclusions of existing synthesised evidence such as systematic reviews and clinical guidelines.

Rapid reviews – appraisal and synthesis of the evidence for generating new conclusions using abbreviated systematic review methods for completion within a short time.

Automated products – computer programme generated analysis addressing user-defined questions derived from a database of evidence created using (unconnected) systematic search, screening and data extraction.

Hartling, 2015 doi:10.1016/j.jclinepi.2015.05.036 [5]

Rapid evidence review products have demonstrated great utility for decision-makers, especially during the COVID-19 pandemic [6]. However, there are several key considerations in their development. Firstly, they are demand-driven and produced to support a specific decision by a particular end user [4, 5, 7]. This, and the timeframe of the decision problem, drives the choice of methods used [5]. Secondly, they require a continuous and close relationship with the end user, involving iterative feedback throughout the work [5], which is essential when restricting the scope of the review, to ensure the findings are directly relevant to decision-making [5, 7]. Thirdly, having a team that includes research staff experienced in systematic reviewing is critical for developing an expedited product [5]. Lastly, the COVID-19 pandemic, with its characteristic need for evidence to address rapidly evolving challenges, highlighted the need to avoid duplication across review groups.

The Wales COVID-19 Evidence Centre (WCEC) was established by Welsh Government in March 2021 to enhance the use of research and evidence in managing the pandemic. It aimed to provide health and social care policy and practice decision-makers timely access to the latest relevant COVID-19 research evidence.

The purpose of this paper is to: 1) describe the bespoke evidence review process developed by WCEC that takes account of the important considerations above, with the aim of supporting the agile and timely production of robust evidence reviews whilst maintaining strong stakeholder engagement to ensure direct relevance to decision-making, and 2) appraise the overall review process and evidence review methods, their strengths and weaknesses and identify further improvements that could be made.

The Wales COVID-19 Evidence Centre (WCEC)

The WCEC brought together a unique collaboration of established research groups within Wales with expertise in conducting rapid reviews, systematic reviews, health technology assessments, economic evaluations, and the analysis of linked population-level routinely collected data. The WCEC operated through a core management team working closely (using videoconferencing) with the Collaborating Partner research teams (Box 2).

The WCEC undertook evidence reviews to address knowledge gaps and the specific needs of government, healthcare, public health and social care stakeholders in Wales. The evidence produced was designed to be of immediate use to decision-makers and to have a direct impact on decision-making, patient and client care, reducing inequalities and identifying future research needs. The work of WCEC was delivered through four main processes: Question Prioritisation Process, Evidence Review Process, Knowledge Mobilisation Process, and Stakeholder Engagement (including public involvement). This paper focuses on the Evidence Review Process, and the Stakeholder Engagement that supports this. The processes for prioritising and setting research questions, and knowledge mobilisation, are described in more detail elsewhere [8, 9].

BOX 2: Wales COVID-19 Evidence Centre (WCEC) Collaborating Partners

WCEC operated through a core management team working closely with six Collaborating Partners:

• Health Technology Wales (HTW) - http://www.healthtechnology.wales/

• Wales Centre for Evidence-Based Care (WCEBC)- A JBI Centre of Excellence - https://www.cardiff.ac.uk/research/explore/research-units/wales-centre-for-evidence-based-care

• Specialist Unit for Review Evidence (SURE) centre - https://www.cardiff.ac.uk/specialist-unit-for-review-evidence

• Public Health Wales Evidence Service - https://phw.nhs.wales/services-and-teams/observatory/

• Bangor Institute for Health & Medical Research (BIHMR) - Centre for Health Economics and Medicines Evaluation - https://cheme.bangor.ac.uk/research/whess.php.en - in conjunction with Health and Care Economics Cymru (HCEC) - https://healthandcareeconomics.cymru/

• Population Data Science - SAIL Databank - https://saildatabank.com/

The core management team comprised a Director and leads for each of the four processes: prioritisation process, evidence review, knowledge mobilisation and impact, and stakeholder engagement. It worked closely (and remotely) with a Public Partnership Group and members of the Welsh Government’s Technical Advisory Cell and Technical Advisory Group (TAC/TAG – sometimes referred to as “Welsh SAGE”) [10]. There was also a methodology subgroup, with representation from all Collaborating Partner groups, meeting on-line fortnightly for methodological support and to share good practice. Members of the Public Partnership Group (PPG) provided public involvement in each review and are involved in the knowledge mobilisation process.

Development of the WCEC evidence review process

The WCEC sought to develop an evidence review process that could deliver robust reviews within four to eight weeks but with flexibility to provide decision-makers with a credible summary of the available evidence within days or weeks when needed. We considered the range of rapid evidence review products identified by Hartling et al (2015) (Box 1), but we were also mindful to avoid having too many types of outputs, as this could be confusing to stakeholders [11]. We developed a phased reviewing approach [12, 13] which utilises three types of rapid review products: a rapid response product (which is called a Rapid Evidence Summary), an evidence inventory product (called a Rapid Evidence Map), and a Rapid Review. These are described in more detail in Table 1.

Best Practice Framework

Our overall process and methods development were informed by guidance for conducting and reporting rapid evidence review products [7, 11–18]. The methods selected for our Rapid Reviews were adapted according to the topic area, type of review question, the extent of the evidence base, urgency of the questions, and the needs of the decision-makers. To support the Collaborating Partner review teams, a Best Practice Framework (Table 2) was developed with recommendations from key sources for methodological shortcuts that could be applied at each stage of the rapid review.

Three key guidance documents were prioritised for developing the Framework summarising the recommendations for best practice of conducting a rapid review [7, 13, 18]. We also referred to two existing guidance documents, developed and already used by two Collaborating Partners for conducting rapid reviews [11] or rapid health technology assessments [19].

The Review Process

The phased review process is outlined in Figure 1 and described in more detail in the next section. Each review was conducted by a dedicated Collaborating Partner review team supported by the core management team. A continuous and close relationship with the decision-maker and relevant stakeholders (including Public Partnership Group representation) was facilitated by three or more online stakeholder meetings.

Question prioritisation process

The review question(s) were submitted by stakeholders (e.g. policymakers/advisors, health and social care leads, public, academic/research groups) and prioritised during a formal consultation process, which is reported in detail elsewhere [9]. Urgent questions could also be submitted directly by policymakers or TAC/TAG members and ‘fast-tracked’ onto the WCEC work programme. Key stakeholders, including those submitting the question and members of the Public Partnership Group (PPG), provided expert (topic and methodological) input throughout the evidence review process.

Review Process Phase I: Rapid Evidence Summary (RES)

In phase I, the review question was allocated to an appropriate WCEC Collaborating Partner (review) team, and an introductory stakeholder meeting organised. This early phase comprised preliminary work to inform the Rapid Review work. However, it was adaptable to produce a final rapid response product (Table 1) within weeks if no Rapid Review was planned.

Introductory stakeholder meeting

The stakeholder meetings included members of the core management team and WCEC public partners, the review team, and relevant stakeholders. The introductory meeting was used to confirm the decision problem or review question including key outcomes, clarify how the evidence would be used, and confirm required timelines. It was also an opportunity for stakeholders to notify the review team of potentially seminal research or useful grey literature sources. Where an ill-defined decision problem / question had been submitted in the Prioritisation Process, this meeting also served to develop a structured review question.

Preliminary search of the literature

The review team then conducted a scoping search and a scan of key COVID-19 resources. This was supported by a tailor-made Resources list, including both COVID-19 specific and generic registries and databases of secondary research (Supplementary information, Additional file 1). This preliminary review of the literature enabled the reviewers to familiarise themselves with the topic area, check the research question has not been addressed by other groups or evidence centres, identify the extent and type of available evidence, and inform the methods and design of the rapid review in Phase II (and develop the protocol). The searches focused on identifying robust secondary or tertiary research. Primary studies were considered if no relevant reviews were identified.

Output from Phase I

The output from this first phase was presented as an annotated bibliography with key findings, using a template to support the efficient and transparent reporting of what was done and found. When there was a high priority urgent decision to address, or insufficient evidence for a rapid review, the Rapid Evidence Summary was published as the final output for the stakeholder. For example, our review of ozone machines and other disinfectant in Schools (RES_23) [20].

If an up-to date, robust and directly relevant evidence review or clinical guideline was identified during the preliminary searches then a critical appraisal and summary of the review was conducted. For example, our review of vaccination in pregnant women (RES_24) [20]. If multiple systematic reviews were identified, then a review of existing reviews was considered for the subsequent phase Rapid Review. For example, in our review of innovations to support patients on elective surgical waiting lists (RR_30) [21] and our review of interventions to recruitment and retain clinical staff (RR_28) [22].

Intermediate stakeholder meeting

The findings of the initial phase (if progressing to a rapid review) were presented at a second, intermediate, stakeholder meeting. Collaborative discussions refined the review question, drafted eligibility criteria, and decided on the overall reviewing approach to be used (if proceeding to Rapid Review). Stakeholders identified important contextual issues, known equality, or economic impacts for consideration in the proposed review.

Review Process Phase II: Rapid Review

Phase II comprised a Rapid Review (RR) of the evidence, usually completed within 1-2 months. This could be supplemented or substituted by a Rapid Evidence Map (REM). The Rapid Review delivered a synthesis or meta-synthesis of the evidence, whilst the Rapid Evidence Map provided a description of the available literature (Table 1). Both were based on a comprehensive search strategy and pre-defined protocol.

Rapid Evidence Map

For broad or complex review questions a Rapid Evidence Map could be conducted, providing an inventory of the nature, characteristics, and volume of available evidence for the particular policy domain or research question. The Rapid Evidence Map was based on abbreviated systematic mapping [23] or scoping review [24] methodology, depending on the type of review question. For example, our review of recruitment and retention of NHS workers [20]. Stakeholders could also request a Rapid Evidence Map as the intended final rapid product. For example, in our review of inequity experienced by LGBTQ+ community [20].

Rapid review

Our Rapid Reviews used an adapted systematic review approach, with some review components abbreviated or omitted to generate the evidence to inform stakeholders within a short time frame, whilst maintaining attention to bias. We followed methodological recommendations and minimum standards for conducting rapid reviews [7, 13, 18]. The approach and decisions made on tailoring the rapid reviews were the responsibility of the individual review teams, according to the type of question, research volume and time frame, in discussion with core management team members and expert stakeholders.

Output from phase II

The template for our final Rapid Review and Rapid Evidence Map reports are based on recommendations for reporting evidence reviews for decision-makers [11, 16]. This incorporates a two-page “top line summary”, the results and recommendations for practice presented up front, and the details of the methods used at the end of the report.

Our review reports were made available via a library on the WCEC website [20]. From May 2022, reports were published on a pre-print server and allocated a doi. Thus, reports could be identified readily in database searches, and other review teams could identify potential duplicate review questions early on. A short lay summary and the links to the pre-print server were included in the WCEC library. The ongoing WCEC work programmes, which included questions in progress, scheduled, and completed, was also published on the website.

Knowledge Mobilisation Process - planning pathway to impact

Final stakeholder meeting

A final stakeholder meeting was used to present the findings of the review to the stakeholders, address any queries, identify the policy and practice implications, and support the development of a knowledge mobilisation plan.

Appraisal of the overall review process and rapid review methods

We appraised our overall approach and rapid review methods to reflect on our experience of implementing the WCEC review process and to identify key learning points.

We compared our methods and practice with the recommendations of Garritty et al. [7], Tricco et al. [13], Plüddemann et al. [18], Mann et al. [11], and Health Technology Wales [19], as the principal resources for our own Best Practice Framework (Table 2). We also compared our rapid review methods with the array of methodological shortcuts recommended in published guidance developed or used across rapid review centres and organisations, as reviewed by Speckemeier et al. [25] (Table 3). That scoping review included guidance for any type of rapid evidence product with a completion time ranging from a day to over 6 months. The output included a table summarising the range of recommendations, or methodological shortcuts, provided in the guidance, and the frequency with which they were reported. However, the authors did not provide an indication of which recommendations were optimal.

The approach used for appraising our rapid review methods

We assessed whether our reviews, mainly completed within two months, aligned with our Best Practice Framework, and whether methods aligned across our different Collaborating Partner groups. Findings were presented at a methods subgroup meeting and discussed to reflect on what worked well or could be improved (and how).

As part this appraisal, key data from all Rapid Reviews and Rapid Evidence Maps completed up until March 2023 were extracted. These included data on the search date, overall reviewing approach, limits applied, sources searched, volume of research identified, study selection process, data extraction process, and approach used for quality assessment. An important consideration here is that the approach used depended on the research question being addressed, the volume and type of research available, and the timeframe within which the review was conducted.

Where the methods of individual reviews met or exceeded the recommendation in the Best Practice Framework the text was highlighted green, for recommendations that were either partially or not always met the text was highlighted orange, and where our methods consistently did not meet the recommendation, the text was highlighted in red. We did not seek to identify individual failures or the frequency with which our methods did not meet the recommendations, but to reflect on our overall process and methodological approach used and identify what changes could be made. The colour coded Framework table was presented at a methods group meeting, and participants given a copy of the data extraction table summarising individual reviews.

Results of the appraisal of our methods

The comparison of the methods used in our reviews with the recommendations in the Best Practice Framework is presented in Table 2 as an additional column to the Best Practice Framework. The full details of the methods used within our Rapid Reviews and Rapid Evidence Maps is available in the Supplementary information, Additional file 2. The comparison of our methods with the range of recommendations identified in the scoping review of guidance conducted by Speckemeier et al. [25] is presented in Table 3.

We identified that our basic methods align with or exceed most recommendations for rapid reviews, notably for developing and refining the review question, the use of preliminary work to inform the scope, the searches, synthesis, and report production (Table 2). A potential gap was that, although our reviews are based on pre-defined protocols, which are developed in collaboration with the stakeholders, these are not registered. However, our protocols are made available on request, which is noted in the reports.

Study selection and data extraction were conducted by two independent reviewers in some reviews but were more usually conducted by a single reviewer with or without verification of a sample or excluded citations/manuscripts. Quality assessment was based on critical appraisal or risk of bias tools specific to the study design(s), which accords with most recommendations, but the assessment was often conducted by a single reviewer with or without a verification of a sample. The selection of literature, data extraction and critical appraisal by a single reviewer meets the minimum requirements only [18], and verification sample or the use of two independent reviewers is generally recommended to reduce bias [7, 13, 18]. The assessment of the confidence in the evidence base was generally subjective. The limited number of studies and diversity of outcomes reported in some reviews meant that the GRADE (Recommendations Assessment, Development and Evaluation) [26] assessment was applied to single studies. This was also the reason why some reviews did not include a GRADE assessment.

An important limitation identified in a minority of our earlier reviews is that the methodological short cuts were not stated or clearly described. This is an important consideration for transparency and validity.

Reflection on our methods and reviewing approach and identification of key learning points

The output of the methods appraisal was shared with the review teams at a methods subgroup meeting. Members were also asked to reflect on their experience of the overall review process.

Aspects of the overall process that were thought to be working well included the stakeholder process for formulating relevant questions and the facilitation of the stakeholder meetings. The methodological discussions that ensued between the WCEC core team and the review team, on planning and conducting the proposed reviews were also valued. These were felt to be beneficial for problem solving and learning from each other. The remote working and cross Wales collaboration were also considered a strength, as were the published reports and impact strategy. Establishing a clear pathway to impact was also key for refining the review question. Both these stages could be supported by a network of policy decision-makers with enhanced abilities in both question formulation and impact work.

In terms of our methods, members acknowledged potential discrepancies between reviewers in allocating study descriptors, in particular for poorly reported or less robust study designs. The algorithm developed by Leatherdale [27] [27] for assessing natural experiments and to inform selection criteria was noted as a potential solution, requiring evaluation. The use of a single checklist for assessing the risk of bias covering multiple study designs (addressing the same type of question) was considered potentially beneficial. However, using the validated checklist developed for any non-randomised comparative study of interventions, ROBINs-I [28], was considered challenging within the context of a rapid review and mainly applicable to identifying bias in studies assessing causal effects of interventions. Likewise, GRADE works best for assessing the confidence in the overall body of evidence for interventions that have been evaluated by randomised trials and where there is at least one meta-analysis to provide a single estimate of the outcome effect [7]. Our reviews cover various forms of evidence, including intervention effects, prevalence, prognostic, diagnostic, economic, meaningfulness, and consequence of public health measures. The use of GRADE in very rapid reviews, in particular non-intervention reviews, was considered challenging, even though recommended for use in emergency settings such as the COVID-19 pandemic [29]. Members acknowledged that it should be included where possible. It was acknowledged that adhering to the minimum standards, such as regarding single reviewer screening of the literature or data extracting could lead to bias or inaccuracies. The need to adequately report the methodological short cuts used and the limitations of the review was also re-iterated. The potential value of more in-depth reviews, closer to systematic reviews in methodology (and including for example, network meta-analysis, meta-ethnography or economic modelling), and taking longer to complete when required, was identified. The learning points are summarised in Box 3.

BOX 3: Key learning points for rapid evidence synthesis with impact

• There is a need for a network of key stakeholders with enhanced abilities to identify focused policy-relevant research questions. The provision of training in developing focused research questions may also be beneficial.

• Identifying how the evidence is going to be used during the introductory stakeholder meeting and establishing a clear pathway to impact was key for refining the review question (or narrowing the scope of the review).

• The continuous stakeholder involvement embedded within our review process was a particular strength, facilitated by the remote working and close collaboration between different research groups and organisations across Wales.

• The core management team should collect protocols for all reviews to support making them available on request.

• Agreed in-house minimum standards are needed for the quality assurance processes, whilst acknowledging that these may be adapted according to the review question type, evidence base available, stakeholder needs and time available. Our reviews should align, where possible, with the minimum standardsrecommended in the Cochrane guidance for rapid reviews of interventions (Garritty et al, 2021), and include:

o Screening title and abstract - two reviewers to dual screen at least 20% of citations, resolving all conflicts. One reviewer to screen remaining citations and one to review all excluded citations, resolving all conflicts if needed.

o Screening full text - One reviewer to screen all manuscripts and one to review all excluded manuscripts.

o Data extraction - single reviewer to extract data (using piloted form), with second reviewer checking for correctness and completeness.

o Risk of bias assessment - single reviewer to rate risk of bias, with full verification of all judgments by a second reviewer.

• It is important to adequately report the methodological short cuts used in our reviews and the limitations of the review. An understanding of these by the stakeholders is also essential to establish trust in the reviews.

• The algorithm developed by Leatherdale (2019), for assessing natural experiments, may be useful to assign study design descriptors and inform the selection of study types for inclusion.

• A single quality appraisal tool that covers multiple study designs may be useful for reviews of intervention effects.

• The GRADE system for assessing the confidence in the overall body of evidence for each outcome should be used, where possible.

Summary of the practice and its appraisal

The Wales COVID-19 Evidence Centre developed a review process that could flexibly react to the needs of decision-makers, to address urgent requests within days, weeks or months as required. For each review, the approach used, and methodological shortcuts applied, are tailored depending on the needs of the decision-maker, timeframe, and volume and type of evidence. A Best Practice Framework, which integrates recommendations in key published guidance, was developed to support reviewers at each stage of the reviews.

We appraised our overall process and methods used in 27 Rapid Reviews and five Rapid Evidence Maps. Our methods aligned with or exceeded most recommendations for conducting rapid reviews, particularly those for developing and refining the review question, undertaking preliminary work to inform the scope, conducting the searches, quality assessment, narrative synthesis, and report production. However, our review protocols were not registered, and study selection, data extraction and quality appraisal were generally conducted by a single reviewer, and the assessment of confidence in the evidence base was generally subjective.

The wider context of the literature

Several publications describe the rapid evidence review methods and overall process used in other centres [16, 30, 31]. The guidance and methods developed by these publications were also considered as part of a recent scoping review by Speckemeier et al 2022. Our methods align with or exceeded the recommendations for methodological shortcuts most frequently reported in published guidance.

The trade-off in achieving speed and efficiency in conducting a rapid review is a reduction in the validity of the results and certainty in the evidence [25, 32]. However, empirical evidence of the impact of using specific methodological shortcuts is limited, and few shortcuts are used consistently in rapid reviews [4, 25, 32–34]. There is little consensus over which shortcuts could apply across different topic areas [4, 25, 32–34]. There is evidence showing that limiting the search strategy can increase the risk of selection, retrieval, and publication bias [25]. The selection of literature and data extraction by a single reviewer can lead to relevant studies being missed and inaccuracies in the data extraction [25, 32]. However, the extent of this impact varies depending on reviewer experience and research topic [25, 32, 35–37]. A crowd-based randomised trial [38] found that single-reviewer abstract screening missed on average 13% of relevant studies and dual-reviewer screening missed 3% of relevant studies. It is important that the type and extent of the methodological shortcuts used are clearly reported, so that the extent of the potential bias and limitations of a review can be assessed.

The Cochrane Rapid Reviews Methods Group advocates that the essential element to success is early and ongoing engagement with the research requester to focus the rapid review and ensure that it is appropriate to the needs of stakeholders [7, 32]. The stakeholder involvement process in our reviews was considered an important strength, facilitated by the remote working and close collaboration between different research groups and organisations across Wales. A potential limitation of the appraisal of our methods is that we did not evaluate the views of the stakeholders’ and policy-makers involved in our reviews. Stakeholder satisfaction in our outputs, however, has been evaluated as part of our knowledge mobilisation process and impact assessment, which is reported separately [8].

Implications for future practice and research

Key learning points are summarised in Box 3. Our rapid review process was developed to support the need for urgent or rapid evidence needs during the COVID-19 pandemic. The same process could support rapid reviews with longer time frames (3–6 months) or more systematic reviews to support policy decision-making. The longer the available timeframe the less abbreviation to the systematic review approach or methodological short cuts that are required.

Identifying a specific decision problem is an integral part of the review process. One of the key learning points identified was the need to enhance stakeholders’ abilities to identify focused policy-relevant research questions. The importance of stakeholders in developing and refining the review question, eligibility criteria and outcomes of interest were highlighted by all the key sources included in the Best Practice Framework. Further research is needed to identify the most appropriate methods of engaging stakeholders early in the process to identify evidence needs and how these translate into focussed research questions.

Further research is needed to assess the impact of using various methodological short cuts on the validity of rapid review findings. Such research can also provide the basis for minimum standards to minimise inaccuracies and bias, in particular for non-intervention reviews.

The quality (or risk of bias) assessment provides important information on the trustworthiness of the results of included studies. Recent methodological advances in the field of risk of bias assessment (which focuses on internal validity) advocate a move away from the use of critical appraisal tools that cover additional concepts such as imprecision, external validity, and reporting [28, 39]. They also recommend that the assessment occurs at domain level, supported by signalling questions, rather than using a checklist approach. An example of which includes the ROBINS-I for non-randomised studies [28]. Existing reviews of quality assessment tools identified numerous tools that can be used in systematic or rapid reviews, but few are designed to cover multiple study designs [40–43] and there is no consensus on the most appropriate tools for rapid reviews [32]. Further work is needed to explore the use of a single tool that covers multiple study designs in rapid reviews of intervention effects [40]. Further work is also needed to develop the optimal approach for selecting appropriate study design descriptors, in the context of a rapid review, of real-world natural experiments or quasi-randomised controlled trials. This is likely to be particularly pertinent when conducting a rapid review of service delivery or public health interventions.

Guidance is required on how to assess the certainty or confidence in the overall body of evidence where the GRADE (or GRADE-CERQual [44]) assessment is difficult. Although it is recommended that assessing the certainty of evidence is based on GRADE for Cochrane rapid reviews of interventions [45], it is also acknowledged that it may not always be easy to implement within either the rapid review [7] or emergency preparedness [46] context.

Our bespoke review process enabled us to successfully address a high volume of review questions in a timely manner using a transparent and adaptable approach. The collaboration between established research teams in Wales and the strong stakeholder involvement embedded in the review process were considered particular strengths of the overall review process. A number of key learning points were identified, which focused on: enhancing stakeholders’ abilities to identify focused policy-relevant research questions; the collection and storage of our review protocols at a central location; tightening our quality assurance process regarding study selection, data extraction and risk of bias assessment; the piloting of an algorithm for assigning study design descriptors; and to incorporate, where appropriate an assessment of the confidence in the overall body of evidence using GRADE or GRADE-CERQual in our reviews.

COVID-19: Coronavirus disease

GRADE: Grading of Recommendations Assessment, Development and Evaluation

GRADE-CERQual: GRADE-Confidence in the Evidence from Reviews of Qualitative research

PPG: Public Partnership Group

REM: Rapid Evidence Map

RES: Rapid Evidence Summary

RR: Rapid review

SAGE: Strategic Advisory Group of Experts on Immunization

TAC: Technical Advisory Cell

TAG: Technical Advisory Group

WCEC: Wales COVID-19 Evidence Centre

Ethics approval and consent to participate

In line with guidance, ethical approval was not required for this study.

Consent for publication

Our manuscript does not contain personal data and does not require consent for publication.

Availability of data and materials

All data relevant to the study are included in the article or uploaded as supplementary materials.

Competing interests

The authors have no conflicts of interest to declare.

Funding

The Wales COVID-19 Evidence Centre is funded by Health and Care Research Wales through Welsh Government.

Authors’ contributions

Conceptualization: RL, AC, NJ-W, AE, MG, R-JL, DJ, MM, JN; Data curation and Analysis: RL, DJ; Investigation: RL, AC, DJ, R-JL MM, DE, JC, HS, TW, LHS, JN, HM, MG; Methodology: RL, AC, DJ, R-JL, MM, DE, JC, HS, JN, HM, JW, EH; Project administration: RL, AC, NJ-W, AE, MG; Writing – original draft: RL; Writing – review & Editing: RL, AC, DJ, MM, DE, JC, HS, TW, LHS, JN, HM, JW, EH, MG, ED, R-JL, NJ-W, AE.

Acknowledgements

We are grateful for the engagement and support of the various stakeholders and members of our Public Partnership Group in delivering our rapid evidence review work.

Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J: A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research 2014, 14(1):2.
Hamel C, Michaud A, Thuku M, Skidmore B, Stevens A, Nussbaumer-Streit B, Garritty C: Defining Rapid Reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews. Journal of Clinical Epidemiology 2021, 129:74-85.
Ganann R, Ciliska D, Thomas H: Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci 2010, 5:56.
Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, Perrier L, Hutton B, Moher D, Straus SE: A scoping review of rapid review methods. BMC Med 2015, 13:224.
Hartling L, Guise JM, Kato E, Anderson J, Belinson S, Berliner E, Dryden DM, Featherstone R, Mitchell MD, Motu'apuaka M et al: A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. J Clin Epidemiol 2015, 68(12):1451-1462.e1453.
Tricco AC, Khalil H, Holly C, Feyissa G, Godfrey C, Evans C, Sawchuck D, Sudhakar M, Asahngwa C, Stannard D et al: Rapid reviews and the methodological rigor of evidence synthesis: a JBI position statement. JBI Evid Synth 2022, 20(4):944-949.
Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, Affengruber L, Stevens A: Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. Journal of Clinical Epidemiology 2021, 130:13-22.
Gal M, Cooper A, Doe E, Joseph-Williams N, Lewis R, Jane Law R-J, Anstey S, Davies N, Walters A, Greenwell J et al: Knowledge mobilisation of rapid evidence reviews to inform health and social care policy and practice in a public health emergency: appraisal of the Wales COVID-19 Evidence Centre processes and impact, 2021-23. Manuscript in preperation 2023.
Joseph-Williams N, Cooper A, Lewis R, Greenwell G, Doe E, Gal M, Pearson N, Kumar R, Law R-J, Edwards A: Working with stakeholders to rapidly identify and prioritise COVID-19 health and social care evidence needs during the pandemic period: processes, results, and lessons from the Wales COVID-19 Evidence Centre. Submitted for publication to BMC Health Research Policy and Systems 2023.
Cooper A, Lewis R, Gal M, Doe E, Williams D, Strong A, Greenwell J, Watkins A, Law R-J, Joseph-Williams N et al: Informing evidence-based policy during the COVID-19 pandemic and recovery period: Wales COVID-19 Evidence Centre. Manuscript in preperation 2023.
Mann M, Woodward A, Nelson A, Byrne A: Palliative Care Evidence Review Service (PaCERS): a knowledge transfer partnership. Health Res Policy Syst 2019, 17(1):100.
Lewis R, Hendry M, Din N, Stanciu MA, Nafees S, Hendry A, Teoh ZH, Lloyd T, Parsonage R, Neal RD et al: Pragmatic methods for reviewing exceptionally large bodies of evidence: systematic mapping review and overview of systematic reviews using lung cancer survival as an exemplar. Systematic Reviews 2019, 8(1):171.
Tricco AC, Langlois EV, Straus SE: Rapid reviews to strengthen health policy and systems: a practical guide. World Health Organization https://appswhoint/iris/handle/10665/258698 License: CC BY-NC-SA 30 IGO ISBN 978-92-4-151276-3 2017.
Collins AM, Coughlin D, Miller J, Kirk S: The Production of Quick Scoping Reviews and Rapid Evidence Assessments: A How to Guide. Available from: https://noranercacuk/id/eprint/512448 2015.
Dobbins M: Rapid Review Guidebook. Hamilton, ON: National Collaborating Centre for Methods and Tools; 2017.
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D: Evidence summaries: the evolution of a rapid review approach. Syst Rev 2012, 1:10.
National Academies of Sciences E, and Medicine: Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: National Academies Press (US). https://doi.org/10.17226/25650; 2020.
Plüddemann A, Aronson JK, Onakpoya I, Heneghan C, Mahtani KR: Redefining rapid reviews: a flexible framework for restricted systematic reviews. BMJ Evid Based Med 2018, 23(6):201-203.
www.healthtechnology.wales
Wales COVID-19 Evidence Centre report library [https://healthandcareresearchwales.org/wales-covid-19-evidence-centre-report-library]
Okolie C, Rodriguez R, Wale A, Hookway A, Shaw H, Cooper A, Lewis R, Law R-J, M. G, Greenwell J et al: A rapid review of the effectiveness of innovations to support patients on elective surgical waiting lists. medRxiv 2022:2022.2006.2010.22276151.
Edwards D, Csontos J, Gillen E, Carrier J, Lewis R, Cooper A, Gal M, Law R-J, Greenwell G, Edwards A: A rapid review of the effectiveness of interventions and innovations relevant to the Welsh NHS context to support recruitment and retention of clinical staff. medRxiv 2022:2022.2005.2011.22274903.
White H, Albers B, Gaarder M, Kornør H, Littell J, Marshall Z, Mathew C, Pigott T, Snilstveit B, Waddington H et al: Guidance for producing a Campbell evidence and gap map. Campbell Syst Rev 2020, 16(4):e1125.
Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, McInerney P, Godfrey CM, Khalil H: Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth 2020, 18(10):2119-2126.
Speckemeier C, Niemann A, Wasem J, Buchberger B, Neusser S: Methodological guidance for rapid reviews in healthcare: A scoping review. Res Synth Methods 2022, 13(4):394-404.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H et al: GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011, 64(4):383-394.
Leatherdale ST: Natural experiment methodology for research: a review of how different methods can support real-world research. International Journal of Social Research Methodology 2019, 22(1):19-35.
Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I et al: ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016, 355:i4919.
Schünemann HJ, Santesso N, Vist GE, Cuello C, Lotfi T, Flottorp S, Davoli M, Mustafa R, Meerpohl JJ, Alonso-Coello P et al: Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what. J Clin Epidemiol 2020, 127:202-207.
Chambers D, Booth A, Rodgers M, Preston L, Dalton J, Goyder E, Thomas S, Parker G, Street A, Eastwood A: Evidence to support delivery of effective health services: a responsive programme of rapid evidence synthesis. Evidence & Policy 2021, 17(1):173-187.
Neil-Sztramko SE, Belita E, Traynor RL, Clark E, Hagerman L, Dobbins M: Methods to support evidence-informed decision-making in the midst of COVID-19: creation and evolution of a rapid review service from the National Collaborating Centre for Methods and Tools. BMC Med Res Methodol 2021, 21(1):231.
King VJ, Stevens A, Nussbaumer-Streit B, Kamel C, Garritty C: Paper 2: Performing rapid reviews. Systematic Reviews 2022, 11(1):151.
Abou-Setta AM, Jeyaraman MM, Attia A, Al-Inany HG, Ferri M, Ansari MT, Garritty CM, Bond K, Norris SL: Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review. PLoS One 2016, 11(12):e0165903.
Haby MM, Chapman E, Clark R, Barreto J, Reveiz L, Lavis JN: What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review. Health Res Policy Syst 2016, 14(1):83.
Affengruber L, Wagner G, Waffenschmidt S, Lhachimi SK, Nussbaumer-Streit B, Thaler K, Griebler U, Klerings I, Gartlehner G: Combining abbreviated literature searches with single-reviewer screening: three case studies of rapid reviews. Syst Rev 2020, 9(1):162.
Taylor-Phillips S, Geppert J, Stinton C, Freeman K, Johnson S, Fraser H, Sutcliffe P, Clarke A: Comparison of a full systematic review versus rapid review approaches to assess a newborn screening test for tyrosinemia type 1. Res Synth Methods 2017, 8(4):475-484.
Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D: Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019, 19(1):132.
Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, König F: Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. J Clin Epidemiol 2020, 121:20-28.
Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM et al: RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019, 366:i4898.
Duval D, Pearce-Smith N, Palmer JC, Sarfo-Annin JK, Rudd P, Clark R: Critical appraisal in rapid systematic reviews of COVID-19 studies: implementation of the Quality Criteria Checklist (QCC). Syst Rev 2023, 12(1):55.
Ma L-L, Wang Y-Y, Yang Z-H, Huang D, Weng H, Zeng X-T: Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Military Medical Research 2020, 7(1):7.
Quigley JM, Thompson JC, Halfpenny NJ, Scott DA: Critical appraisal of nonrandomized studies-A review of recommended and commonly used tools. J Eval Clin Pract 2019, 25(1):44-52.
Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, Niu Y, Du L: The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med 2015, 8(1):2-10.
Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, Bohren MA, Tunçalp Ö, Colvin CJ, Garside R et al: Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci 2018, 13(Suppl 1):2.
Gartlehner G, Nussbaumer-Streit B, Devane D, Kahwati L, Viswanathan M, King VJ, Qaseem A, Akl E, Schuenemann HJ: Rapid reviews methods series: Guidance on assessing the certainty of evidence. BMJ Evid Based Med 2023.
Calonge N, Shekelle PG, Owens DK, Teutsch S, Downey A, Brown L, Noyes J: A framework for synthesizing intervention evidence from multiple sources into a single certainty of evidence rating: Methodological developments from a US National Academies of Sciences, Engineering, and Medicine Committee. Res Synth Methods 2023, 14(1):36-51.

Table 1: Summary of the rapid review products included in the WCEC Rapid Evidence Review process

	Phase I: Rapid Evidence Summary (Preliminary search of the literature; topic exploration)	Phase III: Rapid review For broad policy questions, this may incorporate a two-staged process, where an initial descriptive map or scoping review is conducted to inform the focus of the rapid review
*Product type*	Rapid response	Inventory	Rapid review
Timeframe	~1 week	~1-2 months*
Output/ format	Key messages and annotated Bibliography (with links to full text)	Rapid Evidence Map using abbreviated systematic mapping or scoping methods	Rapid Review based on abbreviated SR methods
Purpose	Provide early access to evidence; gauge potential size and type of existing research; inform the Rapid Review methods/protocol/search strategy; support stakeholder involvement.	Provide a description of the available evidence; identify substantial focus; identify existing research and evidence gaps	Provide a summary and direction of effect and possibly strength of the evidence
	NB If an up-to date, robust and directly relevant evidence review is identified during Phase I further review work may not be required; a critical appraisal and summary of the evidence review will likely suffice (with, if necessary, a limited update). Where multiple SRs are identified these will be reviewed as part of the subsequent Rapid Review.
Methods	A preliminary search of key resources (prioritising COVID-19 resources and sources of robust evidence syntheses)	Protocol and full search strategy developed Limitations on scope and comprehensiveness of review applied: limited number of sources searched; targeted grey literature; limited number of outcomes; study design restrictions (Limits vary by topic, timeframe, and extent of available evidence)
Data type	Based on abstracts of best available secondary/tertiary evidence	Based primarily on abstracts with some full text as required	Based on full text
Integration of evidence	Reference list + Key messages	Narrative summary of study characteristic	Narrative summary of study characterises and evidence synthesis
Risk of bias assessment	Not applicable	Not included	Yes (using validated instrument where feasible)
Limitations/ disclaimers	• Not all relevant evidence will have been identified • Assessment based mainly on titles and abstracts • Quality of the listed/included evidence not assessed • Conclusions cannot be drawn	• Possible that not all relevant evidence identified • Quality of the listed/included evidence not assessed • No synthesis of results conducted; provide description of available evidence • Conclusions cannot be drawn	• Possible that not all relevant evidence identified • There may be potential biases in cutting corners (SR methods). (Transparency in reporting and following a methodology will highlight limitations)

Abbreviations: SRs systematic reviews

* The length of time may need to be extended in some instances and will depend on the breadth and complexity of the research topic/question(s), extent of the evince base, and type of analysis required to synthesise the evidence.

Table 2: Comparison of our rapid review methods with recommendations presented in the Best Practice Framework

As part of the methods appraisal, recommendations in the Best Practice Framework were colour coded green: where WCEC methods met or exceed the recommendation; orange: where some WCEC methods met or partially met the recommendation; or red: were the majority of WCEC methods did not meet the recommendation. The outcome of this appraisal is summarised here as text in the last column (N.B. Multiple recommendations were made by each guidance for individual review stages included in the Framework.)

BEST PRACTICE FRAMEWORK DEVELOPED TO INFORM RAPID REVIEWS CONDUCTED BY WALES COVID-19 EVIDENCE CENTRE (WCEC)

COMPARISON OF THE METHODS USED IN WCEC REVIEWS WITH THE BEST PRACTICE FRAMEWORK RECOMMENDATIONS

REVIEW STAGES

Key published sources providing best practice guidance for conducting and reporting rapid reviews

Existing guidance developed by two WCEC Collaborating Partners for conducting rapid reviews

Garritty, 2021 (Cochrane RR guidance) [7]

Tricco, 2017 (WHO) – key considerations [13]

(Production time usually ~1-4 months)

Pluddemann, 2018 (Restricted SRs) [18] - minimum requirements

[Additional steps to reduce bias]

SURE -

PACeRS (Mann, 2019) [11]

(Production time 8-10 weeks)*

HTW [19]-

Evidence Appraisal Report (EAR)

(Production time 3-6 months)**

[OUTCOME CODING]

Comments on the methods used in WCEC Rapid Reviews (RRs) and Rapid Evidence Maps (REMs)

(Production time usually ~2 months)

Question development & refinement

Involve key stakeholders to set and refine the review question, eligibility criteria, and the outcomes of interest; to ensure the research question is fit for purpose; and regarding any ad-hoc changes that may occur as the review progresses

Work with requester to ascertain intended purpose, scope and timeline

Ensure the proposed approach fits the intended purpose

Clearly formulated research question, with rationale for why research is needed

Patient, Public, and Policy Involvement, where resources permit

Rapid Review Request Form template – incorporating PICO/SPICE framework; Question refined with requestor and Review Advisory Group (RAG).

Topic proposer involved in refining Q and identifying independent topic reviewers and other stakeholders. External topic advisor(s) (could be same as topic proposer) recruited to advice evidence review team

[GREEN]

Key stakeholders and subject experts involved in:

- ascertaining intended purpose, and timeline

-setting and refining research question, eligibility criteria

-feedback on review during/after the review process

Preliminary work to inform scope

Preliminary literature search can help inform conversations with requestor and scope

Initial preliminary search for existing SRs conducted

Initial Topic Exploration Report (TER) conducted, which gives an overview of the evidence that exists on a topic and used to decide if there is enough evidence to do an appraisal.

[GREEN]

Rapid Evidence Summary conducted, which incorporates a preliminary literature search to ascertain if evidence is sufficient for RR; identify existing SRs; inform the scope / refinement of the research question; and decide on the methods for the RR (or if a REM is needed)

PROTOCOL development and approval

Develop a protocol that includes review questions using PICOS or other framework and details inclusion and exclusion criteria.

Protocol should be published (e.g. on PROSPERO)

RR producers typically use a PICO format and develop key questions iteratively with requesters

Register protocol with PROSPERO and include “rapid review” or similar term in the title

Use PRISMA reporting items

Basic protocol published on register, e.g. PROSPERO

[Reviewed by research team, potentially peer review publication]

Registered on PROSPERO where appropriate

Use of PROTOCOL template, incorporating PICO framework.

Published on-line; not registered

[GREEN and RED]

Protocol includes review question(s) using PICOS or other framework and details inclusion and exclusion criteria. Protocol is developed with input from stakeholders and made available on request (which is noted in the final report). Protocol is not published on register.

CONCEPTUAL FRAMEWORK

Use a conceptual framework for complex Qs relating to health policy and system improvements

For complex interventions logic model may be used to help define Q and inform review

[AMBER]

Conceptual framework used in a couple of reviews; not required in most reviews.

SEARCH

Involve an information specialists

Main databases to search: MEDLINE, CENTRAL, and Embase; plus (if required) 2 additional specialist database/sources.

Limited grey literature and supplemental searching

Tailor database selection to topic (commonly used: PubMed/MEDLINE, Cochrane Library, and Embase). Add grey literature search depending on topic, purpose, and timeline.

Use staged search to first identify existing SRs, then studies with other designs that provide the most rigorous evidence to answer question

Peer review search strategy, using a tool eg PRESS checklist

At least one major scientific d/b and one other source (e.g topic specific database or google Scholar)

Limit by date and language (acceptable)

Use previous review as starting point (acceptable)

[No date limit

Include unpublished studies, grey literature

No language limit]

Searches conducted by experienced information specialists

Initial search for existing SRs: CDSR, PubMed - Clinical Queries + Health

Main search: 3-4 databases (MEDLINE, CINAHL, Cochrane Library, Embase, HMIC, JBI EBP, PsycINFO), eTOC's of key journals, citation tracking, and grey literature where time available.

Initially search limited to last 5-10 years depending on literature, English language, and studies published in Organisation for Economic Co-operation and Development countries; no study design restriction. Excluding conference abstracts, doctoral dissertations and book chapters.

Search strategy developed on MEDLINE and first 20 hits sent to requestor for checking relevance

Searches conducted by experienced information specialists

Selection of databases and other sources informed by Literature search SOPs; Database checklist

Unpublished literature not considered

[GREEN]

Searches conducted (or informed) by experienced information specialists. Minimum set of databases, tailored to question/topic area, limited grey literature searched. (Some reviews incorporate more extensive grey literature searches, depending on topic, purpose, and timeline’ and informed by the stakeholders)

STUDY SELECTION

Screening title and abstract - two reviewers to dual screen at least 20% of citations, resolving all conflicts. One reviewer to screen remaining citations and one to review all excluded citations, resolving all conflicts if needed

Full text - One reviewer to screen all manuscripts and one to review all excluded manuscripts

Use standardised forms; pilot for calibration and test use across whole team. Consider using SR software

In lieu of dual screening and selection, use single experienced reviewer for application of inclusion criteria and two reviewers for application of exclusion criteria, or one person for screening with verification of a subset of records by another

Single reviewer of titles, abstracts, and full text (sufficient)

Verification of a random sample of full texts by single reviewer (acceptable)

[Verification of a random sample of full texts by a second reviewer

All full texts by 2 reviewers

Titles, abstracts and all full texts by 2 reviewers]

Screening title and abstract – two independent reviewers (Mann, 2019)

Full text - study selection carried out by one reviewer and checked for accuracy by another.

[AMBER]

Single reviewer screening in most cases – sometimes with ‘add ons’ e.g. internal quality assurance, verification of a sample or excluded citations/manuscripts. Some reviews include dual (independent) citation and full text screening,

DATA EXTRACTION

Single reviewer to extract data (using piloted form), with second reviewer checking for correctness and completeness. Consider splitting data extraction into two parts: i) study characteristics, extracted by single reviewer; and ii) outcome data extracted by 2 independent reviewers.

Limit extraction to a minimal list of required items.

Consider using data from existing reviews to expedite data extraction

Use single reviewer to extract data, with second reviewer checking ≥10% random sample for accuracy. Dual performance or checking may be needed more for quantitative results than descriptive study information.

Limit extraction to key study characteristics and outcomes

Single reviewer (sufficient)

Verification of data extraction of a random sample by a single reviewer (acceptable)

[Verification of data extraction of a random sample by a second reviewer

All data extraction verified by a second reviewer

Seeking additional data sources]

Data extraction carried out by one reviewer and checked for accuracy by another.

(data extraction forms based on PICO framework)

[AMBER]

Single reviewer data extraction with quality assurance/verification of a sample. All data extraction checked in some.

CRITICAL ASSESSMENT OF INCLUDED STUDIES / RISK-OF-BIAS

Use a valid risk of bias (RoB) tool specific to the study design(s)

Single reviewer to rate RoB, with full verification of all judgments by a second reviewer.

Limit RoB ratings to the most important outcomes

Choice of appraisal instrument varies in practice, with both standard and customized approaches used

An approach similar to that for data extraction can be used (i.e. single reviewer, with verification by a second reviewer)

Single reviewer (sufficient)

Rapid appraisal tools (acceptable)

[Partial verification of the risk of bias assessment by a second reviewer. All risk of bias assessment verified by a second reviewer

Detailed appraisal tools]

Quality assessment using amended version of GATE checklist (Mann, 2019) incorporating both internal and external validity; different checklists for quantitative and qualitative studies (or range of SURE critical appraisal checklists for different study design).

Each study assessed by one reviewer and checked for accuracy by another.

No formal quality appraisal process, but the EARs include an informal assessment of evidence quality, taking into account generalisability, applicability, sources of bias and any other relevant strengths or limitations. Existing checklists may be used to prompt items for consideration and inform author’s conclusion

[GREEN and AMBER]

Quality assessment tools specific to the study designs used in most cases and results synthesised in a narrative; individual study appraisal findings documented as an appendix or available on request.

Critical appraisal generally conducted by a single reviewer, with findings often verified by a second for a sample or all.

SYNTHESIS

Synthesize evidence narratively Consider meta-analysis only if appropriate (i.e., studies are similar enough to pool) using appropriate standards

An iterative approach to the synthesis process can involve post hoc protocol adjustments

Narrative synthesis (sufficient)

Summary tables (required)

[Meta-analysis of all outcomes]

[GREEN]

Narrative synthesis only. This was sometimes limited to a descriptive summary of studies and their results, rather than a full synthesis. Summary tables provided.

ASSESSMENT OF BODY OF EVIDENCE

Single reviewer to grade the certainty of evidence, with verification of all by a second reviewer

The quality of the body of evidence and the strength of any recommendations can be assessed using an approach such as the GRADE system

Limitations of the review should be discussed and cautious conclusions provided

Reported key findings includes documenting: reliability, consistency, and relevance of evidence

[AMBER]

Variable use of GRADE to assess certainty of the evidence; overall body of evidence often assessed in a narrative without using a tool

REPORT PRODUCTIONS

Software tools can help automate and track review steps

Standardisation of processes and templates aids in production of the report and enhances transparency of the review.

Final reports often include implications, recommendations for policy, and discussion of research limitations

Basic publication without appendices and added data (sufficient)

[Publication without appendices and added data]

Report format can vary greatly from generation of a reference list through to detailed appraisal.

Format & template informed by initial workshop and approved by end-users, incorporating:

1) methods + context; 2) key findings split into: reliability, consistency, and relevance of evidence; 3) policy and clinical implications; 4) PRISMA flow diagram; 5) table of study summaries; 6) list of included studies

Draft report reviewed by RAG; revised and sent to requestor for response; queries addressed liaising with RAG; second draft submitted to requestor; final review document developed liaising with requestor

Reporting template used

EARs do not make recommendations for NHS Wales, but HTW’s Appraisal Panel may issue Guidance based on the report’s findings

External peer review conducted

[GREEN]

Reporting template used. Reports include implications, recommendations for policy, and research limitations. Appendices used where needed.

Dissemination / knowledge translation

Peer review journal publication infrequent

Reviews added to the PaCERS RR repository (available online), and distributed to decision-makers via e-mail

All Topic Exploration Reports and Evidence Appraisal Reports available online

Sometimes published as peer review journal publications

[GREEN]

Published online and preprint servers (e.g. medrxiv), and disseminated via comms/ stakeholders/ briefings/ symposiums

Abbreviations: CDSR Cochrane Reviews. Search Reviews; EAR Evidence Appraisal Report; GATE Graphic Appraisal Tool for Epidemiological studies; GRADE Grading of Recommendations, Assessment, Development and Evaluations; HTW Health technology Wales; NHS National Health Service; PaCERS Palliative Care Evidence Review Service; PICO Participants Intervention comparator Outcomes; PICOS Participants Intervention comparator Outcomes Study design; PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses; Q question; RAG Review Advisory Group; RoB risk of bias; REM Rapid Evidnce Map; RR rapid review; SOPs Standard Operating Procedures; SPICE Setting Population or Perspective Intervention Comparison Evaluation; SR Systematic reviews; SURE Specialist Unit for Review Evidence; TER Topic Exploration Report; WCEC Wales COVID-19 Evidence Centre; WHO World Health Organisation.

Footnotes:

* Palliative Care Evidence Review Service (PaCERS) use rapid review, which are defined as ‘a review conducted within 8-10 weeks using modified systematic review methods with a highly refined research question, search carried out within limited set of databases and other sources and increasing the transparency of the methods used’.

** Health Technology Wales (HTW) Evidence Appraisal Reports (EARs) are not comprehensive systematic reviews or full technology assessments. They are based on the best evidence that Health Technology Wales can identify and retrieve within the time available using rapid review methodology.

Table 3: Comparison of rapid review methods with range of recommended shortcuts in scoping review of published guidance

Development step*

Range of recommendations for methodological shortcuts in published guidance (most common in bold type, but not necessarily the optimal approach)

No of guidance (n=38)**

Approach (and range of methodological shortcuts) used in WCEC RRs***

Search strategy

• Specific search strategy to address clearly defined question

• Sensitive search strategy

• Individual adapted search strategy

21

5

3

Specific search strategy developed to address a clearly defined question (with pre-defined eligibility criteria specified in a protocol)

Databases searched

• 3 or more databases

• 1 - 2 databases

• Search in limited number of databases

• Preliminary search to identify databases

• Individual choice of databases

12

7

5

2

Searches included >3 databases

Search timeframe

• Search timeframe limited to 10 years

• Search timeframe limited to 5 years

• Search timeframe limited to 3 years

• Customise search timeframe according to research question

• No restriction

3

4

1

10

3

Search timeframe customised according to research question and selected in consultation with stakeholders. (The process is supported by the findings of the preliminary RES***)

Languages

• Limited to English and one further language

• Limited to articles in English language

• Individual restriction

2

20

5

Most limited to English language, but some also included one or more further languages

Search of grey literature

• Inclusion of grey literature

• No grey literature search

• Customised approach

12

8

7

Customised approach used, with grey literature search added depending on topic, purpose, and timeline (decision informed by stakeholders and preliminary RES***); can be extensive in some reviews.

Study design

• Search without restriction of study design

• Prioritising search for systematic reviews, meta-analyses, HTA reports and guidelines (expanding to primary studies if non identified)

• Exclusive search for systematic reviews (SRs) and RCTs published after the most recent systematic review

• Individual choice of study design to include

1

25

6

1

Priority given to locating and summarising evidence from relevant and/or high-quality SRs (or RRs, clinical guidelines, meta-analyses, HTA reports). However, based on the findings of the preliminary RES*** the subsequent RR could focus on SRs only, SRs plus recent primary studies, or exclusively on primary research (with either unrestricted or individual choice of study design).

Inclusion of easily obtainable literature

• Only easily obtainable literature is included

• All available literature is included

• Customised approach depending on availability of evidence

12

4

2

All available literature included in most reviews, with existing reviews used where feasible.

Full-text analysis

• Full-text analysis

• Analysis on abstract level

• Customised approach

30

2

All RRs based on full-text analysis. The REMs generally based on abstracts (with some full text assessment), depending on the requirements of the stakeholders, time frame and volume of research. (Preliminary RES* is based on titles and abstracts)

Screening: number of reviewers

• Screening carried out by two persons and uncertainties clarified through discussion

• Screening is carried out by one person and 20%–25% of articles reviewed by a second person. If the match is <95%, all articles screened independently by a second person

• Screening carried out by one person and second person screens excluded articles (in some cases only for title and abstract screening, while the full text is screened by two people)

• Screening carried out by one person and a second person consulted in case of uncertainties

• Customised approach

8

5

3

13

1

Range of shortcuts used for study selection, depending on the review question, the volume and type of evidence, and proposed timeframe:

· Screening (citations and full text) carried out by one person

· Screening carried out by one person with sample screened by a second

· Screening carried out by one person with uncertainties or sample of full text decisions checked

· Screening carried out by two (or more) persons with disagreements discussed

· Citation screening carried out by one person and full text by two persons with disagreements resolved by third

· Citation screening by one person with 20% reviewed by a two people; full text screened by one person with excludes checked by a second

· Citation screening by one person with 20% reviewed by two people and ≥50% of excludes checked by second person; full text screened by two independent people

Extraction: number of reviewers

• Extraction carried out by one person and all results verified by a second person

• Extraction carried out by one person and results partially verified by a second person

• Extraction carried out by one person

• Customized approach

6

7

14

2

Range of shortcuts used for data extraction depending on the review question, the volume and type of evidence, and proposed timeframe:

· Extraction carried out by one person

· Extraction carried out by one person with sample checked by a second

· Extraction carried out by one person with 10% double extracted for discrepancies

· Extraction carried out by one person with all checked by a second

· Extraction carried out by two independent persons

Risk of bias assessment

• Risk of bias assessment

• No risk of bias assessment

• No independent assessment of the risk of bias but bias potential reported in the included evidence is incorporated

• Customized approach depending on availability of evidence

20

7

5

3

Risk of bias assessment conducted with validated or recognised risk of bias or critical appraisal tools in most reviews. No quality assessment conducted in a few RRs, but narrative summary of key limitations provided.

Abbreviations: HTA Health Technology Appraisal; RCT Randomised controlled trial; RES Rapid Evidence Summary; RR Rapid review; SR Systematic review.

Reference (Scoping review): Speckemeier C, Niemann A, Wasem J, Buchberger B, Neusser S. (2022) Methodological guidance for rapid reviews in healthcare: A scoping review. Res Synth Methods, 13(4): 394-404. doi:10.1002/jrsm.1555

Footnotes:

* This table is based on ‘Table 1: Recommendations for methodological short cuts’ reported in the manuscript by Speckemeier et al (2022). The following steps for developing a review, which are incorporated in our Best Practice Framework and appraisal of our methods, were not included in the table by Speckemeier et al (2022): risk of bias assessment - number of reviewers, synthesis, and assessment of body of evidence.

** Not all guidance documents reported on each development step

*** The methods used within each of our Rapid Reviews and Rapid Evidence Maps are presented in the Supplementary information, Additional file 2. The methods used within individual reviews will vary, depending on the review question, the volume and type of evidence, and proposed timeframe. The approach used are informed by a prior scoping review of the literate (presented as a Rapid Evidence Summary, RES) and developed in collaboration with the stakeholders.

The authors declare no competing interests.

AdditionalFile1Resourcelist.pdf
Additional File 1 - Resource list
AdditionalFile2MethodsusedinWCECReviews.pdf
Additional File 2 - Methods used in WCEC Reviews

A bespoke rapid evidence review process engaging stakeholders for supporting evolving and time-sensitive policy and clinical decision-making: reflection and lessons learned from the Wales Covid-19 Evidence Centre 2021-23

Status:

Version 1

Abstract

Figures

BACKGROUND

METHODS

The Wales COVID-19 Evidence Centre (WCEC)

Development of the WCEC evidence review process

Best Practice Framework

RESULTS

DISCUSSION

CONCLUSIONS

Abbreviations

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1