We adopted the Joanna Briggs Institute (JBI) methodology for umbrella reviews [18], and we used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses-Protocol (PRISMA-P) guidelines to structure this protocol (Additional file 1) [19]. The study protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO, CRD42020183774).
Research question
What is the evidence about scaling in health and social care and what are the knowledge gaps in the literature?
Eligibility criteria
We will address all types of evidence matching the “PICO” criteria (Participants, Intervention, Comparator, Outcome) and “PICo” (Population, phenomena of Interest and Context) to capture the evidence from quantitative and qualitative reviews.
Participants or population
All reviews that include primary studies focusing on individuals (e.g., patients or caregivers, health care providers), or health systems, services or organizations that have been exposed to the scaling of a health or social care innovation.
There will be no restrictions based on socio-demographic factors (e.g., age, ethnicity, socio-economic status) or general health conditions (e.g., comorbidities).
Interventions/phenomena of interest
No restrictions. We will consider all types of scaling, vertical and horizontal. This umbrella review will not only explore the scaling of health and social care innovations, but also any aspect of or any topic relating to such scaling, including (but not limited to) concepts, models, analytical models, tools, cost or impact assessments and user engagement.
Comparator
No restrictions.
Outcomes
We will consider all outcomes reported in the included reviews (i.e., no restrictions), including health outcomes. We will seek outcomes such as (but not limited to) patients/caregivers and/or health care providers and/or policy makers’ perceptions and experiences of barriers, facilitators, acceptability and feasibility of scaling innovations, impact (e.g., adaptability, efficacy, effectiveness), coverage (e.g., proportion of the target population that is reached by the scaling, adoption, fidelity, penetration, maintenance of the innovation), health outcomes (e.g., impact on morbidity, mortality), patient reported outcomes (e.g., quality of life, satisfaction), and health care resources (e.g., cost-effectiveness of the scaled innovation, cost of staff resources).
Types of study
We will consider all types of review (quantitative, qualitative, and mixed-methods) that address, synthesize and summarize pre-existing evidence in the field of scaling. We define a review as a knowledge synthesis of evidence that includes a clear research question, describes the methods used (which are reproducible) to identify and select the primary research studies, and synthesizes data from its included studies [20, 21]. Reviews can include studies with any research design. We will include all types of reviews that have been rigorously conducted according to their chosen methodological approach. We will exclude reviews that do not describe their search strategy and inclusion criteria explicitly at the stage of full-text screening. We will also exclude primary research studies, conference abstracts, comments, opinions, letters and editorials.
Context/setting
We will include reviews regarding any type of health and social care setting (e.g., home care, community, hospital, primary care, specialized care) in any geographical setting (e.g., rural or urban regions, low-, middle- or high-income countries).
We define “health and social care services” as follows:
Health care consists of services provided in institutional or community settings, any form of access to a health-related service (such as dental, podiatry or optical services) and access to health care practitioners (such as nurses, physiotherapists or general practitioners) [22].
Social care consists of interventions that support frail or vulnerable individuals by meeting needs or enabling them to meet needs that arise as a result of physical, mental or emotional impairment [23].
Search strategy
Our information specialist (NR) will develop an Ovid-MEDLINE and Web of Science strategy with input from the project team and a second information specialist. An iterative revision process will be conducted by the members of the research team. Research keywords will include: “scaling”, “reviews” and “health and social care”. We use the concept of scaling up previously developed by members of our team. The Canadian Agency for Drugs and Technologies in Health (CADTH) search filter for Systematic Reviews/Meta-Analysis/Health Technology Assessment was adapted for this project [24]. A second information specialist will review the search strategy using the Peer Review of Electronic Search Strategies (PRESS) tool [25]. Comments will be integrated in a final version of the search strategy. The final version will be approved by the team members. Once approved, this search strategy will be translated into the other databases mention below. The exact search terms will be recorded as the search strategy is refined. This protocol only includes the search strategy conducted in one database (Additional file 2). A systematic literature search will then be performed to identify published studies in the following electronic bibliographic databases: MEDLINE (Ovid), Embase, PsychINFO (Ovid), CINAHL (EBSCO), Web of Science, The Cochrane Library, Sociological Abstract (Proquest), Academic Search Premier (EBSCO), and Proquest Dissertations & Theses Global. No language restriction will be applied. We will search from inception onwards.
Additionally, to identify the grey literature, we will search websites of relevant organizations such as the World Health Organization (WHO), Global Reporting Initiative, the UK’s National Institute for Care and Excellence (NICE), Australia NSW Government, the Institut National d’Excellence en Santé et Services Sociaux (INESSS) in Quebec, Canada, the Canadian Foundation for Healthcare Improvement (CFHI), Canada’s IDRC, the CADTH’s Grey Matters checklist, and clinical trial registries. We will contact experts in the field by email for additional data. In addition to searching databases, the reference list of each included review will be reviewed.
Selection process
The search results will be imported and stored in an EndNote X9 library for reference management and duplicate removal [26, 27]. The resulting records will be exported to the Internet-based system Covidence for the selection process [28]. The team will develop a selection grid. It will be adjusted, if necessary, before the screening of titles and abstracts of all articles. All selection criteria will be discussed between reviewers to ensure common understanding. A pilot screening (titles and abstracts) of 2.5% random sample reviews will be completed. Discrepancies will be resolved by consensus or by a third reviewer if necessary.
First, two independent reviewers will screen titles and abstracts of identified reviews against the eligibility criteria. Articles with abstracts that do not appear to meet the criteria for exclusion, or are ambiguous, or have missing abstracts, will be retained and reviewed in full. Second, full-text examination of the remaining reviews will be assessed for eligibility. The reviewers’ full understanding of the selection criteria will be validated again before beginning this stage. In case of an “unclear response” regarding the eligibility of studies, authors will resolve disagreements through discussion, and if necessary, consult a third senior author. Any reasons for exclusion will be recorded in Excel.
Articles that are not available electronically will be ordered via interlibrary loan. We will contact the corresponding author if an email is available.
Data extraction
Once the reviews have been selected for inclusion, two reviewers will independently conduct the data extraction. We will develop a data extraction form based on the JBI form for review of systematic reviews. A standardized, pre-piloted form will be used to extract data from the included reviews for assessment of study quality and evidence synthesis. A pilot data extraction will be conducted on 5% of selected reviews until conclusive results are reached between reviewers. Any disagreement between the reviewers will be resolved through discussion; if consensus is not reached, they will consult a third senior reviewer. We will not extract data from the primary studies included in the reviews. We will summarize the review findings but we will not re-synthesize the results of primary studies. Extracted information will include:
-
Review characteristics: author, year of publication, country, type of review; aim of the study; language; PICO or PICo; number and characteristics of participants (sex as a biological variable if reported in the included reviews, age); setting and context; number and type of databases sourced and searched; date range of database searching; citation index; number and study design of primary studies included in each review; instrument used to appraise the primary studies and the rating of their quality; method of synthesis/analysis employed to synthesize the evidence; synthesis/summary of results; heterogeneity if applicable; mention by the authors of research needs or gaps in the review.
-
Focus of the reviews: infrastructure (e.g., policies, guidelines, human and material resources); concepts and models (e.g., definition, conceptual model, framework); measurement (e.g., tools, scalability, cost); analytical methods (e.g., mathematical approach); descriptive (e.g., perceptions, barriers, facilitators, acceptability); patient engagement (e.g., authorship, study’s reporting on patient advocates) and scaled interventions (e.g., strategies, training, process, coverage, feasibility, effectiveness).
Assessing quality of reviews
Two independent reviewers will use AMSTAR2 [29] to assess the methodological quality of included reviews and ROBIS [30] for the risk of bias. Based on the nature of the included reviews, we will explore the relevance of using the GRADE assessment system [31, 32]. Disagreements between the reviewers over the quality or the risk of bias in particular studies will be resolved by discussion, with involvement of a third author where necessary. The AMSTAR2 is a validated, widely used instrument for assessing reviews of reviews. It consists of a set of 16 questions answered with the following options: yes, no, partial yes. ROBIS contains 24 questions divided into three phases. Phase 1 assessing relevance (optional), verify if the research question match the umbrella’s PICO; phase 2 identify concerns with the review process, which is performed by four domains; and phase 3 judge risk of bias in the review. Phase 2 and 3 questions answered with the following options: yes, probably yes, probably no, no, no information. The concerns regarding phase 2 domains and phase 3 are classified as high, low or unclear.
Data synthesis
We will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram to describe the process of study selection [33]. We will use tables to describe characteristics of included studies (e.g., first author’s name, year of publication, study aim), analysis details (e.g., appraisal instrument used, methods of analysis, results summary, authors comments, heterogeneity of the results) and we will synthesize the data narratively using categories of scaling evidence that have emerged from the included reviews (e.g., descriptive reviews of scaling, reviews of barriers and facilitators, reviews of determinants, reviews of concepts, reviews of frameworks, reviews of economic evaluation, reviews of scaling interventions, reviews of scaling measures, reviews of effects of scaling interventions, reviews of scalability).
A meta-analysis will not be performed, given that pooling the results of qualitative and quantitative reviews can introduce significant overlap and bias. We will clearly indicate the overlap between primary studies within the reviews in the tables and we will develop a citation matrix. Because our purpose is to identify evidence about scaling in health and social care in the literature, we will present the whole body of knowledge and include the results of all the reviews, regardless of the overlap across primary studies [34].
Living update
Based on the series Living Systematic Reviews [17, 35-38], the search for new reviews will be repeated at regular intervals by the coordinator team, i.e., tri-monthly searches of bibliographic databases and every six months for the grey literature. The studies identified during these updated searches will be analyzed using the same eligibility criteria as used in the initial search. The new information will be integrated at least once a year, when it will be combined with all the “stand by” information. If the information has an effect on the evidence, it will be included in the review and an update will be published.