Topic overlap and research waste at the ‘Overviews of systematic reviews’ level: a meta-research study

doi:10.21203/rs.3.rs-665009/v2

Background

Multiple ‘overviews of reviews’ conducted on the same topic (“overlapping overviews”) represent a waste of research resources, and can confuse clinicians who are required to choose among competing treatments. We aimed to assess the frequency and characteristics of overlapping overviews.

Methods

MEDLINE, Epistemonikos and Cochrane databases were searched for overviews that: synthesised reviews of health interventions and conducted systematic searches. Overlap in topic was defined as: duplication of PICO elements, not representing an update of a previous overview, and not a replication. We also categorized the overviews as broad or narrow in scope.

Results

Of 541 overviews identified (2000–2018), 172 (32%) overlapped across similar PICO. The overlapping overviews fell within 13 WHO ICD-10 medical classifications and 63 topics. The overviews may have overlapped partially or completely, such that a similar portion, major component(s), or complete representation of an overview was duplicated. 149/172 (87%) overlapping overviews were characterized as broad in scope. Most frequently, broad overviews had targeted populations for which multiple interventions were addressed (44%), or least frequently, they had a targeted intervention for multiple populations (17%).

Conclusions

One third of overviews overlapped in content with a majority covering broad topic areas, and fewer considering subsets of the evidence. A multiplicity of overviews on the same topic adds to the ongoing waste of research resources, time and effort across medical disciplines. This study and the database of 172 overlapping overviews can provide a guide to authors about which topics are covered, and gaps in the evidence for future analysis.

Statistical Epidemiology

overviews of reviews

overview of systematic reviews

umbrella reviews

meta-reviews

‘reviews of reviews’

overlap

duplication

redundancy

waste

High-quality ‘overviews of reviews’ and systematic reviews give the best perspective of our current state of evidence on a subject. ‘Overviews of systematic reviews’ (henceforth called overviews) synthetize the results of multiple systematic reviews and help inform evidence-based clinical practice. Overviews, also called umbrella reviews, meta-reviews, or reviews of reviews, are one of the multiple types of evidence syntheses. They are growing in number and popularity, and our bibliometric study of the prevalence of overviews found an 8-fold increase in the number of overviews published between 2009 (n=25) and 2020 (n=332) [1]. The growth in overviews is unlikely to decrease [1].

Several author teams have expressed concerns about the volume of systematic reviews published which are overlapping in content [2-4]. Conflicting results or conclusions across systematic reviews on the same topic can confuse or create uncertainty for policymakers and clinicians who are required to choose among all competing treatments, and may impact and delay clinical decision-making [5]. These concerns can also be extrapolated to overviews reporting discordant findings. Another downside to the publication of multiple overviews on the same topic is that the efforts of the investigators, journal editors and peer reviewers may be unnecessarily duplicated.

Overviews have the potential for overlap because many are broad in scope, thus covering several individual topics. These broad overviews often address less specific research questions than their constituent systematic reviews, including a wider range of study populations and conditions, interventions, and contexts. Overviews that are broad in scope allow for policy relevance; for example, overviews have informed clinical practice guidelines (e.g. [6]) and government health policies (e.g. Australian Government Department of Health, 2015 [7]). Broad overviews can make a large volume of evidence accessible to clinicians and policymakers [8], but may necessitate extensive screening, data extraction, and the synthesis of a large number of systematic reviews.

Overviews may alternatively aim to answer narrow, focused clinical questions, and to identify and explore reasons for variation in the results, interpretation, or conclusions of systematic review analyses [9-18]. As an example, an overview that compared surgical versus conservative treatments for clavicle fractures aimed to determine which systematic review provided the most trustworthy evidence for treatment, and explored reasons for differences in review-level results [16]. Overviews with narrowly focused research questions can be completed more quickly as compared with broad research questions, but may have limited generalizability to different populations and settings [19].

Research is needed to establish if and how overviews overlap in content. For example, overviews may be performed on unique topics which only partially overlap; they may represent updates of previous overviews done by the same team of authors (similar to updated Cochrane reviews); replications; or may be redundant/salami publications on the same topic. Redundant/salami publications are studies that report identical or very similar data in two or more papers without explicitly stating that the data are recycled [20].

This paper is the second of two companion papers. The first paper evaluated the bibliometric characteristics of overviews, and factors affecting citation rates and journal impact factors [21]. In this second paper, we aimed to assess the prevalence and characteristics of overviews published on the same topic, henceforth referred to as overlapping overviews.

2.1. Study design

This is a meta-research study, which aims to evaluate research practices [22]. We followed systematic review guidance for the searching, study selection, and data extraction stages of our study [23].

2.2 Eligibility criteria

As described in our first companion paper [21], we performed a bibliometrics study of overviews published in MEDLINE (Ovid), Epistemonikos, and the Cochrane Database of Systematic Reviews. We included overviews meeting the following selection criteria:

• Synthesises systematic reviews with or without meta-analyses (but the overview may also include primary studies) as a primary focus.

• Searches the literature systematically, and with a search strategy section found in the main body of the paper (i.e. search strategy includes text words and MeSH terms in at least two databases).

• Methods section located in the main manuscript or within a supplementary file (not just in the abstract).

• Focuses on the effects of health interventions or clinical treatments.

We excluded overviews that based their results exclusively on primary studies and methodological studies. Reports that were editorials, letters, or comments were excluded. Overviews of risk, exposure, prevention, measurement instruments, quality indicators, diagnostic, screening or prognostic research were also excluded. We excluded protocols of overviews.

We included overviews published in any language and published from January 1, 2000 to December 31, 2018. Given that the Cochrane Handbook chapter on overviews was first published in 2009, we did not expect to identify overviews published prior to 2000. Reports were translated by one of the authors (French, Spanish, German, Mandarin), when needed.

We reasoned that overviews may exist as a stand-alone report or also packaged as part of a clinical practice guideline and health technology assessment; accordingly, we developed eligibility criteria for both circumstances.

Inclusion criteria for clinical practice guidelines and health technology assessments (HTAs):

Clinical practice guidelines or HTAs aim to primarily include, synthesise and present the results of the systematic reviews; but may also include additional primary studies.

Overlapping overviews eligibility criteria

We determined if two or more overviews were overlapping in topic if they investigated the same population(s) or condition(s), intervention(s), and at least one outcome (PICO). We did not consider an update by the same authors, or if the authors stated the overview was a replication, as an overlapping overview.

2.3 Search

Overviews were retrieved using a validated search filter [24] from MEDLINE (Ovid), Epistemonikos and the Cochrane Database of Systematic Reviews (Appendix A). The Epistemonikos search was limited to the “Broad Syntheses” category, which includes overviews of systematic reviews, HTAs and clinical practice guidelines.

2.4 Overview screening and study selection

The initial search results were imported into Excel 2010 for screening. A pilot screening of the first 19 papers was conducted in duplicate by all screeners to ensure high levels of agreement and common definitions of coding. We screened the titles and abstracts against the stated eligibility criteria first, then eligible full-text articles were reviewed for inclusion. Two independent reviewers screened reports at the title and abstract, and then again at the full text stage, then compared their results. Discrepancies were resolved by consensus, and arbitration by a third reviewer when necessary.

In two previously published studies categorising all methods used in overviews (2013 and 2016) [24, 25], 187 overviews were screened using identical methods (Appendix A). We therefore included these 187 studies and categorized them as “other sources”.

2.5 Data extraction

We extracted data on overviews published between 2000 and 2018. Data extraction was piloted on 20 studies by all authors independently to identify any missing variables, come to agreement on coding definitions, and refine/reword the items. Discrepancies in the piloting phase were discussed and consensus reached by two authors. Full data extraction was performed independently by one investigator and checked by a second reviewer.

We categorized the medical classification of each overview using the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD), a medical classification list by the World Health Organization (WHO) (https://www.who.int/standards/classifications/classification-of-diseases). The main condition or intervention in the title of the overview was entered into the search function of the WHO ICD-10 site (https://icd.who.int/browse10/2016/en) to determine its classification). For example, the title “Nonpharmacological treatment for behavioural and psychological disturbances in older adults with dementia” was catagorised in the ICD-10 classification “Mental and behavioural disorders” because the intervention was treatment for behavioural disturbance in a dementia population. In addition to ICD-10 classification, the topic of each overview was determined through review of the clinical question, including study population(s), intervention(s), and outcome(s).

For overlapping overviews, we additionally extracted whether an author was involved in more than one overview on the same topic. We hypothesised that if an overview was conducted by the “same team of authors” or had some authors in common, then it might be an update of a previous overview, or might represent two forms of self-plagiarism: (a) redundant (duplicate) publication, or (b) salami slicing or a salami publication [20]. Redundant/duplicate publication can be defined as a reporting identical or very similar data in two or more papers without explicitly stating that the data are recycled [20]. Salami publication is a form of redundant publication where different papers from the same data set are published. We also extracted the journal of publication, number of included systematic reviews, search date, inclusion of meta-analysis, and funding status.

2.6 Classification of overviews as broad or narrow in scope

In a post hoc analysis, we classified overviews as being broad or narrow in scope (Figure 1). We hypothesised that broad overviews were more prevalent than overviews with a narrow scope. We defined a broad overview as addressing (a) more than one distinct population (e.g. individuals with cancer, menopause, and lower back pain) or a generalized population (e.g. humans of all ages) and/or (b) multiple interventions (e.g. aerobic exercise, resistance training) for outcomes of interest. Broad overviews could therefore be further sub-classified as being non-targeted (multiple populations and interventions), as having a targeted population (with multiple interventions) or as having a targeted intervention (for multiple populations). We defined narrow overviews as covering only one population and one intervention or comparison (e.g. aripiprazole for patients with schizophrenia).

The scope of outcomes within overviews was not specifically addressed within the classification system in this study. We consider overviews for which outcomes are the main element of exploration to be an exception to our proposed broad and narrow definitions.

One author coded the overlapping overviews which was then checked by a second reviewer.

2.7 Data analysis

Descriptive analysis using frequencies and percentages were performed for categorical data and median and interquartile range (IQR) for continuous data.

The distribution of total overviews by medical classification was plotted in a bubble chart using Excel. The x axis represents medical classification, y axis the number of overviews pertaining to that medical classification and the size of the bubble (third variable) represents the cumulative number of systematic reviews in all overviews included in that classification. We described the gaps in ICD-10 medical classifications covered by all overviews published between 2000 and 2018.

3.1. Search results

After searching MEDLINE, Cochrane, Epistemonikos and other sources, we retrieved 10,145 records (Appendix B Figure 1 flowchart). After removal of duplicates, 8220 records remained, 6733 were excluded at the title/abstract stage, and 946 were excluded at the full text stage. A total of 541 overviews published between 2000 and 2018 were included (Appendix C). Many of the citations were excluded because they did not have a methods section, did not conduct a systematic search, and did not search for and include systematic reviews (Appendix B Figure 1 flowchart). For example, of the 873 citations that might have been included as overviews (as they searched for, and included systematic reviews/meta-analyses, guidelines or HTAs), 122 (14%) did not contain a systematic search strategy.

3.2 WHO ICD-10 Medical Classifications

The 541 overviews covered 20 of the 22 WHO ICD-10 Medical Classifications (Figure 2). The most frequent ICD-10 classification for retrieved overviews (92/541 [17%]) was “factors influencing health status and contact with health services.” Another 62/541 (11.5%) focused on diseases of the musculoskeletal system and connective tissue, 56/541 (10.4%) were about mental and behavioural disorders, 42/541 (7.8%) were on diseases of the circulatory system, and 34/541 (6.3%) were focused on neoplasms (Figure 2). A little under half of the overviews focused on 15 other ICD-10 medical classification (Table 1).

Table 1: Frequency of WHO ICD-10 Medical Classifications in 541 overviews

WHO ICD-10 Medical Classifications	Frequency of overviews	Percent (%)
Blood and blood-forming organs	3	0.55
Certain conditions originating in the perinatal period	7	1.29
Certain infectious and parasitic diseases	22	4.07
Codes for special purposes	12	2.22
Congenital malformations, deformations and chromosomal abnormalities	0	0
Diseases of the circulatory system	42	7.76
Diseases of the digestive system	24	4.44
Diseases of the ear and mastoid process	0	0
Diseases of the eye and adnexa	5	0.92
Diseases of the genitourinary system	26	4.81
Diseases of the musculoskeletal system and connective tissue	62	11.46
Diseases of the nervous system	27	4.99
Diseases of the respiratory system	27	4.99
Diseases of the skin and subcutaneous tissue	7	1.29
Endocrine, nutritional and metabolic diseases	19	3.51
External causes of morbidity and mortality	23	4.25
Factors influencing health status and contact with health services	92	17.0
Injury, poisoning and certain other consequences of external causes	19	3.51
Mental and behavioural disorders	56	10.35
Neoplasms	34	6.28
Pregnancy, childbirth and the puerperium	9	1.66
Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified	25	4.62
Total	541	100.00

3.3 Gaps in WHO ICD-10 Medical Classifications across 541 overviews

The WHO ICD-10 medical classification has 22 classifications in total. No overviews were found for two WHO ICD-10 Medical Classifications, namely ‘congenital malformations, deformations and chromosomal abnormalities’ and ‘diseases of the ear and mastoid process’ (Table 1).

3.4 Overlapping overviews according to medical classification and topic

Of 541 overviews published from 2000 to 2018, 172 (32%) had at least one other overlapping overview (median 2 overviews per topic, interquartile range 2-3, maximum 7). Of the 172 overlapping overviews, 41 (23%) were published on or before 2010, 63 from 2011 to 2015, and 70 from 2016 to 2019. These numbers show how incidence of overlapping overviews has changed chronologically.

Overlap in ICD-10 Medical Classifications

The 172 overlapping overviews fell under 13/22 (59%) WHO ICD-10 Medical Classifications (Appendix D). The overviews may have overlapped partially or completely, such that a similar portion, major component(s), or complete representation of an overview was duplicated. The greatest number of overlapping overviews were classified under “factors influencing health status and contact with health services” (40/172 [23%]), followed by “musculoskeletal system and connective tissue diseases” (36/172 [21%]), and “symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified” (23/172 [13%]).

Overviews topics [population(s), intervention(s) and outcome(s)]

A total of 63 topics involving a combination of the same population(s), intervention(s) and outcome(s) were covered by two or more of the 172 overlapping overviews (Table 2). Overviews were most often partially overlapping in various combinations of their populations, interventions, and outcomes, rather than reporting on fully identical topics. Many overlapping overviews overlapped for common outcomes such as all-cause mortality, serious adverse events, and adverse events.

The most prevalent topic that overlapped across one overview pertained to acupuncture for pain (seven overviews). The topics of cannabinoids for pain and symptoms, and behavioral counseling and pharmacotherapy interventions for tobacco cessation overlapped across six overviews. The topics of exercise therapy for bone and muscle health, over-the-counter analgesics for pain, and treatment of venous thromboembolism with low-molecular-weight heparin (LMWH) and unfractionated heparin (UFH) overlapped across 5 overviews each. Nine topics were covered by four overviews each, 12 topics were covered by three overviews each, and 37 topics were covered by two pairs of overviews (Table 2).

Table 2: Overviews with overlapping patient(s), intervention(s) and/or outcome(s)

Overlapping patient(s), intervention(s) and/or outcome(s)	Frequency	Percentage by topic (%) (n=63)
Acupuncture for management of pregnancy-related symptoms	2	3.2
Acupuncture for managing gynaecologic conditions	3	4.8
Acupuncture for pain	7	11.1
Acupuncture for palliative care of cancer	3	4.8
Acute asthma management in children	2	3.2
Antipsychotic drugs for schizophrenia	4	6.3
Assisted reproductive technologies (ARTs)	2	3.2
Behavioral Counseling and Pharmacotherapy Interventions for Tobacco Cessation	6	9.5
Cannabinoids for pain, and symptoms	6	9.5
Childhood obesity interventions	2	3.2
Chronic treatment in childhood asthma	2	3.2
Complementary and alternative procedures for fibromyalgia	3	4.8
Cupping	2	3.2
Diets to reduce weight and obesity	3	4.8
Effects of coffee	2	3.2
Effects of financial arrangements for health systems in low-income countries	2	3.2
Exercise therapy	5	7.9
Exercise to relieve fatigue	2	3.2
Exercise to relieve pain	4	6.3
Food supplements for body weight reduction	2	3.2
Ginkgo biloba for dementia	2	3.2
Hip fracture pre-op management and rehabilitation	2	3.2
Influenza vaccination	3	4.8
Interventions for improving patient quality of life	2	3.2
Interventions to treat complex wounds	3	4.8
Knee osteoarthritis - physical therapy	2	3.2
Lung cancer cost effectiveness analysis	2	3.2
Lung cancer treatment	2	3.2
Lupus nephritis treatment	2	3.2
Mammography screening	2	3.2
Management of hip and knee osteoarthritis	2	3.2
Manual therapy for the treatment of migraine	2	3.2
Neuraminidase inhibitors for influenza	2	3.2
Non-pharmacological interventions for insomnia	2	3.2
Non-pharmacological interventions for osteoarthritis	4	6.3
Nonpharmacological treatment for behavioural and psychological disturbances	4	6.3
Non-pharmacological treatment for cancer-related fatigue	2	3.2
Opioid use in non-cancer pain	3	4.8
Over-the-counter analgesics for pain	5	7.9
Periodontal treatment and glycaemic control	2	3.2
Periodontal treatment and preterm birth rate	2	3.2
Pharmacologic treatment of low back pain	3	4.8
Pharmacological interventions for smoking cessation	2	3.2
Pharmacological and non-pharmacological treatments for depression	2	3.2
Pharmacological treatments for major depressive disorder	2	3.2
Pharmacological, non-pharmacological, and surgical treatments of low back disorders	4	6.3
Physical activity promotion in children and adolescents	3	4.8
Physiotherapy exercise	2	3.2
Preterm birth	3	4.8
Psychotherapy and non-medication-based interventions	3	4.8
Physiotherapy and exercise	2	3.2
Reduction interventions of alcohol intake	4	6.3
Rheumatoid arthritis – non-pharmacological interventions	2	3.2
Rheumatoid arthritis - pharmacological interventions	2	3.2
Rotator cuff repair surgery rehabilitation	2	3.2
Spinal manipulation	4	6.3
Surgical treatment of low back pain	4	6.3
Therapy for fibromyalgia syndrome	2	3.2
Treatment for multiple myeloma	2	3.2
Treatment of venous thromboembolism with LMWH and UFH	5	7.9
Urinary incontinence	3	4.8
Vitamin D supplementation	4	6.3
Web-based interventions for weight loss	2	3.2

Appendix D lists the 172 overviews that overlapped for the 63 topics with their WHO ICD-10 medical classification, population, interventions, and outcomes.

Classification of overlapping overviews as broad or narrow in scope

Our hypothesis that broad overviews were more prevalent than overviews with a narrow scope was confirmed, as 149/172 (87%) overlapping overviews were characterized as broad in scope (Appendix D). Most frequently, broad overviews had targeted populations for which multiple interventions were addressed [66/149 (44%)], and least frequently broad overviews addressed a targeted intervention for multiple populations [26/149 (17%)]. Broad overviews categorised as non-targeted (56/149) accounted for 38% of broad overviews.

Some broad overviews covering multiple patient populations overlapped with an overview investigating just a portion of these populations. This situation is exemplified by the overlapping reviews by Ernst [26], “Complementary and alternative medicine for rheumatoid arthritis and osteoarthritis: an overview of systematic reviews” and Moe [27], “Effectiveness of nonpharmacological and nonsurgical interventions for hip osteoarthritis: an umbrella review of high-quality systematic reviews”. Both reviews were broad in scope and addressed various therapies including herbal medicines for pain and various other outcomes. The targeted population covered by Moe [27], namely individuals with osteoarthritis of the hip, was covered by the more comprehensive non-targeted populations of patients with osteoarthritis and rheumatoid arthritis covered by Ernst [26].

Sometimes overviews investigating multiple interventions overlapped with an overview covering just a subset of these interventions. An example of this type of overlap happened with the narrow overview by Mac-Namara [28], “Is rituximab effective for induction of remission in lupus nephritis?” and Chen’s [29] broad overview entitled “Treatment for lupus nephritis: an overview of systematic reviews and meta-analyses.” Both overviews investigated pharmacologic treatments in patients with lupus nephritis at various stages. Mac-Namara’s [28] overview targeted any medication regimen involving rituximab, whereas Chen’s [29] non-targeted overview broadly encompassed all interventions and involved more diverse regimens.

We also found that some broad overviews overlapped with the full focus of narrow overviews. For example, a broad overview by De Niet [30] entitled “Review of systematic reviews about the efficacy of non-pharmacological interventions to improve sleep quality in insomnia” included reviews investigating a number of interventions (e.g. cognitive and behavioural interventions, sleep-wake rhythm control, acupuncture, aromatherapy, and more) for sleep quality and duration outcomes for adults with various forms of insomnia. A narrow overview by Ernst [31] “Acupuncture for insomnia? An overview of systematic reviews” focused solely on acupuncture for a similar insomnia population and outcomes.

Narrow overviews we considered to have fully or nearly fully overlapping topics made up a small proportion of our sample [32, 33] and [34, 35]. Hasuike [33] and Botero [32] overviews overlapped quite considerably: patients with Type 1 or Type 2 DM with peridontitis, receiving periodontal treatment, with a primary outcome of impact on hemoglobin A1c (HbA1c). However, Botero [32] additionally includes a secondary outcome of fasting blood glucose, which Hasuike [33] does not. Hasuike [33] was published after Botero [32] but the dates of searches were comparable. Similarly, Lopez [34] and Vivares-Builes [35] overlapped almost identically except for one extra outcome in overviews on pregnant women receiving peridontal treatment for pre-term birth.

Figure 3 visually represents three overlapping overviews from our sample [36-38]. Overview A addressed any type of acupuncture therapy for cancer patients receiving palliative care. Overview B addressed a narrower intervention, acupuncture excluding any acupressure therapy, for a slightly broader population of cancer patients receiving palliative or supportive care. Overview C had an even narrower targeted intervention of acupuncture on point P6, and investigated multiple populations including cancer patients with chemotherapy-induced nausea and vomiting, post-operative nausea and vomiting, and pregnancy-related nausea and vomiting. Nausea and vomiting were outcomes common to all three overviews.

Authors who publish overviews on similar topics

In 11/63 (17%) topics, at least one author was involved in two or more of the overviews that overlapped. These redundant/duplicated overviews had partially overlapping participants, interventions, outcomes, and indications/settings. Of the two overviews on spinal manipulations [39, 40], the 2011 overview was written by two authors, and a year later, one of these authors published an overview with the same populations, intervention, and outcomes. The authors had not referenced their earlier 2011 paper, perhaps seeking to hide the redundancy.

Summary of results

This is the first study to examine overlap across a cohort of overviews. Our methodological assessment identified 32% of overviews dated between 2000 and 2018 that overlapped in content across 13 WHO ICD-10 medical classifications and 63 topics. If we would have included overviews without systematic searches/methods sections, we believe there would have been more overlapping overviews identified. As many as 7 overviews (median of 2) were completed on the same topic (e.g. regarding acupuncture for pain). We found that it was common for some overlapping overviews to cover broad topic areas, whereas others considered only subsets of the evidence. Though slight differences in the scope of the overlapping overviews were observed, we feel the multiplicity represents some unnecessary overlap. This study and the database of overviews found in Appendix D can provide a guide to authors about which topics are covered, and gaps in the evidence for future analysis. Overview authors can use the appendix to determine if the topic they are wanting to examine is already published.

Legitimate reasons for observed overlap

Observed overlap can be legitimately justified for several reasons. Overviews may be out-of-date and therefore an update including more recent studies published is needed. This reason is justified especially when systematic reviews on the topic are inconclusive, and their synthesis may reconcile discrepancies in their results, interpretation and conclusions. A duplicate overview may be warranted when an older overview used inappropriate or invalid methods, or was of low methodological quality (e.g. if whether re-assessments of risk of bias, re-analyses of data, or (re)evaluation of GRADE assessments are required). Broader, more comprehensive overviews may be warranted if existing overviews are narrow in focus. A rationale for why an overlapping overview is needed should be provided by the authors.

Finally, replication is also an appropriate reason to conduct an overview on the same topic with the same or similar PICO. Reproducibility of research by independent and conflict-free academics to obtain the same (or similar) results when repeating an experiment or test is one of the hallmarks of good science [41]. As defined by Karunananthan et al. [42] for reviews, there are two types of replications. The first type involves repetition of PICO, using the same or very similar methods as a previous study to determine whether similar results can be obtained [42]. The second type involves broadening or narrowing the PICO of a previous overview. In this type of replication, the PICO are overlapping but broader than the original study, to explore generalizability of the findings; or are narrower, to provide a more in-depth review of a subgroup included in a previous broader overview [42]. This later type of replication allows authors to check the boundaries of independent variables and how the operationalization and documentation of concepts in the first study influenced the results.

With replication of the same research results, decision makers, healthcare workers and patients can be confident in the consistency and trustworthiness of the research [43]. Dr Moher recently suggested two to three replicated systematic reviews with similar eligibility criteria would help ensure reliability of the findings, but four or more might represent unnecessary duplication and research waste [43].

Potential for discrepant results and conclusions across overlapping overviews

We would hope that overviews on the same topic would find the same results and come to similar conclusions, but this is often not the case with systematic reviews [12, 44-46]. Potential discrepancies can cause endless debates (e.g. [47]), and confuse and possibly mislead clinicians and policymakers [5]. Avoidance of this problem is the responsibility of researchers thinking about conducting overviews, funders, and publishers of this type of research.

Potential solutions and recommendations

To avoid unnecessary redundancy, several strategies can be used. First, protocols of overviews should be registered in a targeted database; second, journal editors and peer reviewers, funders and commissioners should require acknowledgement of other similar overviews and scrutinize the rationale where a de novo overview is proposed, rather than repurposing or updating an existing overview; and third, authors should cite the other known overviews that overlap in scope with a clinical or methodological rationale as to why the study is needed [48]. Authors should clearly outline in their study protocol how their overview is different than earlier similar overviews based on: (a) the search strategy, (b) the results of the search, and (c) the screening and inclusion criteria, and then justify why a new overview is needed. Authors should also define why the choice of an overview design is preferable over other synthesis types such as scoping reviews, systematic reviews with meta-analyses, or network meta-analyses.

We do not advocate that authors conduct ‘meta-overviews’ (i.e. overviews of overviews) to try and explain differences in results across multiple overviews, as this would produce more redundant and potential useless research. Our list of 541 overviews have undergone some basic quality checks for inclusion (reported methods, conducted a systematic search, synthesised systematic review with or without meta-analysis), and can be used in policy decisions. There is currently no quality appraisal tool for overviews, and hence overlapping overviews cannot at this time be chosen based on a quality assessment. To choose one overview among several on the same topic, we suggest policymakers choose the overview with the closest research question in terms of population, interventions and outcomes. Recency and comprehensiveness of the evidence accumulated should be next examined to determine which overview represents the most current and thorough for policy decisions.

Currently, there is no dedicated database registry for protocols of overviews, such as there is for systematic reviews (i.e. PROSPERO). A dedicated database for overview protocols, as well as the development of a MeSH term for overviews would h0elp in their identification by prospective overview authors, who when finding a similar overview, could choose a different topic or new scope to explore. A published search filter for overviews [24] can help in their identification, as would imbedded filters in databases such as Epistemonikos, MEDLINE, and Embase. The Cochrane Database of Systematic Reviews attempts to avoid duplication of effort by publishing only one overview and systematic review per topic of interest. Other journals should follow suit and avoid duplicate publication of overviews with similar PICOs.

During the protocol phase and conduct of the overview, guidance [49] and methodological studies [50-53] should be consulted by to ensure rigor and a consistently high level of quality. Overviews of high quality will reduce the need for overlapping overviews and aid in avoiding wasting researchers’ time, effort and resources.

Strengths and limitations

Despite the growing popularity of overviews as a method to synthesise systematic reviews, to our knowledge this is the first study to examine overlap across a cohort of overviews. A strength of our research is that we based our methods on systematic review guidance, and searched using a validated search strategy for overviews. We selected the overviews based on stringent eligibility criteria using two independent reviewers, who then compared their results and identified and resolved discrepancies.

One of our major limitations was classifying overlapping topics in overviews. Many broad overview topics could have been classified under several ICD-10 classifications. For example, the overview entitled “Melatonin for health” was classified under “Mental and behavioural disorders” due to the primary theme of its outcomes (sleep latency, pre-operative anxiety, prevention of agitation and risk of breast cancer). This same overview could have also been classified under the ICD-10 classification “Factors influencing health status and contact with health services”. There was also room for error or subjectivity when extracting the overview eligibility criteria, i.e. PICOs to identify overlap. Performing this step in duplicate was an attempt to ensure accuracy but we recognize that some of the overview topics were specialized and nuanced. Another limitation is that we excluded protocols of overviews; including them would have given us a broader sense of overlap.

Our empirical evaluation identified duplication in 32% of overviews dated between 2000 and 2018. Sixty-three topics across 13 WHO ICD-10 medical classifications had overlapping overviews. As many as 7 overviews addressed the same topic (e.g. acupuncture for pain) representing unnecessary overlap. We found that it was common for overlapping overviews to cover broad topic areas, whereas others considered only subsets of the evidence. Most frequently, broad overviews had targeted populations for which multiple interventions were addressed, and least frequently broad overviews addressed a targeted intervention for multiple populations. A multiplicity of overviews on the same topic adds to the ongoing waste of research resources, time and effort across medical disciplines.

To avoid duplication of research and redundancy, protocols of overviews should be registered in a targeted database, and overviews should cite other studies on similar topic with a rationale. This study and database of overviews provides a guide to authors about which topics are covered, and gaps in the evidence for future analysis.

This is the first study to examine overlap across a sample of overviews of reviews.
Of 541 overviews published from 2000 to 2018, 172 (32%) were considered overlapping in topic with another overview.

The majority of overlapping overviews covered broad topic areas, whereas others considered only subsets of the evidence. Most frequently, broad overviews had targeted populations for which multiple interventions were addressed, and least frequently broad overviews addressed a targeted intervention for multiple populations.
A limitation was that the coding of overlap was prone to error or subjectivity when determining the overlap in PICOs. Performing this process in duplicate was an attempt to ensure accuracy. To avoid overlap, a dedicated database for overview protocols, as well as the development of a MeSH term for overviews would help in their identification by prospective overview authors

Overlapping overviews, without warranted rationale, represent wasted efforts on the part of investigators and can confuse and mislead clinicians and policymakers.

1. Lunny, C., et al., Bibliometric study of ‘overviews of systematic reviews’: evaluation of prevalence, citation and impact factor. 2021, Research Square.

2. Ioannidis, J.P., The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q, 2016. 94(3): p. 485-514.

3. Siontis, K.C. and J.P.A. Ioannidis, Replication, Duplication, and Waste in a Quarter Million Systematic Reviews and Meta-Analyses. Circ Cardiovasc Qual Outcomes, 2018. 11(12): p. e005212.

4. Moher, D., The problem of duplicate systematic reviews. BMJ : British Medical Journal, 2013. 347: p. f5040.

5. Ioannidis, J.P.A., The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. The Milbank quarterly, 2016. 94(3): p. 485-514.

6. Zhang, W., et al., OARSI recommendations for the management of hip and knee osteoarthritis, part I: critical appraisal of existing treatment guidelines and systematic review of current research evidence. Osteoarthritis Cartilage, 2007. 15(9): p. 981-1000.

7. NHMRC, The Australian government's natural therapies review 2019-20 and the role of the national health and medical research council's natural therapies working committee. Australian Journal of Herbal and Naturopathic Medicine, 2019. 31(4): p. 140-141.

8. Caird, J., et al., Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evidence & Policy: A Journal of Research, Debate and Practice, 2015. 11(1): p. 81-97.

9. Chalmers, P.N., et al., Do arthroscopic and open stabilization techniques restore equivalent stability to the shoulder in the setting of anterior glenohumeral instability? a systematic review of overlapping meta-analyses. Arthroscopy, 2015. 31(2): p. 355-63.

10. Fountoulakis, K.N., R.S. McIntyre, and A.F. Carvalho, From Randomized Controlled Trials of Antidepressant Drugs to the Meta-Analytic Synthesis of Evidence: Methodological Aspects Lead to Discrepant Findings. Curr Neuropharmacol, 2015. 13(5): p. 605-15.

11. Lucenteforte, E., et al., Discordances originated by multiple meta-analyses on interventions for myocardial infarction: a systematic review. J Clin Epidemiol, 2015. 68(3): p. 246-56.

12. Osnabrugge, R.L., et al., A systematic review and critical assessment of 11 discordant meta-analyses on reduced-function CYP2C19 genotype and risk of adverse clinical outcomes in clopidogrel users. Genetics in Medicine, 2015. 17(1): p. 3-11.

13. Pantoja T, O.N., Ciaponni A, Herrera C, Lewin S, Oxman A, Paulsen E, Rada G, Wiysonge C Strategies for improving health systems in low-income countries: lessons learnt from four overviews of systematic reviews of health systems interventions. in Cochrane Colloquium. 2015. Vienna.

14. Pollock A, F.S., Brady M, Langhorne P, Mead G, Mehrholz J, van Wijck F Completing the first Cochrane overview of stroke reviews: experiences of the Cochrane Stroke Group in Cochrane Colloquium. 2015. Vienna.

15. Useem, J., et al., Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis. PLoS One, 2015. 10(12): p. e0144980.

16. Zhao, J.G., J. Wang, and L. Long, Surgical Versus Conservative Treatments for Displaced Midshaft Clavicular Fractures: A Systematic Review of Overlapping Meta-Analyses. Medicine (Baltimore), 2015. 94(26): p. e1057.

17. Zhao, J.G., et al., Intramedullary nail versus plate fixation for humeral shaft fractures: a systematic review of overlapping meta-analyses. Medicine (Baltimore), 2015. 94(11): p. e599.

18. Siontis, K.C., T. Hernandez-Boussard, and J.P.A. Ioannidis, Overlapping meta-analyses on the same topic: survey of published studies. BMJ : British Medical Journal, 2013. 347: p. f4501.

19. Gurevitch, J., et al., Meta-analysis and the science of research synthesis. Nature, 2018. 555(7695): p. 175-182.

20. Ding, D., et al., Duplicate and salami publication: a prevalence study of journal policies. International journal of epidemiology, 2020. 49(1): p. 281-288.

21. Lunny, C., et al., Bibliometric study of ‘overviews of systematic reviews’: evaluation of prevalence, citation and impact factor (preprint). 2021: Researchsquare.com.

22. Ioannidis, J.P., et al., Meta-research: evaluation and improvement of research methods and practices. PLoS biology, 2015. 13(10): p. e1002264.

23. Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., Welch, V.A. (editors). , Cochrane Handbook for Systematic Reviews of Interventions version 6.0. . 2019, Cochrane. Available from www.training.cochrane.org/handbook: London, United Kingdom.

24. Lunny, C., J.E. McKenzie, and S. McDonald, Retrieval of overviews of systematic reviews in MEDLINE was improved by the development of an objectively derived and validated search strategy. J Clin Epidemiol, 2016. 74: p. 107-18.

25. Lunny, C., et al., Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. Journal of Clinical Epidemiology, 2020. 118: p. 69-85.

26. Ernst, E. and P. Posadzki, Complementary and alternative medicine for rheumatoid arthritis and osteoarthritis: an overview of systematic reviews. Current pain and headache reports, 2011. 15(6): p. 431-437.

27. Moe, R.H., et al., Effectiveness of nonpharmacological and nonsurgical interventions for hip osteoarthritis: an umbrella review of high-quality systematic reviews. Physical therapy, 2007. 87(12): p. 1716-1727.

28. Mac-Namara, M. and G. Rada, Is rituximab effective for induction of remission in lupus nephritis? Medwave, 2014. 14(7): p. e6010-e6010.

29. Chen, Y., et al., Treatment for lupus nephritis: an overview of systematic reviews and meta-analyses. Rheumatology international, 2017. 37(7): p. 1089-1099.

30. De Niet, G.J., et al., Review of systematic reviews about the efficacy of non‐pharmacological interventions to improve sleep quality in insomnia. International Journal of Evidence‐Based Healthcare, 2009. 7(4): p. 233-242.

31. Ernst, E., M.S. Lee, and T.-Y. Choi, Acupuncture for insomnia? An overview of systematic reviews. The European journal of general practice, 2011. 17(2): p. 116-123.

32. Botero, J., C. Rodríguez, and A. Agudelo‐Suarez, Periodontal treatment and glycaemic control in patients with diabetes and periodontitis: an umbrella review. Australian Dental Journal, 2016. 61(2): p. 134-148.

33. Hasuike, A., et al., Systematic review and assessment of systematic reviews examining the effect of periodontal treatment on glycemic control in patients with diabetes. Medicina oral, patologia oral y cirugia bucal, 2017. 22(2): p. e167.

34. Lopez, N.J., S. Uribe, and B. Martinez, Effect of periodontal treatment on preterm birth rate: a systematic review of meta‐analyses. Periodontology 2000, 2015. 67(1): p. 87-130.

35. Vivares-Builes, A.M., et al., Gaps in knowledge about the association between maternal periodontitis and adverse obstetric outcomes: an umbrella review. Journal of Evidence Based Dental Practice, 2018. 18(1): p. 1-27.

36. Ezzo, J., K. Streitberger, and A. Schneider, Cochrane systematic reviews examine P6 acupuncture-point stimulation for nausea and vomiting. Journal of Alternative & Complementary Medicine, 2006. 12(5): p. 489-495.

37. Towler, P., A. Molassiotis, and S. Brearley, What is the evidence for the use of acupuncture as an intervention for symptom management in cancer supportive and palliative care: an integrative overview of reviews. Supportive care in cancer, 2013. 21(10): p. 2913-2923.

38. Wu, X., et al., Effectiveness of acupuncture and related therapies for palliative care of cancer: overview of systematic reviews. Scientific reports, 2015. 5(1): p. 1-15.

39. Posadzki, P., Is spinal manipulation effective for pain? An overview of systematic reviews. Pain Medicine, 2012. 13(6): p. 754-761.

40. Posadzki, P. and E. Ernst, Spinal manipulation: an update of a systematic review of systematic reviews. NZ Med J, 2011. 124(1340): p. 55-71.

41. Popper, K., The logic of scientific discovery. 2005: Routledge.

42. Karunananthan, S., et al., When and how to replicate systematic reviews. Cochrane Database of Systematic Reviews, 2020(2).

43. Moher, D., The problem of duplicate systematic reviews. 2013, British Medical Journal Publishing Group.

44. Lucenteforte, E., et al., Discordances originated by multiple meta-analyses on interventions for myocardial infarction: a systematic review. Journal of Clinical Epidemiology, 2015. 68(3): p. 246-256.

45. Augustin, G., et al., Discordant outcomes of laparoscopic versus open appendectomy for suspected appendicitis during pregnancy in published meta-analyses: an overview of systematic reviews. Surgical Endoscopy, 2020. 34(10): p. 4245-4256.

46. Dettori, J.R. and D.C. Norvell, Discordant Systematic Reviews: Which to Believe? Global Spine Journal, 2020. 10(2): p. 237-239.

47. Trinquart, L., D.M. Johns, and S. Galea, Why do we think we know what we know? A metaknowledge analysis of the salt controversy. International Journal of Epidemiology, 2016. 45(1): p. 251-260.

48. Pieper, D., et al., Systematic review finds overlapping reviews were not mentioned in every other overview. Journal of clinical epidemiology, 2014. 67(4): p. 368-375.

49. Pollock, M., Fernandes, R.M., Becker, L.A., Pieper, D., Hartling, L.C, Chapter V: Overviews of Reviews. , in Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Available from www.training.cochrane.org/handbook, T.J. In: Higgins JPT, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). , Editor. 2020, Cochrane: UK.

50. Lunny, C., et al., Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design. Syst Rev, 2016. 5: p. 4.

51. Lunny, C., et al., Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev, 2017. 6(1): p. 231.

52. Lunny, C., et al., Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev, 2018. 7(1): p. 159.

53. Lunny, C., et al., Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. J Clin Epidemiol, 2020. 118: p. 69-85.

Ethics approval and consent to participate

N/A

Transparency declaration

The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Data sharing and availability of data and materials

All data is in the main manuscript or the supplementary files.

Competing interests

The authors have declared no competing interest.

Funding

This work did not receive any funding.

Dissemination declaration

The results will be disseminated in conference and university workshops, social media, and email lists.

Authors’ contributions

CL conceived of the study design, wrote the study protocol, extracted and analyzed the data, wrote and revised the final manuscript. GS, YWZ, CL, LL, PHY, MD, PN screened citations and full-text articles. CL and JHZ abstracted and analysed the data. CL, JZ, TN, AC, GS, AS, ST, SS, SA, YWZ, LL, PHY, MD, PN, UE, BJS, EKR, and JMW edited and approved of the final manuscript,

Acknowledgements

No acknowledgements.

AppendixABandC.docx
Appendices Appendix A – Search Strategy Appendix B - Figure 1: PRISMA flowchart of study selection Appendix C – 541 overviews included
AppendixDOverlappingoverviews20210824.xlsx
Appendix D lists the 172 overviews that overlapped for the 63 topics with their WHO ICD-10 medical classification, population, interventions, and outcomes.

Topic overlap and research waste at the ‘Overviews of systematic reviews’ level: a meta-research study

Status:

Journal Publication

Version 2

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Results

Discussion

Conclusions

Highlights

References

Declarations

Supplementary Files

Status:

Journal Publication

Version 2