An Overview of Retraction Status and Reasons of Non-Cochrane Systematic Reviews in Medicine

doi:10.21203/rs.3.rs-125016/v1

Download PDF

Research

An Overview of Retraction Status and Reasons of Non-Cochrane Systematic Reviews in Medicine

https://doi.org/10.21203/rs.3.rs-125016/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 24 Jun, 2021

Read the published version in Journal of Clinical Epidemiology →

Version 1

posted

You are reading this latest preprint version

Background

Many previous studies have analyzed the status of retracted publications from different perspectives, but so far no study has focused on systematic reviews (SRs). The purpose of this study is to analyze the retraction status and reasons of non-Cochrane SRs in medicine.

Methods

We searched MEDLINE and Embase from their inception until April 18, 2020, as well as Retraction Watch Database and Google Scholar with no language restriction to find non-Cochrane SRs that were retracted for any reason. Two reviewers independently screened and extracted data. We described the characteristic and reasons of retraction and the duration from publication to retraction.

Results

We identified 150 non-Cochrane SRs in medicine retracted between 2004 and 2020. The majority of retracted SRs were led by authors from China (n=113, 75.3%) and affiliated with hospitals (n=106, 70.7%). Most SRs were published in journals with an impact factor ≤3 (n=85, 56.7%). The largest proportion of retraction notices were issued by the publisher and editor(s) jointly (n=60, 40.0%); seven did not report this information. Fraudulent peer-review (n=61, 37.9%) was the most common reason for retraction, followed by unreliable data (n=41, 25.5%) meaning errors in study selection or data analysis. The median time between publication and retraction was 14.0 months. SRs retracted due to research misconduct took longer to retract than SRs retracted because of honest error.

Conclusions

The situation with retracted SRs is critical worldwide, and in particular in China. The most common reasons for retraction are fraudulent peer-review and unreliable data, and in most cases the study is retracted more than a year after publication. Efforts should be made to improve the process of peer review and adherence to the COPE retraction guidance, and authors should strengthen their skills in SR methodology.

Health Economics & Outcomes Research

Retraction

Systematic review

non-Cochrane

Research ethics

This manuscript systematically reviews retracted systematic reviews (SRs) in medicine, and for SR developers, users, researchers and other stakeholders, much efforts should be made from policy to methodology.
The development process and evidence source has a great influence on the quality of guidelines, and the quality poses an important part as barriers or facilitators in guideline dissemination and implementation.
These findings contribute to explore detailed reasons and design strategies in the evidence-based practice, of which, evidence users and guidelines developers should pay more attention to reliability of guideline itself.

Retractions of published articles are sometimes necessary to maintain the integrity of the scientific literature and to alert the reader of potential serious problems identified within the article ^[1,2]. The Committee on Publication Ethics (COPE) published retraction guidance in 2009 ^[2]. Reasons for retraction include unreliable data, redundant publication, plagiarism, and failure to disclose conflict of interest. Unreliable data may result from honest error (such as mis-calculation or experimental error) or from research misconduct (such as data fabrication)^[2]. A study published in 2011 found that in the past decade, the number of retraction notices has shot up 10-fold, even though the literature has expanded by only 44% ^[3]. Another study published in 2019 showed that of the 21,859,178 publications indexed in the Web of Science database between 1978 and 2017, 2,859 were retracted, of which, the United States had the most retractions, and China had the highest retraction rate ^[4]. Retractions involved papers led by researchers from over 50 countries worldwide ^[5]. Having a paper retracted does not only affect the academic status and reputation of individual authors, leading possibly to administrative, civil, and criminal sanctions, but also correlate with decreased productivity, funding and resources ^[5-7]. Retractions of publications in the field of biomedicine also propose a great threat to public health ^[8], for example, the Lancet recently retracted studies related to hydroxychloroquine for the treatment of COVID-19 due to veracity of data ^[9].

Systematic reviews (SR), which are being widely used in medicine, have had a revolutionary impact in many fields of science and helped to establish evidence-based practice ^[10,11]. Scientifically rigorous SRs, carried out following formal protocols, have long been considered at the top of the hierarchy of studies for evidence-based decision-making. SRs integrate information, address well-defined questions and lead to new insights, but also provide a robust overview of a problem or field of research, and serve as important evidence for development of clinical practice guidelines. Therefore, SRs are an essential part of clinical decision-making and research ^[12,13]. To avoid waste of research, no new studies should be done without a SR of existing evidence ^[14]. However, while the number of published and registered SRs has increased substantially over the past 20 years, the extent to which their implications are correctly understood is poor. The majority of produced SRs are unnecessary, misleading, or conflicted ^[15-19], and this is a particular problem among non-Cochrane SRs that tend to use less rigorous methods than Cochrane reviews ^[20]. In recent years, there is a growing call to reduce the number of articles that need to be later retracted.

To the best of our knowledge, no study has yet specifically aimed to describe the main features of retracted SRs. Therefore, on behalf of PRESS (Publication Science of Retracted Studies) working group, the present study aims to perform an in-depth analysis of the status and reasons of retracted non-Cochrane SRs and meta-analysis in the field of medicine.

Search strategy

Two reviewers (Qianling Shi and Zijun Wang) comprehensively searched MEDLINE (via PubMed) and Embase from their inception through April 18th, 2020. No language restriction was applied. We conducted the search by combining the MeSH and free words including “Retraction”, “Withdraw”, “Remove”, “Meta-analysis”, “Systematic Review” and their derivatives. The search strategy was also peer reviewed by an external specialist. Details of the search can be found in the Additional file 1. We also retrieved data from the Retraction Watch Database (RWdb, www.retractionwatch.com) and Google Scholar (https://scholar.google.nl/) for relevant studies that met the inclusion criteria. RWdb is a public platform for retractions of scientific papers, it is considered as the largest and most comprehensive database of retracted articles ^[21].

Inclusion and exclusion criteria

We included all non-Cochrane SRs published in the field of medicine that were retracted. The corresponding retraction notices were also extracted for analysis. Studies were excluded if they were: 1) duplicates; 2) used the terms of “retraction” or “retracted” but with a different meaning from the one considered by the present review; 3) SRs on topics unrelated to medicine; 4) SRs on topics related to veterinary medicine; 5) retracted Cochrane SRs; and 6) retracted SRs that were republished.

Study selection

Two reviewers (Qianling Shi and Zijun Wang) independently screened first the titles and abstracts after eliminating duplicates, and then the full-texts to identify relevant articles, using pre-defined criteria. The specific bibliographic software EndNote X9 was used. Discrepancies were discussed, if necessary, with a third researcher (Qi Zhou). The process of study selection was documented using a flow diagram.

Data extraction

Two reviewers (Qianling Shi and Zijun Wang) independently extracted the following data with a standardized data collection form: 1) basic information (title of retracted article, country and affiliation of the first author, year and month of online publication and retraction); 2) journal information (journal name, impact factor [IF] of 2018 according to ISI Web of Knowledge, quartile in Journal Citations Reports (JCR) of Thomson Reuters of 2018); 3) retraction characteristics (who retracted the article [the decision to retract a paper can be made by editors, authors or the author's employer, publisher or others], reasons for retraction, availability of retraction notice).

After considering classifications of retractions in previous studies, retraction reasons for SRs were classified into the following categories ^[1,2,5,22].

Repetitive research: the content of the retracted article is the same as that of a previously published paper;
Data falsification: data has been manipulated or made up;
Duplicate publication: the paper had been published more than once or there was significant overlap between manuscripts (usually as a result of author misconduct);
Authorship issues: not all authors aware of manuscript submission;
Plagiarism: duplication of text from previously published articles;
Unreliable data: data has error due to miscalculation or experimental error; in the case of SRs, this can be an error in, for instance, study selection, data collection, data analysis, or interpretation of result.
Fraudulent peer-review: it can arise when editors rely on authors’ recommended reviewers. The names are often genuine but have a false e-mail address that enable the authors to write a favourable review of their own paper, thereby facilitating acceptance.
Others: reasons that cannot be assigned into any of the above categories.

The reasons were classified by one reviewer (Qianling Shi) and checked for agreement by another reviewer (Zijun Wang) using the information given in the retraction notice. Any discrepancies were settled by discussion. If we were unable to retrieve the notice, we searched the RWdb to find information about the reasons and characteristics of retraction. When multiple reasons for the retraction were given, all were described and noted.

Statistical analysis

Descriptive statistics of the retraction characteristic, reasons for retraction, and time to retraction (defined as the time from original publication date to retraction date) of articles were calculated. Two reviewers (Qianling Shi and Zijun Wang) analysed all datasets independently with SPSS 18 Software. Comparisons of the causes for retraction among sub-groups (IF, journal rank in JCR) were performed using the Chi square test or variance analysis, and t test was used to compare the times to retraction. Statistical significance was set at P < 0.05. Categorical data were reported using absolute numbers and percentages. Continuous variables were described using median with range (minimum to maximum) or interquartile range (IQR). We also analyzed the adherence to all recommendations of the COPE guidance on retraction which suggest that retraction notices should be linked to the retracted article, be clearly identifiable as a retraction, be published promptly and be freely available to all readers. Retractions should also state who retracted the article and reason(s) for retraction, and avoid statements that are potentially defamatory or libellous ^[2].

Search results

Electronic and additional search resulted in a total of 1,945 records. Of these records, 945 were excluded as duplicates. After screening the titles, abstract and full texts, a total of 150 non-Cochrane SRs in medicine retracted between 2004 and 2020 were included. The process of study selection including the reasons for exclusion is shown in Figure 1.

Study characteristics

The characteristics of the 150 included studies are displayed in Table 1. The total number of published SRs is large and has been growing over the past decade (from 2011 to 2020), and the number of retracted SRs is small but increased as well (Figure 2). The oldest included SR, published in 2003, was retracted in 2004, and the highest retraction rate was in 2015 (n=43, 28.7%). The first authors of retracted SRs were from 18 countries, most from Asia (n=126, 84.0%). China was the leading country of origin for the total number of retracted publications (n=113, 75.3%), followed by the United States (n=5, 3.3%), Italy (n=5, 3.3%), the United Kingdom (n=4, 2.7%) and Iran (n=4, 2.7%). The most frequent type of organizations of the first author were hospitals (n=106, 70.7%), followed by universities (n=41, 27.3%).

A total of 79 journals were involved. Tumor Biology retracted the most (n=23, 15.3%), followed by Molecular Biology Reports (n=13, 8.7%), Medicine (n=9, 6.0%) and Molecular Neurobiology (n=9, 6.0%). All included SRs were written in English (except one that was published in Chinese. Most articles (n=120, 80.0%) were published in journals listed in the Science Citation Index (SCI). Most retracted SRs (n=85, 56.7%) were published in low impact (≤3) journals, and the distribution across the JCR ranking quartiles was relatively even.

Not all retraction notices were clearly labelled and linked to the retracted articles. For a total of 148 (98.7%) studies a retraction notice was published, the first of which in 2004. For two (1.3%) studies no retraction notice was issued, but the study was presented with a watermark to identify the article was retracted. For these studies, we determined the issuer of the retraction notice as unclear.

Characteristic of the retractions

Issuer of the retraction notice

We categorized the entities that retracted the 150 SRs into nine groups (Table 2). The largest proportion of retraction notices were issued by the publisher and editor(s) jointly (n=60, 40.0%), followed by the editor alone (n=27, 18.0%) and author alone (n=16, 10.7%). For nine (6.0%) included studies, we were unable to determine who issued the retraction notice. Of these nine notices, two were from the author(s) and/or editor, one was from publisher or journal, three used the words as “we”, one used the words as “they”, and for two no notice was issued. Although most retraction notices declared who retracted the article, 7 (4.7%) did not explicitly state this information.

Reasons of retraction

A total of 161 reasons for retraction were mentioned for the included 150 SRs (Table 3). We divided the retraction reasons into three categories: honest error (unreliable data), misconduct (data fabrication, plagiarism, duplicate publication, authorship issues, compromised peer review), and unclear (not possible to distinguish “honest error” from “misconduct”). The vast majority of SRs (n=100, 66.7%) were retracted due to some form of research misconduct.

The most common reason for retraction was fraudulent peer-review (n=61, 37.9%), followed by unreliable data (n=41, 25.5%) and plagiarism (n=30, 18.6%). Most reasons classified as unreliable data were due to error in either the selection of included studies, or in data analysis (Table 4). Other reasons included duplicate publication (n=9, 5.6%), authorship issues (n=8, 5.0%) and only one (0.6%) was retracted due to data fabrication. No articles were retracted due to repetitive research. We assigned seven (4.4%) retraction statements, which appeared to use deliberately ambiguous wording that made it difficult to distinguish honest errors from suspected (or proven) misconduct, into others (n=7, 4.4%). No reason was provided for four (2.5%) retractions. Figure 3 shows the growth and variation in reasons for retractions over time.

In 11 (7.3%) studies two reasons for retraction were given. Among the 11 retracted SRs, four were retracted due to fraudulent peer-review and plagiarism, two were retracted due to plagiarism and authorship issues, two were retracted due to fraudulent peer-review and authorship issues, one was retracted due to plagiarism and authorship issues, one due to plagiarism and unreliable data, and one due to unreliable data and concerns about third party involvement in editing submitted files.

In addition to this, causes for retraction also varied according to journal IF (P＜0.001) and journal rank in JCR (P＜0.001): retractions due to unreliable data were most frequent from journals with a high IF (equal or more than 5) or high journal rank (Q1 and Q2), while a faked review process was the main cause of retraction from journals with a low IF (less than 5), low journal rank (Q3 and Q4) and journals with no citation index. In-depth analysis showed that the most common reason for retraction for articles with led authors from hospitals and universities were fraudulent peer-review and unreliable data, respectively.

Retraction interval

The median (range) time between publication of the 150 SRs and their retraction was 14.0 (0.0~108.0) months. Most of the included SRs (n=105, 70.0%) were retracted within 20 months after publication. Seven SRs were retracted within one month, of which, five on the same day the article was published, we therefore determined the month to retraction was zero.

The duration from publication to retraction also varied by the reasons (Table 5). When excluding the reason of data fabrication (only one study with an interval of 75 months), authorship issues resulted in the longest interval with a minimum of six month and a maximum of 50 months, followed by fraudulent peer-review, plagiarism, duplicate publication and unreliable data. In general, SRs with research misconduct (including data fabrication, fraudulent peer-review, authorship issues, plagiarism, and duplicate publication) took longer to be retracted (16.0 [10.00~31.3] months) than honest error (7.0 [4.0~22.0] months) in total, P＜0.001.

Reliability of retracted SRs

Retracted SRs have the potential to mislead current practice and future research through citations that occur before the publication of the retraction note, and erroneous citations that are made after the retraction. It is therefore important for researchers and readers to understand how much influence retracted SR can have on other studies. Our research group divided the reliability of retracted SRs into three categories:

High risk: The findings or conclusions of retracted SRs will be changed due to retraction so that citations will be severely affected (including unreliable data, data falsification).
Low risk: The findings or conclusions of retracted SRs will not be changed due to retractions so that citations will not be affected (including repetitive research, duplicate publication and authorship issues).
Unclear: we are uncertain about whether the findings or conclusions of retracted SRs will be changed or whether citations will be affected (including compromised peer review and plagiarism)

The results of our study showed that 42 (28.0%) retracted SRs were considered as “high risk”, and only 12 (8.0%) as “low risk”.

Our study identified a total of 150 retracted non-Cochrane SRs in the field of medicine by 18 April 2020. The number of retractions steadily increased over time. The most common reasons were fraudulent peer-review and unreliable data. More than 75% of retracted articles were from China. Detailed analysis of the retraction notices revealed that some aspects of COPE guidelines on retraction were not adhered to. The median duration from publication to retraction was more than one year, and articles involving apparent misconduct took longer to retract than those retracted for honest error.

The results from our study suggested that faked peer review was the most common reason for retraction. The number of retracted SRs peaked in 2015, which may be related to the large-scale retractions of academia happened in this year. The publisher SpringerNature retracted 64 articles in ten of its journals on the basis of fake peer review on August 18, 2015 ^[23]. The retractions came only months after BioMed Central retracted 43 articles on similar grounds ^[24]. Two years later, Springer retracted another 107 papers published in Tumor Biology^[25]. Data from the wake of large-scale retraction scandals and recent studies in other fields showed that most of the retractions were from China, and majority of SRs were conducted by researchers from hospitals and published in journals with low IF. Tumor Biology and Molecular Biology Reports retracted the largest number of studies; these results were similar to those identified in other studies ^[5,23-30]. SRs involving research misconduct (especially authorship issues and fraudulent peer-review) took longer to retract than research with honest errors, which was also observed in previous studies ^{[23-25,31-34]}.

As commented by Alison McCook on the Retraction Watch blog ^[35], retractions due to fabricated peer reviews are becoming a trend. With the current global research output with 20,000 journals publishing more than 2 million articles per year and thousands of scientists publishing a paper every five days ^[36-37], researchers worldwide are under great pressure to publish papers indexed by SCI for graduation, promotion, and acquisition of research funds. The need to publish has grown beyond the capacity of the scientists ^[38-39]. Peer review aims to ensure that articles are “open and honest”, but this target may be compromised when editors rely on authors' reviewer recommendations. This form of misconduct is particularly intense in China, but it would be a mistake to look at this as a Chinese or Asian problem. The overwhelming incentives in scientific publishing exists everywhere in the world and possess a serious risk of scientific misconduct.

We also found that unreliable data was one of the most important reasons for the retraction, involving more than one fourth of the studies, and some SRs had included retracted primary studies ^[40-42]. SRs have the potential to provide sufficient sample size and generalizable population information to make more powerful evidence-based conclusions. Thus, great care must be used in the data analysis and determination of eligible studies. The methodology for conducting SRs has advanced enormously in the recent 30 years, so authors should use the latest methods to strengthen the study, and journals should refuse to publish SR not meeting rigorous standards ^[43-45]. It is recommended to use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and A Measurement Tool to Assess Systematic Reviews (AMSTAR) tool in the design, reporting and evaluation of SRs ^[46-48]. More importantly, previous studies have shown that retracted literature continues to be cited as valid and legitimate work in many scientific disciplines, even after retracted ^[49]. SRs are cornerstones of clinical practice guidelines in the medical field, and if retracted SRs are cited, the recommendations will be affected. Especially for SRs related to critical conditions, this will pose a great threat to medical care ^[50]. Therefore, appropriate production, publication and citation of high-quality studies is essential for clinical decision-making and research.

In addition, our analyses suggested that some aspects of COPE guidelines on retraction were not adhered to. We found that some retracted SRs were not linked to a correspondence notice or given a “retracted” label, and some retraction notices did not explicitly state the retraction reasons and who issued the retraction. In addition, we classified issuers into editor, author, published and journal as notice reported, but it maybe difficult to distinguish many of these, because the editor represents the journal that is owned by the publisher, therefore, these are basically the same entity. Publication of a clear and unambiguous retraction notice is an important part of the retraction, and the notice should be clearly identifiable, freely available, published promptly and linked to the original article that is retracted (which also should be clearly labeled as retracted) ^[51].

Further suggestions

Regardless of the reasons for retraction, retracted SRs sometimes continue to be cited, so retracting alone is not enough. It is important to recognize the dangers related to citing retracted literatures and have clear guidance about how to address this issue. We propose the following interim recommendations:

Authors should strengthen and refine the learning of methodology. We recommend strictly adherence to the relevant guidelines to improve the quality from determining the clinical question until the reporting of results. We recommend open registration of all SRs to reduce the risk for duplicate work ^[52]. We also emphasize the need to adhere to the principles of “5 dont’s of academic publishing”: ghostwriting, ghost submission, ghost revision, fake peer review, and falsified authorship. In this process, it should be standard practice to double-check the final list of included studies against the list of retracted articles ^[53]. On the other hand, more detailed guidance on the inclusion and exclusion of retracted articles in SRs is needed, especially in cases when the reason for retraction is not directly associated with a risk of inaccurate data, which are unlikely to affect the results of the SR ^[54].

Journal editors and publishers should improve the quality of peer review and increase transparency of the retraction process. More background checks should be done when selecting reviewers (even if first following the recommendations). Artificial intelligence could provide valuable and efficient solutions to check the eligibility and potential conflicts of peer reviewers. Experienced librarians, information specialists, or other search strategy experts can also be invited as methodological peer reviewers for SRs ^[55]. Data should be strictly screened and checked before the manuscript is published, and most importantly, systematic and organized manipulation of the publication process should be altered and tackled ^[51]. We recommend to follow or adopt a checklist (such as COPE guidance) and templates when retracting articles to provide adequate information, so that readers can know who retracted the article and why the findings are considered unreliable. On the other hand, if an included study gets retracted after the SR was published, issuing a correction should be considered and the authors should be given the chance to update their SR.

For national and other academic institutions, the key to stemming the flow of poor-quality misleading research requires unified and standardized criteria and a change in academic culture. First, national academic committees should strengthen the development of a credibility system, and put in place relevant laws and regulations. Second, for studies with serious academic misconduct (such as data falsification and faked peer review), an effective system of precautions and penalties should be implemented, and giving considerations in revoking research funding for such authors. Third, academic institutions including but not limited to universities and hospitals should provide training in ethics of science and academic integrity to students, teachers, clinicians and other researchers, clarify the purpose of scientific research and reform unreasonable incentive mechanisms. For example, the way research success and article quality is measured could be reconsidered.

Evidence users including health professionals, especially guideline developers, should always involve methodological experts in the formulation of recommendations. Methodologists should rigorously review the cited SRs in terms of quality and make the decision whether to directly adopt, update or reproduce the review, to ensure the quality of evidence and that recommendations are not affected by the retracted publications ^[56-57]. Clinicians, public health officials and other investigators can use the AMSTAR tool to assess the methodological quality ^[48].

Strengths and limitations

This study is to our knowledge the first to characterize the retraction status and reasons among non-Cochrane SRs in medicine. We divided the risk of citation of retracted SRs into three categories according to the influence they can have on other studies. However, our study had also some limitations. First, we only searched the major international databases and websites, and did not search for example local or regional databases for research in other languages. Second, our study was limited to non-Cochrane SRs, and we excluded SRs that were republished, but some originally retracted SRs may have been republished with a new title or in a new journal, and thus incorrectly classified as still retracted. Third, with the enormous amount of published SRs and rapid reviews, especially during the ongoing COVID-19 pandemic, the rate and reasons of retraction may evolve rapidly.

An increasing number of non-Cochrane SRs in medicine are being retracted. China is the leading country of origin for retracted SRs. The most common reasons for retraction are fraudulent peer-review and unreliable data. The average time from publication to retraction was 14 months, but some SRs were retracted even as much as nine years after publishing. Adherence to the COPE guidelines needs to be improved to provide adequate information and increase transparency of the retraction process, and authors should strengthen their knowledge in methodology.

SR: systematic reviews; COPE: Committee on Publication Ethics; RWdb: Retraction Watch Database; IF: impact factor; JCR: Journal Citations Reports; IQR: interquartile range; SE: standard error; SD: standard deviation; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; AMSTAR: A Measurement Tool to Assess Systematic Reviews;

Ethics approval and consent to participate: Not applicable.

Consent for publication: Not applicable.

Availability of data and materials: The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests: All authors have no conflicts of interest to declare.

Funding: None

Author contributions: YL and QL (male) designed the paper and were consulted to revise the manuscript. QL (female) selected the literature, extracted data, carried out the statistical analysis, produced the tables and figures, and wrote the first edition of the paper. ZJ, QZ, RZ, XG and SE helped selected the literature and extracted data. SY helped statistical analysis. YF and XZ helped modify the first edition of the paper. All authors contributed to the review and approval of the final manuscript.

Acknowledgments: We thank Janne Estill, Institute of Global Health of University of Geneva for providing guidance and comments for our manuscript.

Moylan EC, Kowalczuk MK. Why articles are retracted: a retrospective cross-sectional study of retraction notices at BioMed Central. BMJ Open. 2016 Nov 23;6(11):e012047. doi: 10.1136/bmjopen-2016-012047.
Wager E, Barbour V, Yentis S, et al. Retractions: Guidance from the Committee on Publication Ethics (COPE). Croat Med J. 2009;50(6):532-535. doi: 10.3325/cmj.2009.50.532.
Van Noorden R. Science publishing: The trouble with retractions. Nature. 2011 Oct 5;478(7367):26-28. doi: 10.1038/478026a.
Tang L. Five ways China must cultivate research integrity. Nature. 2019 Nov;575(7784):589-591. doi: 10.1038/d41586-019-03613-1.
Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A. 2012 Oct 16;109(42):17028-17033. doi: 10.1073/pnas.1212247109.
Wang T, Xing QR, Wang H, Chen W. Retracted Publications in the Biomedical Literature from Open Access Journals. Sci Eng Ethics. 2019 Jun;25(3):855-868. doi: 10.1007/s11948-018-0040-6.
Stern AM, Casadevall A, Steen RG, Fang FC. Financial costs and personal consequences of research misconduct resulting in retracted publications. Elife. 2014 Aug 14;3:e02956. doi: 10.7554/eLife.02956.
Hamilton DG. Continued Citation of Retracted Radiation Oncology Literature-Do We Have a Problem? Int J Radiat Oncol Biol Phys. 2019 Apr 1;103(5):1036-1042. doi: 10.1016/j.ijrobp.2018.11.014.
Mehra MR, Desai SS, Ruschitzka F, et al. Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020 May 22:S0140-6736(20)31180-6. doi: 10.1016/S0140-6736(20)31180-6. Retraction in: Lancet. Jun 13;395(10240):1820.
Annane D, Jaeschke R, Guyatt G. Are systematic reviews and meta-analyses still useful research? Yes. Intensive Care Med. 2018 Apr; 44(4): 512-514. doi: 10.1007/s00134-018-5102-3.
Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018 Mar 7;555(7695):175-182. doi: 10.1038/nature25753.
Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol 2019 Jan 4;70: 747-770. doi: 10.1146/annurev-psych-010418-102803.
Institute of Medicine. Clinical practice guidelines we can trust [M]. Washington DC: National Academies Press. 2011. doi: 10.17226/13058.
Lund H, Brunnhuber K, Juhl C, et al. Towards evidence-based research. BMJ. 2016;355: i5440. doi: 10.1136/bmj.i5440.
Zorzela L, Golder S, Liu Y, et al. Quality of reporting in systematic reviews of adverse events: systematic review. BMJ. 2014 Jan 8;348:f7668. doi: 10.1136/bmj.f7668.
Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016 Sep;94(3):485-514. doi: 10.1111/1468-0009.12210.
Yao L, Sun R, Chen Y, et al. The quality of evidence in Chinese meta-analyses need to be improved. J Clin Epidemiol. 2016 Jun;74:73-79. doi: 10.1016/j.jclinepi.2016.01.003.
Siontis KC, Ioannidis JPA. Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. Circ Cardiovasc Qual Outcomes. 2018 Dec;11(12):e005212. doi: 10.1161/CIRCOUTCOMES.118.005212.
Yan P, Yao L, Li H, et al. The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study. J Clin Epidemiol. 2019 May;109(1): 20-29. doi: 10.1016/j.jclinepi.2018.12.013.
Moseley A M, Elkins M R, Herbert R D, et al. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009 Oct;62(10):1021-1030. doi: 10.1016/j.jclinepi.2008.09.018.
Brainard J, You J. Rethinking retractions. Science. 2018;362(6413):390-393. doi: 1126/science.362.6413.390.
King EG, Oransky I, Sachs T E, et al. Analysis of retracted articles in the surgical literature. Am J Surg. 2018 Nov;216(5):851-855. doi: 10.1016/j.amjsurg.2017.11.033.
Retraction of articles from Springer journals. London: Springer, August 18, 2015. [cited 2020 Apr 31]. Available online: https://www.springer.com/gp/about-springer/media/statements/retraction-of-articles-from-springer-journals/735218.
Dyer O. Major publisher retracts 43 papers, alleging fake peer review. BMJ. 2015 Apr 1;350:h1783. doi: 10.1136/bmj.h1783.
Stigbrand T. Retraction Note to multiple articles in Tumor Biology. Tumour Biol. 2017 Apr 20. doi: 10.1007/s13277-017-5487-6.
Al-Ghareeb A, Hillel S, Mckenna L, et al. Retraction of publications in nursing and midwifery research: A systematic review. Int J Nurs Stud. 2018 May;81:8-13. doi: 10.1016/j.ijnurstu.2018.01.013.
Cassão BD, Herbella FAM, Schlottmann F, et al. Retracted articles in surgery journals. What are surgeons doing wrong? Surgery. 2018 Jun;163(6):1201-1206. doi: 10.1016/j.surg.2018.01.015.
Chen W, Xing QR, Wang H, et al. Retracted publications in the biomedical literature with authors from mainland China. Sci Eng Ethics. 2018 Jan;114:217-227. doi: 10.1007/s11192-017-2565-x.
Lei L, Zhang Y. Lack of Improvement in Scientific Integrity: An Analysis of WoS Retractions by Chinese Researchers (1997-2016). Sci Eng Ethics. 2018 Oct;24(5):1409-1420. doi: 10.1007/s11948-017-9962-7.
Ribeiro, M.D., Vasconcelos, S.M.R. Retractions covered by Retraction Watch in the 2013-2015 period: prevalence for the most productive countries. Sci Eng Ethics. 2018 Feb;114:719-734. doi: 10.1007/s11192-017-2621-6.
Moylan EC, Kowalczuk MK. Why articles are retracted: a retrospective cross-sectional study of retraction notices at BioMed Central. BMJ Open. 2016 Nov 23;6(11):e012047. doi: 10.1136/bmjopen-2016-012047.
Haug CJ. Peer-Review Fraud--Hacking the Scientific Publication Process. N Engl J Med. 2015 Dec 17;373(25):2393-2395. doi: 10.1056/NEJMp1512330.
Rapani A, Lombardi T, Berton F, et al. Retracted publications and their citation in dental literature: A systematic review. Clin Exp Dent Res. 2020 Aug;6(4):383-390. doi: 10.1002/cre2.292.
Nair S, Yean C, Yoo J, et al. Reasons for article retraction in anesthesiology: a comprehensive analysis. Can J Anaesth. 2020 Jan;67(1):57-63. doi: 10.1007/s12630-019-01508-3.
Collier R. Shedding light on retractions. CMAJ. 2011 Apr;19;183(7):E385-386. doi: 10.1503/cmaj.109-3827.
Chevret S, Ferguson ND, Bellomo R. Are systematic reviews and meta-analyses still useful research? No. Intensive Care Med. 2018 Apr;44(4):515-517. doi: 10.1007/s00134-018-5066-3.
Ioannidis JPA, Klavans R, Boyack KW. Thousands of scientists publish a paper every five days. Nature. 2018 Sep;561(7722):167-169. doi: 10.1038/d41586-018-06185-8.
Gao J, Zhou T. Retractions: Stamp out fake peer review. Nature. 2017 May 31;546(7656):33. doi: 10.1038/546033a.
Arns M. Open access is tiring out peer reviewers. Nature. 2014 Nov 27;515(7528):467. doi: 10.1038/515467a.
Iwamoto, J, Sato, Y, Tanaka, K. et al. Prevention of hip fractures by exposure to sunlight and pharmacotherapy in patients with Alzheimer’s disease. Aging Clin Exp Res. 2009 Aug-Oct;21(4-5):277-281. doi: 10.1007/BF03324916. Retraction in: Aging Clin Exp Res. 2018 Jun 5;1.
Iwamoto J, Takeda T, Matsumoto H. Efficacy of oral bisphosphonates for preventing hip fracture in disabled patients with neurological diseases: a meta-analysis of randomized controlled trials among the Japanese population. Curr Med Res Opin. 2011 Jun;27(6):1141-1148. doi: 10.1185/03007995.2011.570747. Retraction in: Curr Med Res Opin. 2017 Jun;33(6):1181.
Iwamoto J, Matsumoto H, Takeda T. Efficacy of risedronate against hip fracture in patients with neurological diseases: a meta-analysis of randomized controlled trials. Curr Med Res Opin. 2008 May;24(5):1379-1384. doi: 10.1185/030079908x297321. Retraction in: Curr Med Res Opin. 2017 Apr 5;1.
Li L, Tian J, Tian H, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014 Sep;67(9):1001-1007. doi: 10.1016/j.jclinepi.2014.04.003.
Li XX, Zheng Y, Chen YL, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy. 2015 Apr;119(4):503-510. doi: 10.1016/j.healthpol.2014.09.002.
Hamilton DG. Continued Citation of Retracted Radiation Oncology Literature-Do We Have a Problem? Int J Radiat Oncol Biol Phys. 2019 Apr 1;103(5):1036-1042. doi: 10.1016/j.ijrobp.2018.11.014.
Moher D, Liberati A, Tetzlaff J, et al. PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. doi: 10.1136/bmj.b2535.
Page MJ, McKenzie JE, Bossuyt PM, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. MetaArXiv 2020, September 14. [cited 2020 Sep 15]. Available online: https://doi.org/10.31222/osf.io/jb4dx.
Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: A Measurement Tool to Assess Systematic Reviews. BMC Med Res Methodol. 2007 Feb 15;7:10. doi: 10.1186/1471-2288-7-10.
Bar-Ilan J, Halevi G. Post retraction citations in context: a case study. Scientometrics. 2017;113(1):547-565. doi: 10.1007/s11192-017-2242-0.
Yang KH, Chen YL, Li YP, et al. Members of the Lanzhou International Guideline Symposium. Editorial: can China master the guideline challenge? Health Res Policy Syst. 2013 Jan 9;11:1. doi: 10.1186/1478-4505-11-1.
Patel J. New COPE guidelines on publication process manipulation: why they matter. Res Integr Peer Rev. 2018 Nov 26;3:13. doi: 10.1186/s41073-018-0059-x.
Ge L, Tian J H, Li YN, et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J Clin Epidemiol. 2018 Jan;93:45-55. doi: 10.1016/j.jclinepi.2017.10.012.
Altman DG, Moher D. Declaration of transparency for each research article. BMJ. 2013 Aug 7;347:f4796. doi: 10.1136/bmj.f4796.
Faggion CM Jr. More detailed guidance on the inclusion/exclusion of retracted articles in systematic reviews is needed. J Clin Epidemiol. 2019 Dec;116:133-134. doi: 10.1016/j.jclinepi.2019.07.006.
Grossetta Nardini HK, Batten J, Funaro MC, et al. Librarians as methodological peer reviewers for systematic reviews: results of an online survey. Res Integr Peer Rev. 2019 Nov 27;4:23. doi: 10.1186/s41073-019-0083-5.
Chen YL, Yang KH, Marušic A, et al. A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. Ann Intern Med. 2017 Jan;166(2):128-132. doi: 10.7326/M16-1565.
Brouwers MCKK, Kerkvliet K, Spithoff K, AGREE Next Steps Consortium. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016 Mar 8;352:i1152. doi: 10.1136/bmj.i1152.

Table 1. Characteristic of retracted systematic reviews (N=150)
Categories	Number (Percentage)
Year of retraction
2015~2020	137 (91.3%)
2010~2014	11 (7.3%)
2005~2009	1 (0.7%)
2003~2004	1 (0.7%)
Country of first author
China	113 (75.3%)
USA	5 (3.3%)
Italy	5 (3.3%)
UK	4 (2.7%)
Iran	4 (2.7%)
Japan	3 (2.0%)
Brazil	3 (2.0%)
Korea	2 (1.3%)
Singapore	2 (1.3%)
Others*	10 (6.7%)
Organization
Hospital	106 (70.7%)
University	41 (27.3%)
Others	3 (2.0%)
Journal^†
Tumor Biol	23 (15.3%)
Mol Biol Rep	13 (8.7%)
Medicine	9 (6.0%)
Mol Neurobiol	9 (6.0%)
Plos One	6 (4.0%)
Eur J Med Res	5 (3.3%)
J Orthop Surg Res	5 (3.3%)
Others	80 (53.3%)
Impact Factor (IF) of journal
0＜IF≤3	85 (56.7%)
3＜IF≤5	22 (14.7%)
IF＞5	13 (8.7%)
Not indexed in SCI^‡	30 (20.0%)
Quartile of Journal Citation Report
Q1	32 (21.3%)
Q2	35 (23.3%)
Q3	39 (26.0%)
Q4	14 (9.3%)
Not indexed in SCI^‡	30 (20.0%)
Remarked as retracted article
Yes	148 (98.7%)
No	2 (1.3%)
Retraction notice
Published	148 (98.7%)
Unpublished	2 (1.3%)
Note * Others included Canada (n=1), Colombia (n=1), New Zealand (n=1), Sweden (n=1), Denmark (n=1), Malaysia (n=1), Argentina (n=1), Saudi Arabia (n=1), Nigeria (n=1), † Journal was presented in their abbreviations ‡ SCI: Science Citation Index

Table 2. Number of retractions listed by the issuer (N=150)
Categories	Number (Percentage)
Editor	27 (18.0%)
Author	16 (10.7%)
Publisher	10 (6.7%)
Journal	5 (3.3%)
Editor and author	8 (5.3%)
Editor and publisher	60 (40.0%)
Editor, author and publisher	8 (5.3%)
Unclear	9 (6.0%)
Not reported	7 (4.7%)

Table 3. Reasons of retraction（N=161）
Categories *	Number (Percentage)
Fraudulent peer-review	61 (37.9%)
Unreliable data	41 (25.5%)
Plagiarism	30 (18.6%)
Duplicate publication	9 (5.6%)
Authorship issues	8 (5.0%)
Others^†	7 (4.4%)
Not reported	4 (2.9%)
Data fabrication	1 (0.6%)
Note * Some retractions were due to multiple reasons ^†Others included accidental publication twice due to technical reasons of publishers (n=1), accidental publication in advance due to technical reasons of publishers (n=3), conflict of interest (n=2), concerns about third party involvement in editing submitted files (n=1).

Table 4. Detailed reasons of retraction due to unreliable data
Categories *	Number
Error in search strategy	4
Inadequate reporting of search strategy	3
Lack of key search items	1
Error in selection of included studies	18
Included studies that did not meet the inclusion criteria	9
Included duplicate studies	5
Included studies that were retracted	3
Included unpublished studies accessible only during peer review for another journal	1
Concerns in inclusion/exclusion criteria	1
Concerns in reasons for excluded studies	1
Error in data collection	10
Error in study design ^†	3
SE were extracted instead of SD	2
Data in some figures were not found in the primary article	1
Exact reason not reported	5
Error in data analysis	16
Incorrect methods of constructing forest plots	7
Incorrectly labelled figures	2
Incorrect analysis of publication bias	2
Incorrect subgroup analyses	1
Lack of sensitivity analyses	1
Lack of adequate analyses for some outcomes	2
Exact reason not reported	6
Error in data reporting ^‡	2
Error in result interpretation	4
Incorrect conclusions	3
Exact reason not reported	1
Others	7
Multiple errors in the methodology	5
Incorrect use of the reporting guidelines	2
Errors in reference list	2
Note：SE: Standard Error; SD: Standard Deviation. * Some retractions were due to multiple reasons. ^†Error in study design included for example inclusion of data from experimental groups in the study, or errors in number of study arms in the trial, number of participants reported or treatment duration. ‡ Data provided in the figure were not consistent with those shown in result or abstract sections.

Table 5. Time from publication to retraction by cause for retraction
Categories*	Time to retraction^†
Authorship issues	27.5 (6.0~50.0)
Fraudulent peer-review	16.0 (2.0~51.0)
Plagiarism	15.5 (0.0~51.0)
Unreliable data	7.0 (1.0~108.0)
Duplicate publication	14.0 (0.0~40.0)
Data fabrication	75^‡
Others	3.0 (0.0~53.0)
Not reported	2.5 (0.0~6.0)
Total	14.0 (0.0~108.0)
Note * Some retractions were due to multiple reasons ^†Data was showed by median (from minimum to maximum), months ‡ Only one SR was retracted due to data fabrication

Download PDF

Journal Publication

published 24 Jun, 2021

Read the published version in Journal of Clinical Epidemiology →

Version 1

posted

You are reading this latest preprint version

An Overview of Retraction Status and Reasons of Non-Cochrane Systematic Reviews in Medicine

Status:

Journal Publication

Version 1

Abstract

Figures

Contributions To The Literature

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Status:

Journal Publication

Version 1