The results of this study provide trends and a critical appraisal of the methodological quality, risk of bias and reporting characteristics of the most cited systematic reviews in dentistry, and showed a critically low methodological quality, a high risk of bias and a low adherence to some PRISMA items of the 50-top cited systematic reviews in dentistry. This finding suggests that study quality and citation rates are not associated. Whereas this is not surprising, the popularity of dental studies with questionable quality and lack of transparency in reporting may rise problems for the decision-making process in clinical practice.
The scientific community has seen considerable advances in standards, methods, and systems for conducting and reporting systematic reviews10,16. However, those efforts are still not fully covered by researchers, and the readers and consumers of systematic reviews should be aware of those falls. Systematic reviews have been consistently published in dentistry over the past year. From 2021 to 2022 more than 4,900 systematic reviews were published under the systematic review filter and dentistry MeSH term, only in Pubmed. Bibliometric analysis has been used as a practical tool for measuring the scholarly impact of a topic, while characteristics of most cited papers can be assessed. Many authors have used these tools to search for trends in dentistry, assess if a topic is under-researched and to retrieve and examine which are the most distinguished studies2,9,17,18. Additionally, more cited papers are used by scholars and clinicians for content and may be even considered by patients looking for “the best evidence available”.
We have included the most cited papers in Scopus as the leading citation metrics. The top cited list is dynamic, as scientific interests and research tendencies evolve over the years9. In the research field, a publication cited 100 or more times is considered as a “classic” article19. The most cited systematic reviews in dentistry varied from 252 up to 980 citations. These differences in the number of citations between the first and last 50 ranked systematic reviews may be attributed to the rapidly evolving nature of science in conjunction with the preference of authors to cite review articles. Although citation is time affected, the most cited systematic reviews in dentistry were published between 2002 and 2015. Still, it is worth noting that there is a tendency of authors to cite a publication that is already abundantly cited rather than re-assessing its relevance and quality. In this sense, it is mandatory to assess the quality of the re-cited systematic reviews, as low-quality systematic reviews may pose a threat to scholars, clinicians and patients.
Among several assessment tools for systematic reviews, the most researched and most often used tool to assess the quality of systematic reviews of interventions is the AMSTAR-213. AMSTAR-2 does not give a score, but rather rate the overall confidence of the systematic review. Our findings showed an overall rating of the confidence in the methods was critically low for most systematic reviews, corroborating previous study on this topic20. This means that these reviews had at least two critical flaws in the AMSTAR-2 critical domains. Moreover, the risk of bias assessment of 50-top cited SR through the ROBIS tool showed a high risk of bias or concerns for several domains, including domain 1 and 2 which are related to the methodological aspects. This finding is in line with the methodological quality observed in the AMSTAR-2 instrument. A previous study has shown a strong correlation between the ratings for AMSTAR-2 questions and the overall domain rating in ROBIS21. The critical AMSTAR-2 domain which received large numbers of negative answers (45 studies) was the item 2 (report of the review contains an explicit statement that the review methods were established prior to the conduct of the review and the report justify any significant deviations from the protocol). As there is a positive influence of previously registering a protocol in the final report quality of systematic reviews22, the lack of a previously protocol registration could have influenced those suboptimal quality scores. Prior registration of systematic reviews is important as it establishes strict criteria to adhere and thereby avoid bias23.
An important tool to guide SR reporting is the PRISMA Statement. This reporting guideline ensures that every step of the systematic review is clearly reported24. There is a positive influence of previously registering a protocol in the final report quality of systematic reviews22. Although PRISMA Statement was first published in 2009, it was replacing QUORUM, which was already a tool to address reporting of meta-analysis published by an international group. This means that even the included top-cited publications that were published before 2009 could have used QUORUM to ensure better reporting25. From the fifty papers assessed, several items of the PRISMA statement showed low adherence, such as protocol and registration, methods for risk of bias of included studies, methods for effect of risk of bias, results of risk of bias of included studies, results of risk of bias across studies, and funding. Those findings raise some questions about whether authors are unaware of the statement or authors become aware only when the journal asks for the mandatory submission of a checklist.
Factors which influence the counts of citation of a publication include recitation of highly cited articles, author reputation in a specific field, and preference of a specific (reputable) journal. The most highly cited systematic reviews included in this study were published in journals with higher impact factors in the field of dentistry, as previously shown9. Journal Impact Factor (JIF), in a given year, is the average number of citations received per paper published in that journal during the preceding 2 years. Journals employ several methods to increase its impact factor, including publishing more of review articles which are generally cited more than research reports26. Researchers look for references in higher impact factor journals and with higher citation rates to support their decisions based on ‘high quality and credible studies’, which seem to be quantitative and objective indicators directly related to published science. However, impact factor is not a measure of quality of research. As well as it is questionable to judge the quality and validity of a publication based on citation counts. Further, it could be expected that open access papers could receive more citations27, however, more than half of included studies were not published as open access.
The present study has some strength. This is the first study to address the methodological quality of 50-top cited systematic reviews in dentistry and highlighted that study quality and citations rate is not associated. It did not restrict the selection criteria of SR based on the study design. However, it has some limitations, including the bias associated with the drawbacks of the databases used, the biased citing, including self-citation or negative citation. Another limitation includes the effect of time, meaning that such methods favor older studies and omits or undervalues the important articles from the last 10 years. Furthermore, the AMSTAR-2 tool was designed to evaluate systematic reviews of intervention studies, and in our study the design of the primary studies included in the systematic reviews varied. Still, systematic reviews of interventions included in our analysis presented high risk of bias. The only high quality study and moderate quality studies included in this analysis are amongst the most recently published (2012 to 2015), which reflects a trend to a higher use of reporting guidelines and better research practices. Yet, many recent publications were also rated as critically low, meaning that the importance of transparently conducting and adequate reporting studies is not being given adequate importance.