Validation of a Search Strategy for Randomized Clinical Trials Related to Periodontitis

doi:10.21203/rs.3.rs-235025/v1

Download PDF

Research Article

Validation of a Search Strategy for Randomized Clinical Trials Related to Periodontitis

https://doi.org/10.21203/rs.3.rs-235025/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 03 Apr, 2021

Read the published version in Archives of Public Health →

You are reading this latest preprint version

Background: Systematic reviews, considered the gold standard for the assessment of scientific evidence, may present conflicting findings for the same clinical issue, and such dissent may be justified, for example, by the forms of elaboration of the electronic search strategy. The aim of this paper is to validate a search strategy to identify randomized clinical trials related to periodontitis.

Results: The search strategy under evaluation compared to the gold-standard showed a sensitivity of 93.2% (_95%CI, 83.8-97.3), specificity of 99.9% (_95%CI 99.8-99.9) and a precision of 77.5% (_95%CI, 66.48-85.63). In addition, the needed number of articles to read was 1.3.

Conclusion: According to the proposed methodological approach, the search strategy under evaluation performed well in the identification of randomized clinical trials related to periodontitis.

Health Economics & Outcomes Research

Health Policy

Methodological Studies

Research Design

Systematic Review

Periodontitis

Sensitivity

In the last decades, the systematic review, defined as a type of study that synthesizes the scientific evidence existing in the literature, has provided ample space in the field of health, including dentistry. It is estimated that in 2010, 11 systematic reviews were published per day[1]. Most likely, this number should be even higher today. Regarding dentistry, it is estimated that approximately 1,000 (thousand) systematic reviews were published in 2017, according to the MEDLINE using PubMed platform (MEDLINE /PubMed).

Although it is considered the gold standard for the assessment of scientific evidence, systematic reviews of randomized clinical trials often present conflicting findings for the same issue[2]. Considering the reproducibility of this design, the aforementioned conflict between the findings does not seem justifiable, a priori.

However, a closer inspection of the stages of the systematic review protocol may explain this phenomenon. One of them concerns the elaboration of the electronic search strategy, which can be simplified as a “specific algorithm". The construction of this syntax occurs through index terms/synonyms and symbols to retrieve articles that report evidence about a particular research question in an electronic bibliographic database[3, 4].

However, it is highlighted that an electronic search strategy holds strong subjectivity in itself, to the point that scholars argue that different researchers invariably tend to build different strategies on the same object of interest[5, 6]. For example, a quick search in the periodontal literature on strategies aiming to identify periodontitis-related studies can identify different ways of elaborating strategies with different sets of references retrieved with varying accuracy. It can impact directly in the quality of the systematic search and its results as well as the time required for its execution[7–10].

Therefore, strengthening the means to validate search strategies, estimating quantitative indicators of their performance, such as the strategy sensitivity and specificity, is a reasonable way to increase the quality of the identification of studies and, consequently, of the findings of systematic reviews. This article aims to validate a search strategy for the identification of randomized clinical trials related to periodontitis.

2.1 Study design

This is a methodological study for the validation of a search strategy to identify randomized clinical trials related to periodontitis on MEDLINE /PubMed.

2.2 Procedures for identification of the gold-standard set and the retrieved articles using the search strategy under evaluation

Stage 1 – Application of the Cochrane Collaboration’s methodological filter

Initially, the methodological filter was applied to identify randomized clinical trials, which was validated by the Cochrane Collaboration (Cochrane Highly Sensitive Search Strategy – HSSS) and has high sensitivity and precision for MEDLINE /PubMed [11], as follows:

#1 randomized controlled trial[Publication Type]

#2 controlled clinical trial[Publication Type]

#3 randomized [Title/Abstract]

#4 placebo[Title/Abstract]

#5 clinical trials as topic[MeSH Terms]

#6 randomly[Title/Abstract]

#7 trial[Title]

#8 #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7

#9 animals [mh] NOT humans [mh]

#10 #8 NOT #9

A chronological filter was also used from January 01 to March 31, 2018.

Stage 2 – Application of the eligibility criteria

Among the articles retrieved concerning to randomized clinical trials, those related only to periodontitis were identified. In addition, exclusion criteria comprised of studies involving animal models and reviews of randomized clinical trials. This stage of reading titles and abstracts was performed by two authors (SSC and AOL) and confirmed by a more experienced periodontist (ISGF), in case of disagreement. When necessary, full-text versions were evaluated.

Thus, after performing the above stages, a set of references that composed the gold-standard articles was selected: randomized clinical trials, related to periodontitis.

Stage 3 – Definition of the search statement related to periodontitis

The search statement for identifying the condition of interest (periodontitis) was developed using the tool "advanced search" in MEDLINE/PubMed, as follows: 1) controlled vocabulary terms related to periodontitis were identified; 2) a periodontist and general dentist (ISGF and SSC) identified the main keywords and their derivations; 3) the retrieved articles were carefully analyzed, and the terms that were associated with studies not related to periodontitis were discarded, for example the term "gingivitis"; and 4) the procedure was repeated until the strategy was considered adequate, using the Peer Review of Electronic Search Strategies (PRESS, Supplemental File- Appendix A)[4] checklist with the assistance of an experienced librarian (VSSS).

Finally, the following search statement for identifying periodontitis was developed:

#1 “Periodontitis” [Title/Abstract]

#2 “Periodontitis” [MeSH Terms]

#3 “Disease, Periodontal” [Title/Abstract]

#4 “Disease, Periodontal” [MeSH Terms]

#5 “Diseases, Periodontal” [Title/Abstract]

#6 “Diseases, Periodontal” [MeSH Terms]

#7 “Periodontal Disease” [Title/Abstract]

#8 “Periodontal Disease” [MeSH Terms]

#9 “Parodontosis” [Title/Abstract]

#10 “Parodontosis” [MeSH Terms]

#11 “Parodontoses” [Title/Abstract]

#12 “Parodontoses” [MeSH Terms]

#13 “Pyorrhea Alveolaris” [Title/Abstract]

#14 “Pyorrhea Alveolaris” [MeSH Terms]

#15 #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14

Stage 4 – Application of the search strategy under evaluation

Stage 4 comprised the combination of the search statement developed (Stage 3) with the abovementioned methodological filter (Stage 1) and subsequent application in MEDLINE/PubMed for the retrieval of randomized clinical trials related to periodontitis, defined as the search strategy under evaluation.

2.3 Data analysis procedures

The proposed search strategy was evaluated by analyzing the extent to which it retrieved the studies in the gold-standard articles set, and the sensitivity, specificity, and precision of this strategy were calculated according to the following formulas:

The 95% confidence intervals (_95%CI) of the strategy performance values were calculated for each estimated measurement.

At the end of Stage 1, a total of 18,056 articles were retrieved according to the Cochrane Highly Sensitive Search Strategy (HSSS) methodological filter combined with the chronological filter. Of these, 178 were conducted using an animal model, 18 were letters to the editor, seven were scoping reviews, 11 were reviews of reviews, and 715 were systematic reviews of clinical trials (Fig. 1). As for the other studies, 17,127 clinical trials were conducted on humans, of which only 59 were related to periodontitis, comprising of gold-standard articles set obtained in Stage 2.

After the evaluation of the search strategy with PRESS, a search statement was obtained, employing controlled vocabulary terms, title and abstract filters, connected by Boolean operators, combined with a chronological filter. Thus, according to Fig. 1, at the end of the Stage 3, 3.843 articles were obtained. By using the final strategy under evaluation, 72 randomized clinical trials related to periodontitis were retrieved at the Stage 4. It is noteworthy, however, that from the total of retrieved articles, one study (1.4%) did not belong to the set of randomized clinical trials identified with the Cochrane Collaboration methodological filter, although this was of interest of the bibliographical search. Therefore, the study was excluded from the analysis, being classified as an outlier. The PubMed search history performed in November, 16th, 2018, can be seen in Supplemental File- Appendix B.

Thereby, among the 71 retrieved articles, 65 were related to periodontitis. However, when only randomized clinical trials were selected, 55 articles were included in the performance analysis of the search strategy under evaluation. The findings of the performance analysis showed that the final search strategy had a sensitivity of 93.2% (_95%CI 83.8–97.3), specificity of 99.9% (_95%CI 99.8–99.9), precision of 77.5% (_95%CI 66.5–85.7) and number needed to read of 1.3 studies (Table 1).

Table 1

Sensitivity, specificity, and precision, their respective 95% confidence intervals (_95%CI), and number needed to read for the comparison between the search strategy under evaluation and the set of gold-standard articles.
Indicator	Value	_95%IC
Sensitivity (%)	93.2	83.8–97.3
Specificity (%)	99.9	99.8–99.9
Precision (%)	77.5	66.5–85.6
NNR * (absolute value)	1.3	-
* Number needed to read (NNR) = 1/ precision

4.1 Main results

The main findings of the present study suggest that the search strategy under evaluation, used to identify randomized clinical trials related to periodontitis in MEDLINE/PubMed, presented a good performance when compared to the gold-standard strategy, based on validity indicators - sensitivity, specificity, precision and number needed to read.

4.2 Comparison with other types of study about the topic

Studies that carried out validation of a strategy to identify randomized clinical trials specifically related to periodontitis were not found. However, there are investigations that validated strategies to identify studies related to other areas. In addition, some studies[11, 12] carried out validation of search strategies employing a method similar to the one presented here, with a gold-standard based also on the HSSS filter of the Cochrane Collaboration.

With a similar goal, a strategy to obtain a sensitive search about randomized clinical trials on diet and nutrition was developed[12]. The gold-standard of the aforementioned study was based on the HSSS, and 298 systematic reviews of the Cochrane Collaboration were employed, rather than original articles as in the present study. Also, it was observed that the best strategy of the study on diet and nutrition showed sensitivity of 88.6%, close to the indicator estimated in this study 93.2%.

Similarly, another investigation, in an attempt to recognize articles related to adverse effects to surgery[13], the sensitivity of two search strategies was estimated. In their best strategy, sensitivity measurements of 93% for MEDLINE and 95% for Embase were obtained. Thus, indicators similar to those estimated in this investigation were also observed.

4.3 Strengths

The search strategy proposed in this study can be well applied to the elaboration of systematic reviews of randomized clinical trials related to periodontitis, since it will promote a reduction in the operational time of an important stage of this type of secondary study - the identification of publications to be included[14–17].

According to one of the performance indicators evaluated, the number needed to read[18, 19], for every 13 articles identified, 10 would likely be of interest to the researchers, conferring a higher operational speed for this stage. Therefore, the elaborated strategy can be useful for reducing time and human resources for the elaboration of bibliographic researches.

Consequently, there can be a considerable cost reduction for the performance of systematic reviews related to periodontitis, which are commonly useful for the synthesis of evidence[5, 7, 20]. In addition, there is an increase in the validity of the review since the strategy developed showed high sensitivity in the identification of studies on the topic of interest.

It should be noted that the adoption of the gold-standard search strategy was based on two pillars. The first one, which has recognized validity, since a filter developed by the Cochrane Collaboration (HSSS)[11] was used to identify all randomized clinical trials in the period determined in this investigation. The second pillar concerns to the construction of a search statement, specific for periodontitis, developed independently by two researchers with experience and qualification in the field of knowledge, improving the reliability of the identification of relevant studies.

It is also noteworthy that this search statement was evaluated by a professional with a background in Librarianship, according to the recommendations of PRESS, aiming to improve the quality of the research in the database[21, 22].

4.4 Limitations

In this study, the consultation only to the platform MEDLINE/PubMed can represent a limitation since it restricts the extrapolation of the good performance of the strategy developed to other databases[17]. However, the adaptation of MEDLINE/PubMed search syntax to the main electronic databases, such as Embase or Web of Science, is a usual procedure, it does not require great effort by the researchers[5].

Another limitation refers to the chronological filter applied to the Cochrane strategy for the identification of randomized clinical trials, which included the three initial months of the year 2018. This decision provided a convenience sample of the studies published that year, instead of a probabilistic sample that would be more desirable to increase the representativeness of the included studies in the referred year.

In this sense, the next steps for this investigation include the use of all randomized clinical trials over a year to minimize the potential problem of generalized restriction. Another improvement would be an evaluation of the quality of the investigations retrieved using the evaluated strategy, since this step was not performed in this study.

The developed search strategy exhibited good performance for the adequate retrieval of randomized clinical trials related to periodontitis. Additionally, it can be a useful tool in reducing time and cost for researchers.

HSSS - Highly Sensitive Search Strategy

NNR - Number needed to read

7.1 Ethics approval and consent to participate:

Not applicable

7.2 Consent for publication:

Not applicable

7.3 Availability of data and materials:

The datasets during and/or analysed during the current study available from the corresponding author on reasonable request.

7.4 Competing interests:

The authors declare that they have no competing interests.

7.5 Funding:

None.

7.6 Authors' contributions:

AOL and SSC conceived the study and analyzed data. All authors wrote the paper, read, and approved the final manuscript

7.7 Acknowledgements:

Not applicable

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.
Ioannidis JP. Meta-research: The art of getting it wrong. Res Synth Methods. 2010; 1(3-4):169-184.
Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009; 62(9): 944-952.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40-46.
Jenkins M. Evaluation of methodological search filters--a review. Health Info Libr J. 2004;21(3):148-163.
Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Res Synth Methods. 2018;9(3):408-416.
Faggion CM, Atieh MA, Park S. Search strategies in systematic reviews in periodontology and implant dentistry. J Clin Periodontol. 2013;40(9):883-888.
Natto ZS, Abu Ahmad RH, Alsharif LT, et al. Chronic Periodontitis Case Definitions and Confounders in Periodontal Research: A Systematic Assessment. Biomed Res Int. 2018; 2018: 4578782.
Manrique-Corredor EJ, Orozco-Beltran D, Lopez-Pineda A, Quesada JA, Gil-Guillen VF, Carratala-Munuera C. Maternal periodontitis and preterm birth: Systematic review and meta-analysis. Community Dent Oral Epidemiol. 2019; 47(3):243-251.
Peddis N, Musu D, Ideo F, Rossi-Fedele G, Cotti E. Interaction of biologic therapy with apical periodontitis and periodontitis: A systematic review. Aust Dent J. 2019; 64(2):122-134.
Glanville JM, Lefebvre C, Miles JN, Camosso-Stefinovic J. How to identify randomized controlled trials in MEDLINE: ten years on. J Med Libr Assoc. 2006;94(2):130-136.
Durão S, Kredo T, Volmink J. Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method. J Clin Epidemiol. 2015;68(6):610-616.
Golder S, Wright K, Loke YK. The development of search filters for adverse effects of surgical interventions in medline and Embase. Health Info Libr J. 2018;35(2):121-129.
Budhram D, Navarro-Ruan T, Haynes RB. The efficiency of database searches for creating systematic reviews was improved by search filters. J Clin Epidemiol. 2018; 95:1-6.
Lefebvre C, Glanville J, Beale S, et al. Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study. Health Technol Assess. 2017;21(69):1-148.
Shariff SZ, Sontrop JM, Haynes RB, et al. Impact of PubMed search filters on the retrieval of evidence by physicians. CMAJ. 2012;184(3): E184-190.
Pillastrini P, Vanti C, Curti S, et al. Using PubMed search strings for efficient retrieval of manual therapy research literature. J Manipulative Physiol Ther. 2015;38(2):159-166.
Bachmann LM, Coray R, Estermann P, Ter Riet G. Identifying diagnostic studies in MEDLINE: reducing the number needed to read. J Am Med Inform Assoc. 2002;9(6):653-658.
Olaussen A, Semple W, Oteir A, Todd P, Williams B. Paramedic literature search filters: optimised for clinicians and academics. BMC Med Inform Decis Mak. 2017;17(1):146.
Rollin L, Darmoni S, Caillard JF, Gehanno JF. Searching for high-quality articles about intervention studies in occupational health--what is really missed when using only the Medline database? Scand J Work Environ Health. 2010;36(6):484-487.
Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017; 358: j4008.
Spry C, Mierzwinski-Urban M. The impact of the peer review of literature search strategies in support of rapid review reports. Res Synth Methods. 2018;9(4):521-526.

supplementalfile.docx

Download PDF

Journal Publication

published 03 Apr, 2021

Read the published version in Archives of Public Health →

Editorial decision: Accept
08 Mar, 2021
Reviewers invited by journal
11 Feb, 2021
Reviews received at journal
11 Feb, 2021
First submitted to journal
11 Feb, 2021
Editor assigned by journal
10 Feb, 2021

You are reading this latest preprint version

Validation of a Search Strategy for Randomized Clinical Trials Related to Periodontitis

Status:

Journal Publication

Version 1

Abstract

Figures

1 Background

2 Methods

3 Results

4 Discussion

4.1 Main results

4.2 Comparison with other types of study about the topic

4.3 Strengths

4.4 Limitations

5 Conclusions

6 List Of Abbreviations

7 DECLARATIONS

8 REFERENCES

Supplementary Files

Status:

Journal Publication

Version 1