Principal findings of the study
This cross-sectional study showed that in 35.7% (35/98) of reviews of orthodontic interventions seeking of adverse effects was defined as an objective. In 85.7% (84/98) of these reviews, findings related to adverse effects of interventions were sought and in 84.7% (83/98) the reviewers reported on these findings. In more than 90% (89/98) of included systematic reviews, the reviewers discussed (weighed) potential adverse effects of interventions somewhere in the review. Five types of adverse effects accounted for 83.1% (162/195) of adverse effects that were sought and reported in the eligible reviews.
Comparisons with other studies
The proportion of included reviews that defined seeking of adverse effects as a research objective was low, i.e., 35.7% (35/98) in both Cochrane and non-Cochrane systematic reviews (Table 4). Assessing potential adverse effects of interventions is considered a mandatory item when setting the research question for Cochrane intervention reviews [1]. Not defining seeking of adverse effects as a research objective can mislead end-users of systematic reviews. Authors therefore need to include this item in their research objectives and editors and peer reviewers should verify its implementation.
The proportions of reviews that reported findings related to adverse effects of interventions were higher in this sample of orthodontic reviews (84.7% (83/98) compared with gastroenterology reviews (66.7% (52/78) [18], Cochrane reviews of interventions (75.6% (59/78), and Database of Abstracts of Reviews of Effects (DAREs) reviews (48.1% (38/79) [38]. Explanations for these higher proportions could be: (1) the time period of inclusion of reviews (2) the research design and type of interventions of the studies included in the reviews (3) the field of research. Orthodontic research could be more focused on assessing adverse effects of interventions than other fields, because this assessment is an integral part of routine clinical practice. For example, assessing adverse effects such as undesired treatment results and relapse and stability are part of everyday problems in orthodontic practice and accounted for 40.5% (79/195) of adverse effects sought and reported in this sample of reviews of orthodontic interventions (Table 8).
Strengths & Limitations
This cross-sectional study has the following strengths: (1) scoping searches were conducted to identify knowledge gaps, (2) pilot studies were conducted to calibrate researchers and fine-tune research questions and methodology, (3) a protocol was developed and published a priori [6], and (4) all raw data were included with this manuscript or recorded in Open Science Framework (https://osf.io/ka7mp/). This study also has limitations. First, the findings of this cross-sectional study are expected to be better than those reported in the orthodontic literature as a whole, because we assessed reviews published in the 5 leading orthodontic journals and those listed in the Cochrane Database of Systematic Reviews. Second, the risk of selective (non) reporting bias regarding adverse effects in the eligible reviews. Third, only reviews published in a pre-established period (August 1 2009 until July 31 2021) were assessed instead of having included reviews published during a larger time span. However, we chose this period, because the inception date coincides with the launch of the PRISMA statement [31,32] and represents the current knowledge status on our research questions.
Implications and future research
Several of our findings seem promising at a first glance. For example, the proportion of reviews that sought and reported adverse effects was relatively high, i.e., (84.7% (83/98), but a variety of issues has to be considered when interpreting this finding. First, this proportion only refers to whether or not reviewers implemented this item, but not how. For example, the reviewers could have reported on just one or a selection of all adverse effects assessed and reported in the eligible studies for their reviews. Second, this proportion also does not give any information on the magnitude, and duration of adverse effects nor on the time points for assessing them. Third, we do not know whether all adverse effects were indeed sought and reported as originally planned in the registered protocols of the included reviews. For example, Parsons et al. [39] showed that this was not the case in their sample of systematic reviews of health care interventions. In 35% (51/146) of these reviews they found discrepancies between what was planned in the protocol as registered in PROSPERO and what was reported on adverse effects in the final published reviews. Fourth, a wide body of evidence has shown that adverse events were often assessed inconsistently and reported inadequately in clinical trials and that most results on these events were not available in public sources [8,40–42]. If these limitations also apply to the clinical trials that fed the reviews of this study one should further question the validity of the findings on adverse effect of systematic reviews of orthodontic interventions.
Strategies to improve the validity of what is reported on adverse effects of orthodontic interventions in systematic reviews include developing tailored core outcome sets on these effects [43] as well as guidelines for assessing and reporting them in both primary research and systematic reviews. Additional strategies on synthesizing adverse effects in systematic reviews at multiple levels were published in a recent paper by Qureshi et al. [19]. By implementing such strategies progress on the assessing and reporting of adverse effects of orthodontic interventions in both primary studies and systematic reviews can be made.
In conclusion the promising findings of this study should be interpreted with caution by its end users, because they could be jeopardized by numerous uncertainties. Much research is ahead to create valid and usable knowledge on adverse effects of orthodontic interventions involving a wide body of stakeholders.