Adequately conducted systematic reviews with meta-analyses are considered the highest level of evidence within evidence-based medicine [1, 2]. The number of systematic reviews and meta-analysis guiding clinical practice and decision making has been steadily increasing during the past four decades [3].
Despite their place at the top of the hierarchical research pyramid, conventional meta-analyses are still at risk of type I error (alpha) due to results reaching significance by chance and type II errors (beta) due to results not reaching significance even when an effect is present. The risk of these errors is generally accepted at consensus-based levels (typically 5% for type I and 10–20% for type II errors), but may increase beyond those levels due to publication bias, poor trial designs, data heterogeneity and poorly conducted or inadequately powered meta-analyses with multiple significance testing [1, 4, 5]. The investigated effect of a meta-analysis can reach significance even though the effect might be so small that it is not clinically relevant [6].
Several tools exist for controlling for non-random causes of type I and II errors as rigorously described by the Cochrane Handbook [2]. However, little emphasis has been put on mitigating the purely random causes of type I and II errors [7]. As an example, correction for multiplicity issues due to use of several outcomes has historically been underprioritised[5, 8]. As shown by Turner, Bird and Higgins in 2013, 70% of the meta-analyses reported in Cochrane Reviews had less than 50% power to detect even a relatively large 30% relative risk reduction [9]. Moreover, there is an increased risk of an exaggerated intervention benefit in small trials due to reporting bias or methodological flaws [10].
Pooling randomised clinical trials in a meta-analysis may introduce heterogeneity due to differences in design, interventional protocols, included participants, and outcomes [11, 12]. This heterogeneity can be an advantage as it may reflect the naturally occuring variation in clinical practice compared to trials with very strict or similar protocols, thereby mimicking real-world treatment [1]. In a meta-analytic setting, however, heterogeneity needs to be adequately examined and considered by Trial Sequential Analysis [13, 14].
Trial Sequential Analysis was developed to control the risk of type I and II errors in meta-analysis. It can be used to estimate the diversity-adjusted required information size (DARIS or the ‘meta-analytic sample size’) in random-effects meta-analysis, which serves to indicate whether the chosen power level was reached with the meta-analysis. If adequate power was not reached, DARIS may guide the scaling of future trials[14]. It also establishes when firm evidence is reached for or against a specific intervention. Further, Trial Sequential Analysis can establish futility boundaries and thus indicate when non-significant results are due to lack of intervention effect and not lack of power [13, 15].
The Trial Sequential Analysis is a sequential method using 𝜶-spending monitoring boundaries [16]. For dichotomous outcomes, one needs a proportion of participants with the outcome in the control group (Pc), an a priori chosen anticipated intervention effect (relative risk reduction (RRR)), alpha level, beta level, and the diversity (D2) of the trials included in the Trial Sequential Analysis for the calculation of the DARIS in a random-effects model meta-analysis [1, 15]. For continuous outcomes, one needs a minimally relevant clinical difference, the associated variance (squared standard deviation), the alpha level, beta level, and the diversity (D2) of the trials included in the Trial Sequential Analysis for the calculation of the diversity-adjusted required information size (DARIS or the ‘meta-analytic sample size’) in a random-effects model meta-analysis [17].
Furthermore, Trial Sequential Analysis can advantageously be used alongside the Grading of Recommendations Assessment, Development and Evaluation (GRADE). By calculating the DARIS, the reviewers are informed of the power of their analysis and can therefore grade imprecision in a more informative manner [18]. Also, the analysis can supply the reviewers with a trial sequential adjusted confidence interval to evaluate inconsistency alongside the heterogeneity [16].
To date, numerous systematic reviews have used Trial Sequential Analysis since it was first presented at the beginning of this millennium [19]. As with all methods, the Trial Sequential Analysis can be misused and misinterpreted. A rigorous process starting when writing the protocol through to the reporting phase of the results in the review is necessary. Predefined parameters such as alpha level, beta level (and power), relative risk reduction, minimally relevant clinical difference, and heterogeneity can largely affect the results of the analysis and should therefore be enclosed in pre-published or registered protocols prior to searching for literature for the systematic review. Failure to do so might ultimately alter the conclusion of the meta-analysis and thereby directly misguide clinical practice [5, 16, 20].
Objective
In this review, we aim to systematically evaluate the use of Trial Sequential Analysis in existing systematic reviews and meta-analyses. Specifically, we seek to evaluate how the authors prepared and conducted their Trial Sequential Analysis, and interpreted their results in the assessment of imprecision in the obtained meta-analytic results. We want to identify the most common major mistakes and errors in order to update recommendations for a more proper use of the Trial Sequential Analysis Programme in future systematic reviews.