Could the Trustworthiness in RAndomized Controlled Trials (TRACT) checklist be trusted?

doi:10.21203/rs.3.rs-2823793/v4

Randomized controlled trials (RCTs) are considered the gold standard study design in clinical effectiveness research, albeit some RCTs have been found to be fraudulent post-publication. The Trustworthiness in RAndomized Controlled Trials (TRACT) checklist examines RCT integrity through an assessment covering seven domains. While the authors claim that it can identify problematic trials, it has never been validated. A critical appraisal of the checklist highlights major failures in terms of health measurement methodology, including the arbitrary nature of items chosen. We applied the checklist to 16 RCTs coauthored by the checklist developer, and identified potential violations related to retrospective registration, ethical implications, authorship, methodology, implausible timeframes, zero participants to follow-up, implausible baseline characteristics, and excessive inter-group within trial as well as inter-trial similarities. In addition, we identified inconsistencies between the trials’ registration and data in the final published manuscripts. If we uncritically assume that the checklist is valid, then a formal institutional investigation into the trial portfolio its developer is warranted; especially targeting these 16 trials objectively shown to be problematic. The checklist should be considered as neither investigative nor conclusive of potential problems with RCTs until it is further refined and validated.

Sexual & Reproductive Medicine

Research integrity

Randomized trial

Trustworthiness

Trustworthiness in RAndomized Controlled Trials (TRACT) is a recent checklist for recognizing potential integrity issues in randomized controlled trials (RCTs) [1]. RCTs assessing the efficacy and effectiveness of interventions are an essential ingredient of evidence-based medicine [2, 3]. They are, however, also susceptible to research misconduct, which can endanger patient care by jeopardizing the evidence [4-6].

Details on how the TRACT checklist was created, available in its preprint [1], show that it was developed using a four-stage methodology, which included defining the scope, reviewing the evidence, piloting and fine-tuning the checklist, and holding a consensus process. This was followed by a Delphi process, where stakeholders assessed and discussed the checklist and offered improvements. The TRACT checklist proposes seven domains (governance, author group, plausibility of intervention use, timeframe, drop-out rates, baseline characteristics, and results) and seeks responses to two or three signaling questions in each domain to guide editors, reviewers, and researchers in identifying possible issues in RCTs. A score of significant concerns for any domain is meant to raise a cautionary flag for possible trustworthiness issues that require further investigation. According to its authors, the checklist may be used as a screening tool to triage RCTs for their risk of being based on fabricated data [1]. However, to the best of our knowledge, this tool has never been validated and it is unclear as to what extent it is prone to produce false positive or false negative findings.

The purpose of this study is to critically appraise the tool and review its face validity by using a methodologically sound benchmarking strategy.

TRACT checklist

We proposed that a legitimate validation of the TRACT checklist is to apply it to RCTs coauthored by its developers since their trials should serve as a benchmark for trustworthiness. Using a group of their trials, we utilized the seven domains of the TRACT checklist and applied two or three signaling questions to each domain, with responses of: no concerns, some concerns/no information, or substantial concerns. The signaling questions addressed trial registration, protocol availability, ethical approval, author affiliation and contributions, intervention rationale, feasibility and consistency, trial duration, dropout rates and causes, baseline characteristics, and outcome reporting, measurement, and analysis.

Selection and assessment of RCTs

We searched the public domain (PubPeer) for RCTs containing comments or concerns about TRACT checklist coauthors to serve as a starting point. We also asked colleagues to identify additional RCTs co-authored by TRACT’s coauthors which they have potential concerns about. Since the majority (n = 16) of the identified RCTs were co-authored by the TRACT checklist lead author, we used his trials for further examination of their trustworthiness using the TRACT checklist. Figure 1 summarizes the selection process of RCTs.

Application and synthesis of TRACT checklist

Two reviewers (AS and HA) applied the TRACT checklist to each included RCT and recorded their findings on the standardized TRACT checklist form (Appendix 1). In the pilot phase, the two reviewers applied the TRACT checklist on five RCTs with PubPeer comments and extracted the findings. The reviewers met to discuss any conflicts and agreed on the way forward for the rest of the trials being investigated. The rest of the trials were then evaluated by two independent, blinded researchers with disagreements resolved through discussion or consultation with a third reviewer (AR). Figure 2 shows the extracted data, including domains, and queries from the TRACT checklist.

Evidence synthesis

We identified RCTs with some or significant concerns regarding each domain and signaling set of questions. Based on the number and severity of concerns across domains, we also categorized each RCT as having a low, moderate, or high risk of reliability issues. RCTs with low risk had no major concerns and fewer than three concerns; moderate risk was assigned if there was one major concern or three to five concerns; and high risk was assigned if there were more than one major concern or more than five concerns. We narratively synthesized the data and presented them in tables and figures.

Sub-analyses

We undertook additional analyses to determine the effect of TRACT-positive concerns on the outcomes of RCTs published by a particular center. We evaluated whether there was evidence of additional potential integrity violations in RCTs. Finally, we examined these implications regarding the reliability of evidence from these RCTs or, alternatively, the reliability of the TRACT as a valid instrument.

Critical appraisal of the checklist items

We reviewed the TRACT checklist domains and items in detail. While the seven domains covered by the checklist are applicable to RCTs, we believe that there may be additional trustworthiness issues that are not currently addressed by the tool. For example, while governance and authorship issues may be a source of bias, it does not account for all potential concerns or trial-specific nuances. Further, we felt that some of the items on the checklist are arbitrary or open to interpretation, such as ethical ramifications or unrealistic deadlines, as they require detailed background information like patient flow numbers at the time the trial was conducted or number of procedures per year conducted in the trial centers. Other items may not be applicable or pertinent to certain types of clinical trials, such as those involving multiple centers or countries. Thus, relying solely on the TRACT checklist may not provide an accurate assessment of an RCT’s trustworthiness. As a result, the use of the TRACT checklist alone may not be sufficient to ensure the trustworthiness of RCTs, and additional measures may be required to address these limitations. Future research should consider the development of more robust tools to provide a comprehensive RCT integrity evaluation.

TRACT checklist evidence synthesis

Figure 1 shows the validation steps for TRACT checklist and evidence synthesis. In the pilot phase, searching PubPeer resulted in five RCTs (coauthored by the TRACT lead developer) with comments that have potential TRACT-positive concerns. Seeking suggestions from colleagues resulted in the identification of 11 additional trials by the same author with potential TRACT-positive concerns. In addition, we identified other RCTs with PubPeer comments coauthored by the lead developer and three other co-authors of the TRACT checklist. Sticking to the inclusion criterion (RCTs included should have a common coauthor) resulted in a total of 16 RCTs with several potential violations according to the TRACT checklist (Figure 2).

Out of the 16 trials, there were concerns related to possible retrospective registration, ethical implications, and authorship. Additionally, there were concerns related to methodology, implausible timeframes, zero participants to follow-up, implausible baseline characteristics, and excessive inter-group and/or inter-trial similarities. Furthermore, some trials showed conflicting information regarding outcomes and changes in the primary outcome from registration to publication (i.e., high risk of subjective outcome reporting). In total, we identified 71 potential concerns across the 16 trials (55 major and 16 minor concerns), apart from checklist item #2c, which was virtually violated by all captured trials.

‘Prolapse 2015’ [7], ‘IVF ET 2018’ [8], and ‘STOP 2023’ [9] had the most concerns followed by ‘Metformin 2021’ [10] and ‘Pessary 2022’ [11], all of which were classified as high-risk. ‘IUI in cervical factor 2007’ [12] was the only trial classified as low risk, while no trial was classified as moderate risk. The TRACT lead developer coauthored 11 publications in the first month of 2023 alone [13-23], leading us to consider the item “large number of RCTs published in a short time frame by one author/in one institute” to be fulfilled in all reviewed trials.

A total of 9 trials (56.25%) showed a discrepancy in reported findings between the published abstract and the final article [7-10, 24-28]. Four pairs of them exhibited inter-trial similarities (50%) [8, 10, 24, 26, 29-32]. On contrary, four trials demonstrated inter-group dissimilarities that are unlikely to be attributed to chance (25.00%) [8, 25, 27, 30]. Three trials (18.75%) reported illogical embryological and/or clinical data [8, 10, 33]. Three trials (18.75%) had a discrepancy in reported outcomes between the published abstract and the final article [9, 10, 28]. Retrospective change in the primary outcome in the trial registry was also noted [33].

Tables 1 to 5 show the number and type of concerns identified in all trials. Therefore, the reviewed RCTs have multiple violations of the TRACT checklist as it currently stands, indicating a high risk, and the number of RCTs published in a short period by one author/institute was an alarming trend that needs further investigation on its own.

Sub analyses

We undertook an investigation of five studies conducted in one research center (Table 6). The RR and 95% CI of Vuong et al., 2018 [8] and Dang et al., 2021 [27] were similar (1.05 [0.87, 1.27] and 1.11 [0.93, 1.32]), and Dang et al., 2019 [30] and Vuong et al., 2020 [33] were nearly identical (0.81 [0.66, 0.99] and 0.81 [0.66, 1.00]). Dang et al., 2019 [8] [30] claimed to have screened 1113 twin pregnancies within 15 months, of which 93% resulted from IVF. Vuong et al., 2021 [34] reported recruiting 151 out of 401 screened cases per month, which is a massive number for a single center.

Additionally, the recruitment rate was affected by the concurrent trials in some cases, as seen in Vuong et al., 2018 [8], where the average rate increased from 89 to 119 IVF cycles per month due to another ongoing trial. The same was observed in Vuong et al., 2020 [33] and Dang et al., 2021 [27], where 772 women were screened and 100 women were recruited per trial per month. There was an overlap in recruitment between two studies (Vuong et al., 2020 [33] and Dang et al., 2021 [27]). Inconsistencies were noted in the trial reports regarding participant exclusions due to involvement in other trials.

The ‘IVF ICSI trial’ enrolled PCOS patients with likely >24 antral follicle count (AFC), while the ‘IVM vs IVF trial’ concurrently enrolled participants with more than 24 AFC. Although the two trials (‘IVM vs IVF’ and IVF vs ICSI’) were recruiting concurrently, they did not refer to each other, causing reporting discrepancies. It remains unclear how PCOS patients with >24 AFC were counseled regarding both trials, as they were eligible for participation in either trial based on the Rotterdam criteria for PCOS.

RCTs, considered the gold standard in clinical research [2, 3], may be affected by biases, methodological flaws due to poor design, or compromised data integrity. The results of these trials may be untrustworthy and require further scrutiny before being included in systematic reviews, clinical practice guidelines or used in decision-making. Risk of bias tools are available but validated screening tools to evaluate the trustworthiness or integrity of RCTs are scarce. The TRACT checklist is a recently developed tool to evaluate the trustworthiness of RCTs. We critically-appraised the tool itself and evaluate its face validity using RCTs coauthored by its developer and identified significant violations in 16 RCTs. The objective analysis in this study revealed concerns about the tool itself as well as showing its lead developer to have coauthored several trials that harbor potential data integrity flaws.

The TRACT tool was developed to identify potential issues of trustworthiness but, to the best of our knowledge, has never been validated. If proven to be valid through further research, the TRACT checklist may serve as a tool for assessing the trustworthiness of RCTs, but it appears to have limitations and needs to be further refined prior to validation. While the seven domains covered by the checklist are applicable to RCTs, there may be additional trustworthiness issues that are not addressed. As a result, the use of the TRACT checklist alone may not be sufficient to ensure the trustworthiness of RCTs, and additional measures may be required to address these limitations.

We utilized the TRACT checklist to identify governance and authorship issues, etc., that may have introduced bias as the TRACT checklist may not account for all potential concerns or trial-specific nuances. Some of the items on the checklist are arbitrary or open to interpretation, such as ethical ramifications or unrealistic deadlines. Other items may not be applicable or pertinent to certain types of clinical trials, such as those involving multiple centers or countries. Thus, relying solely on the TRACT checklist may not provide an accurate assessment of the RCT’s trustworthiness. Future research should consider the development of more robust tools to provide a comprehensive RCT integrity evaluation.

In this study, using the TRACT checklist, 16 RCTs co-authored by its developer showed red flags that need to be investigated further. These violations include concerns related to retrospective registration, ethical implications, authorship, methodology, implausible timeframes, zero participants to follow-up, implausible baseline characteristics, and excessive inter-group and/or inter-trial similarities. Additionally, we identified inconsistencies between the trial registry and the final article regarding the reported start date, number of participants recruited end date, follow-up, and/or data collection dates in several trials. Our evaluation employs a simple methodology to offer perceptions of the face value of the TRACT criteria and the RCTs used to verify. As a result, the integrity of both the TRACT checklist and the RCTs of its developer are now under question.

When discussing the face validity of the TRACT checklist and the RCTs it evaluates, two potential concerns must be considered. First, while the TRACT checklist has promised to assess the quality of RCTs, its efficacy is constrained by significant limitations that need addressing. The checklist may not be exhaustive or valid to detect all issues with RCTs' credibility, possibly due to its failure to encompass all sources of credibility issues, which may vary depending on the evaluated RCT. Hence, researchers must be vigilant in considering other critical aspects of RCTs and evaluate the credibility of RCTs that pass the TRACT checklist critically. Secondly, the trials being flagged as potentially untrustworthy highlights two possibilities, either the checklist works, and the trials should be investigated further, or the checklist is unreliable with a high false positive rate. An internal investigation of his co-authored trials would be required to clarify the situation. Until this is resolved, researchers must exercise caution when using the TRACT checklist or RCTs authored by the TRACT developer to inform systematic reviews, guidelines or clinical practice. In all cases, the aforementioned issues may impact the TRACT checklist's generalizability since RCTs evaluated with the checklist may or may not have validity issues. Thus, researchers must be cautious when evaluating RCTs using the TRACT checklist.

The TRACT checklist may be a useful tool for researchers evaluating the credibility of RCTs. However, it has possible limitations that cannot be currently disregarded. When using the TRACT checklist to evaluate RCTs, researchers should exercise caution and be mindful of the possibility of bias in its development and lack of validation. As such, it should only be used in an experimental setting and not used in practice until it is further refined and validated. It is crucial to employ multiple sources of evidence and critical thinking abilities to ensure the credibility of RCTs.

TRACT: Trustworthiness in RAndomized Controlled Trials

RCTs: Randomized controlled trials

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable. The authors approved the publication of this manuscript

Competing interest

The trials that were employed to evaluate the effectiveness of the TRACT checklist were co-authored by an individual who is recognized for making accusation about research conducted at Egyptian universities, and some co-authors of this manuscript are employed by these institutions, with few of their studies are no exception.

Funding

None

Author’ contribution

The authors' contributions to the manuscript include conceiving the idea (HA and AR), extracting data and applying the TRACT checklist (HA and AS), arbitrating between reviewers to resolve any disagreement (AR), drafting the manuscript (HA and AG), revising the extracted data and the TRACT application (TA, and HE), leading and supervising the project (HA), and revising and critically appraising the manuscript and agreeing on the final version (all authors).

Table 1. RCTs with governance and authorship issues

Short name	Prolapse 2015 [7]	IVF ET 2018 [8]	Esomeprazole 2018 [31]	HALON 2019 [26]	IVM 2020 [33]	NOTABLE 2021 [32]	Pessary 2022 [11]	Gonadotrophin 2022 [28]	STOP 2023 [9]
Trial registration issues	Retrospective trial registration (NTR1197).	Retrospective trial registration (NCT02471573)	Retrospective trial registration (PACTR201504000771349)		> Retrospective change of the trial start date in the trial registry (NCT02471573). > A discrepancy in the trial registration date between the trial registry and the final article.		Retrospective trial registration (UTN:U1111-1164-2636)
Apparent concerns regarding ethics	Different reporting of ethical approval document between published study protocol [35] and the final article.			Ethical approval and registration of this trial and NOTABLE 2021 [32] in the same day.		Ethical approval and registration of this trial and HALON 2019 [26] in the same day.
Authorship concerns	The investigated author gained authorship on the final article being not an author of the published protocol [35] (honorary authorship??)						Different authors between the published abstract, the published protocol [36], and the final article	The investigated author gained authorship on the final article being not an author of the published abstract (honorary authorship??)	The investigated author gained authorship for merely drafting the manuscript (honorary authorship??)

Table 2. RCTs with methodology and timeframe issues

Short name

Prolapse 2015 [7]

Levothyroxine 2017 [25]

IVF ET 2018 [8]

Esomeprazole 2018 [31]

HALON 2019 [26]

Pessary 2019

IVM 2020 [33]

NOTABLE 2021 [32]

ICSI vs. IVF 2021 [27]

Metformin 2021 [10]

Pessary 2022 [11]

STOP 2023 [9]

Methodology issues

A discrepancy in methodology reporting between the trial registry (NTR1197) and the final article/ published protocol [35] (non-multicenter vs. multicentre)

A discrepancy in the reported 2ry endpoints between the published protocol [37] and the final article.

A discrepancy in the reported number of centers between the published abstract and the final article.

A discrepancy in the reported gestational age at inclusion between the published abstract and the final article.

A discrepancy in the reported number of patients per group, which contradicts the reported randomization method used.

Implausible Reported storage duration of pessaries for seek of randomization (7 years).

Unreported block sizes used for randomization.

Time frame issues

A discrepancy in the reported data collection time points between the trial registry (NTR1197) and the final article.

A discrepancy in the reported study start and end date between the trial registry ( ChiCTR-TRC-13004097), the published abstract, and the final article.

Implausible caseload within the reported time frame.

A discrepancy in the reported start, recruitment, and end dates between the trial registry (PACTR201504000771349) and the final article.

A discrepancy in the reported study completion date between the trial registry (NCT 02631837) and the final article.

Implausible caseload within the reported time frame.

A discrepancy in the reported start date between the trial registry (NCT02630329), the published protocol [38], and the final article.

A discrepancy in the reported study completion date between the trial registry and the final article.

Unusual randomization time frame (randomization of 38 women in 16 months and then 29 women in 35 months)

Implausible caseload within the reported time frame

A discrepancy in the reported actual day of last follow-up between the trial registry ( PACTR201608001752102) and the reported abstract.

A discrepancy in the reported study start date between the trial registry (UTN:U1111-1164-2636), the published abstract and the final article.

Table 3. Overview of trials with issues in reported results.