There are several prior analyses that report on how many studies provide results to the public. Huser et al. analysis from 2013 reported that 27.8% of analyzed interventional trials had published a linked result article.3 A systematic review by Bashir et al. from 2017 found that a median of 23% (ranging from 13% to 42%) were linked to a result article.8 With much increased public attention during the COVID-19 pandemic, we were motivated to find out what would be the percentage for COVID-19 studies. Our results, as of the query date, show that only 15.7% of COVID-19 interventional trials have a linked result article. However, it is too early to arrive at a formal number due to the relatively recent completion date (or formal ongoing status) of many trials.
Our methodology quickly identified result publications for prominent trials, such as trials involving vaccines approved in the US. Targeted review of those studies shows that such studies updated their CTG record frequently, which gives more confidence in the study metadata and study status (completed, terminated, or ongoing). In terms of paring trials with their result-reporting journal articles, the majority of linked result articles for interventional COVID-19 trials were found via abstract-link (55.0%), perhaps due to the easier practice of including the NCT ID in the article abstract.
The main advantage of our approach is offering researchers and the public a structured overview of literature with valuable metadata that combines information from scientific literature (PubMed) and clinical trial registry (CTG). It allows researchers to sort or aggregate articles based on various useful parameters (trial phase, sponsor, intervention and many others). Such capability is not possible with existing tools. Neither PubMed search nor clinical trial registry allow for review that would combine data from both sources. It allows for an overview of the clinical research in a given disease generated though automated computer script. For example, a review of all articles for a given intervention (such as hydroxychloroquine) could reveal if there is a consensus opinion on its efficacy or if there is a divide and more research is needed. In the case of hydroxychloroquine, a review of five results articles from four clinical trials in the US (on the prioritized short list) all expressed that the intervention was ineffective. A review of a full article master list (worldwide scope; not restricted to trials with at least 1 US site) would show a total of 81 articles from 38 trials studying hydroxychloroquine (see supplemental file for the master article list called ‘regCOVIDpublications_publication_list_int.csv’).
Levels of trial visibility
Our results show various levels of trial result reporting ranging from zero to multiple result articles. We found 96 COVID-19 interventional trials that had multiple study result articles, as well as multiple registry record updates. On the next level are trials with exactly one result article. Considering trials with at least one linked journal article, 70.6% of those have exactly one article. Within the set of trials with exactly one article, 26.9% only had a publication of publication type protocol and not of publication type study result article, which is most valuable. Finally, the vast majority of COVID-19 trials do not have any linked result publications (2 250 studies, 84.3%), making it difficult for interested parties to know the outcome of the trial. An even more extreme case of minimal trial information are trials with no linked result articles and zero updates (besides the initial registration) to the CTG study record (459 interventional trials, 17.2% of 2 669 total interventional trials). Our project, regCOVID, is the first to utilize number of registry record updates (and the type of this update) as a novel, computed study metadata construct to further categorize studies by level of activity. This can be helpful in comparing studies with identical official study status and improve the prioritization of result publications stemming from these studies.
Result deposition: As an alternative to publishing study results through an article, many studies chose to distribute study results by depositing them on CTG. A total of 61 trials deposited basic summary results. Within those, 35 trials only did registry result deposition and have no study result article and the remaining 26 trials did both result deposition and published a result article.
Trial registration timing
As part of our analysis, we found that trials register at three different points in time: (1) prior to trial initiation, (2) after trial initiation and prior to completion (during), and (3) after trial completion. For the set of all COVID-19 trials the breakdown was 2 237 (44.9%) prior to trial initiation, 2 157 (43.3%) during the trial, and 584 (11.7%) after the trial completion. In comparison, when considering all studies initiated in 2020 (not restricted to COVID-19), 59.3% registered prior to starting, 27.1% registered during the study and 13.6% registered after the study was completed. The comparison shows that COVID-19 studies are more likely to register late (during the study; proportion of 43.3% for COVID-19 studies versus 27.1% for general studies).
Publication timing
Publication of study results, including peer review, can be a complex and lengthy process. Prior studies indicate that it can take 21 months.9 In Supplemental S1 we review where trials stand in this ‘writing phase’. In a pandemic, like COVID-19, the quick publication of trial results is important for understanding which interventions are effective. Prior approval of COVID-19 vaccines and in the context of hospital staff and intensive bed shortages, clinicians were keen to learn about the efficacy of numerous tested interventions. A shorter publication timeline was targeted. Using our set of registered COVID-19 studies, on average, articles, that are not protocols, were published 149 days after the start of the trial. We used trial start date as an anchor since many trials list on CTG anticipated completion dates in the future.
Publishing prior to formal study completion: While primarily clinical trials publish study results articles after the formal trial completion date, for high profile trials it is not uncommon to see the opposite situation. During an ongoing pandemic, timely publication of results is important. For example, for the widely known trial regarding the Moderna COVID-19 vaccine (NCT04470427) which has an official primary completion date of 27 October 2022, the study result article was published in December 2020 (PMID: 33378609). This situation is, in fact, quite common. We found 364 trial result articles linked to 164 COVID-19 trials that are not formally completed as of the query date.
Other considerations
Termination reason: The updating of the study registry record can be very important to the public and researchers. An especially important update is change of study status to terminated. Namely, the reason for termination can provide a highly valuable insight. Such type of update is unlikely to be published as a separate article in a medical journal and the trial registry is the most suitable platform to communicate such an update. Of note is the fact that not all registries support record update and some may only focus on initial registration. To complement our intervention and publication prioritization, we also briefly analyzed the termination reason metadata supported by CTG registry. Most terminated trials (103, 87.3% of 118 terminated COVID-19 studies) specified a termination reason that helped explain why the trial was terminated. Most often, COVID-19 trials were terminated due to the inability to recruit and enroll participants. Other termination reasons were: intervention safety concerns, futility of the intervention, or availability of results from other trials making trial continuation unnecessary.
Publication bias: While manual review of abstracts of result publications was out of scope, we understand the potential presence of publication bias that may lead some trials to not formally publish results in a medical journal. For example, with reports of clearly terminated plans for further vaccine developments by some sponsors, a lack of result articles for certain trials and vaccine candidates hints at possible publication bias in vaccine trials.
Other manual trial trackers: Besides computational methods to obtain the most relevant COVID-19 journal articles, alternatively, it is possible to rely on websites (and research teams) that provide manually reviewed lists of completed studies with reported results. For example, The New York Times maintains a vaccine and therapy tracker.5 Another study tracker is published by the NIH.10 While it was out of scope to manually curate a sophisticated list of COVID-19 studies, or do a comprehensive review and comparison of our results to manual COVID-19 study trackers, we did compare the vaccine subset of COVID-19 studies identified through our methodology with those identified by the New York Times and NIH COVID-19 vaccine study trackers. Our motivation was to see how inclusive our methodology was. Our computerized approach study identification methods identified 25 of 32 phase 3 vaccine trials included in the New York Times vaccine tracker and included 5 of 6 trials present in the NIH study tracker.
Generalization to other diseases: regCTGpublications
Due to the computerized nature of our methodology, the method and developed script can be applied to other conditions to achieve an analogous overview of interventions and ranked list of publications. Our project called regCTG11 finds a list of studies for a given condition (generalization of regCOVID).1 A second project called regCTGpublications (or regCTGpub for short) generates a ranked list of result articles for trials in a given condition (generalization of regCOVIDpub). The regCTGpub project repository12 contains web-based result reports (analogous to table 1 and table 2) for select medical conditions (such as Age-Related Macular Degeneration, Alzheimer, etc.).
Limitations
Our study has several limitations. First, we rely on structured links between a registered study and the result article. A prior study for trials completed from 2004 to 2008 indicates that the negative predictive value of such a link may be as low as 56%.13 In other words, an unlinked result article may exist for a trial. However, in recent years, journal requirements to include NCT trial identifiers in an abstract may now be better enforced. Second, researchers have no obligation to publish result articles in a medical journal. Our study uses indexed medical journal publications, though sponsors may make study results public via a press release, instead. Third, our study uses only a single, US-based, clinical trial registry: ClinicalTrials.gov, though, on the other hand, other registries often do not allow linking of a result publication in a registry record, don’t support basic summary result deposition and have limited or no API access options. Also, the CTG registry has a significant number of non-US studies: as of March 2021, 60% of studies in the recruiting status were non-US only. Fourth, one part of our algorithm, that can be turned off or re-configured for a different country, focused on trials with at least one US site. We chose this because some legal mandates are tied to this factor. Also, approval in the US (by Food and Drug Administration) is a significant factor in world-wide regulatory context (with some exceptions). Fifth, interventions are entered into CTG as free text and proper linkage of identical interventions (expressed using similar intervention strings, such as ‘anti-sars-cov-2 convalescent plasma’ and ‘convalescent covid 19 plasma’) depends on a computational algorithm that can miss some linkage of identical interventions.