In June 2020 ivermectin, an FDA approved anti-parasitic drug, was shown to have anti-viral effects against SARS-CoV-2 in-vitro1. Following this, approximately 86 clinical trials investigating ivermectin for COVID-19 have been registered globally. In late 2020, clinical trials began reporting significant clinical benefits for ivermectin in the treatment of COVID-19. From late 2020 onwards, multiple groups produced meta-analyses which reported that ivermectin had a significant effect on survival, hospitalisations, clinical recovery and viral clearance2, 3. Our meta-analysis was first presented in January 2021 and published in July 20214. It suggested that ivermectin resulted in a significant 56% improvement in survival, favourable clinical recovery, and reduced hospitalisations. Such optimistic results from multiple meta-analyses have escalated public interest in using ivermectin for the treatment and prevention of COVID-19, despite the WHO only recommending its use within clinical trials5, 6.
However, as with all meta-analyses, a key limitation is the quality and completeness of the available evidence. During our original assessment of studies, standardised Cochrane Risk of Bias measures (RoB 2) had classified several studies as ‘high risk of bias’7. A study by Niaee et al from Iran which reported a randomised methodology, was found to have significant differences in baseline characteristics across treatment arms8. This suggested that the participants were not randomised appropriately, which could bias the outcomes. Secondly, a study by Okumus et al from Turkey did not provide any information on allocation concealment and it was unclear if the participants or investigators were blinded, which risks introducing observation bias9. Lastly, a study by Hashim et al from Iraq provided insufficient details on the randomisation process, lack of clarity on participants who were analysed and involved unblinded assessment of a subjective outcome10.
Furthermore, some studies were then identified to be potentially fraudulent. For example, on 15th July 2021, a study by Elgazzar et al from Egypt was retracted from pre-print server Research Square due to “ethical concerns”11. It has been reported that the data for approximately 79 participants were duplicates, some deaths were recorded on dates before the trial had started and instances of plagiarism were identified in the text12. Similarly, a study conducted in Lebanon by Raad et al was also identified to have duplicate data for multiple participants when the patient-level database was analysed in September 202113. Before these inconsistencies were identified, the Elgazzar and Raad studies had been included in multiple meta-analyses which suggested significant benefits for ivermectin in the treatment of COVID-192, 3. In our original meta-analysis, the Raad study accounted for 11.8% of the effect of ivermectin on hospitalisation and the Elgazzar study accounted for 12.6% of the effect of ivermectin on survival4.
These instances suggest that the data available to the support the use of ivermectin for COVID-19 is not reliable. In July 2021, after the potentially fraudulent studies were identified, we retracted our published meta-analysis and began working on an updated analysis, assessing the effects of stratifying by trial quality on the overall results. Clinical trials evaluating ivermectin for the treatment of COVID-19 had been identified by systematic search of 8 databases. An in-depth evaluation of study quality was conducted, in addition to the standard Cochrane risk of bias tool (RoB 2) and CONSORT checklist7. Firstly, we evaluated trials based on the effectiveness of their randomisation process by comparing baseline characteristics across treatment arms using the chi-square test. Secondly, randomisation dates were checked to ensure patients were randomised into the treatment arms on similar dates. Thirdly, checks were conducted to evaluate if recruitment to treatment arms was balanced at each investigational centre. Furthermore, we analysed patient-level databases, where available, to check for any evidence of duplicate participants, unexpected homogeneity or heterogeneity. From this, a meta-analysis was conducted with sub-groups of clinical trials at different risk of bias levels.
Looking at the key survival outcome, the analysis includes 12 studies with a total of 2628 participants4 (Table 1). This included 4 studies at a low risk of bias, 4 with some concerns, 3 at a high risk of bias and 1 potentially fraudulent study. The analysis demonstrates that on including all 12 studies, ivermectin results in a significant 51% increase in survival (p=0.01). On excluding the potentially fraudulent Elgazzar study, ivermectin results in a borderline significant 38% increase in survival (p=0.05). On excluding the high risk of bias studies, ivermectin results in a non-significant 10% increase in survival (p=0.66). Lastly, on excluding studies with some concerns of bias, ivermectin results in a non-significant 4% increase in survival (p=0.90) (Figure 1). These observations demonstrate that the significant effect of ivermectin on survival was dependent on the inclusion of studies with a high risk of bias or potential medical fraud.
There are added challenges with clinical trials investigating COVID-19 treatments. In a rapid response to COVID-19, many small-scale studies have been conducted globally for potential agents. However, not all trials have reported findings. An example is a trial for Nitazoxanide in Brazil with 600 participants which was completed in October 2020 but has not reported any results yet14. Publication bias impacts meta-analyses, with significant results more likely to be reported. Some non-randomised trials may also be over-interpreted. For example, in a non-randomised retrospective cohort study, remdesivir demonstrated an improvement in clinical recovery and reduced mortality risk by 62%15. However, when evaluated in the WHO’s randomised placebo-controlled SOLIDARITY trial, remdesivir had little to no effect on mortality16. Any initial promising findings from a small number of sources need to be interpreted with caution, studied further and considered within the wider body of evidence.
The results from this analysis highlight the need for rigorous quality assessments when evaluating clinical trials of drugs for COVID-19. Existing and widely used risk of bias assessment tools are not enough. These tools provide a systematic framework for identifying potential key sources of bias in a trial’s internal methodology but work on the fundamental assumption that a published study is reporting accurate and complete findings. They allow reviewers to make judgements on the assumption that basic standard procedure is followed, the data is real, and that no information is being intentionally hidden.
With cases of potential medical fraud now identified, it is essential that access to patient-level databases is provided. If authors fail to provide this data, the study should be considered with a higher index of suspicion. Additionally, it should be mandatory that all registered trials report their findings. We understand that these are significant changes to established procedures. However, the failure to recognise the potentially fraudulent studies, which led to multiple meta-analyses suggesting significant benefits of ivermectin for COVID-19, indicates that the tools currently used to evaluate the quality of clinical trials are insufficient. These events warrant our stringent recommendations.
