We first summarize the results of the nine rapid reviews across the three case studies and then provide a more in-depth analysis of the findings for each case study. As presented in Table 1 and the Methods, case study 1 addressed the effectiveness of nivolumab for adult patients with Hodgkin’s lymphoma, case study 2 early versus deferred androgen suppression for advanced prostate cancer, and case study 3 unconditional cash transfers for reducing poverty and vulnerabilities. Table 2 summarizes the characteristics of the Cochrane reviews that we used as reference standards; Additional file 2 also presents the studies that served as each topic’s “starter set” for the abbreviated literature searches of each case study. Table 3 summarizes the number of abbreviated literature searches and the flow of the literature during screening for each of the nine rapid reviews.
Changes in conclusions
Our primary question of interest was whether the Cochrane authors would have drawn the same or different conclusions had they relied on a rapid review instead of the Cochrane review. Overall, the conclusions would have been the same as in the Cochrane reports for seven of the nine rapid reviews. For the two oncological topics (case studies 1 and 2), the three rapid reviews for each topic rendered the same conclusions as the respective Cochrane reviews, although the authors would have had less certainty about their conclusion in case study 2. For case study 3, the public health topic, only one of three rapid reviews led to the same conclusion as the Cochrane review. The other two rapid reviews did not contain enough information for the authors to draw conclusions anymore (see Table A2 in Additional file 1).
Sensitivity to identify relevant studies
For each topic, we assessed the sensitivity of the abbreviated literature search to identify studies that the Cochrane review had included. In addition, we were interested in the combined sensitivity of the abbreviated literature searches, the review of the relevant reference lists, and the single-reviewer screening of the abstracts and full texts.
The sensitivity of the abbreviated searches for the two oncological topics was high. They detected 100% (3 of 3) of the included studies for case study 1 and 90% (9 of 10) for case study 2 (see Figures 2 and 3). For the public health topic (case study 3), the sensitivity of the abbreviated search was only 33% (7 of 21; see Figure 4). The underlying assumption for these numbers is that the identification of at least one of sometimes several publications of the same study can be equated with the identification of the study. For case study 3, all the studies that the searches did not identify were either grey literature or from journals not indexed in PubMed.
Searching the reference lists did not identify a single missed study in any of the three rapid reviews for case study 2. For case study 3, the reference list searches identified 3 to 4 studies that had been missed by the abbreviated literature searches (see Table 3). In case studies 2 and 3, the single-reviewer screening missed several relevant studies that the abbreviated literature searches had identified. Across these six rapid reviews, the single-reviewer screening missed a median of 31% (range 14% to 56%) of the relevant studies. When we combined the abbreviated searches with the references list searches and single-reviewer screening, the sensitivity remained at 100% (3 of 3) for case study 1. For case study 2, the single-reviewer screening missed several relevant studies and led to a decrease in the sensitivities for all three rapid reviews. For case study 2, the three rapid reviews identified 40% (4 of 10), 50% (5 of 10), and 60% (6 of 10), respectively; for case study 3, the respective numbers were 38% (8 of 21), 43% (9 of 21), and 48% (10 of 21) (see Figures 2 to 4; Table A1 in Additional file 1).
In the following sections, we summarize each case study in more detail.
Case study 1
The objective of the Cochrane review for case study 1 was to assess the benefits and harms of nivolumab in adults with Hodgkin’s lymphoma. This Cochrane review [16] included three prospective uncontrolled studies published in 26 publications (see Table 2). The abbreviated literature searches identified all three studies but only 3 of the 26 publications. The references list searches detected one additional eligible publication for the third rapid review. Across the three rapid reviews, the single-reviewer screening missed none of the relevant studies identified by the abbreviated literature searches (see Table 3). Based on the abbreviated literature and reference list searches, the rapid reviewers had to screen only small proportions of the abstracts that the investigators screened for the Cochrane review (15% [99 of 675], 15% [98 of 675], and 15% [100 of 675]). The rapid reviewers spent a median of 2.1 hours checking the reference lists and the literature screening (abstracts and full texts). As presented above, the abbreviated literature searches alone and all three rapid reviews identified 100% (3 of 3) of the eligible studies. Consequently, the Cochrane reviewers’ conclusions would not have changed (see Table A2 in Additional file 1). The approach, however, detected only 12% (3 of 26) to 15% (4 of 26) of the relevant publications about these studies. Figure 2 depicts the studies and number of respective publications that each of the three rapid reviews identified for case study 1 (see also Table A1 in Additional file 1).
Case study 2
The Cochrane review for case study 2 focused on early versus deferred androgen suppression for treating advanced prostate cancer [18]. It identified 10 eligible RCTs published in 53 manuscripts (see Table 2). Similar to case study 1, the rapid reviewers had to screen only small proportions of the abstracts screened for the Cochrane review (1% [196 of 19,380], 1% [197 of 19,380], and 1% [204 of 19,380], respectively). The reviewers spent a median of 6.5 hours screening the literature and scanning the reference lists.
Despite the low number of records that the abbreviated search retrieved (24 of 53), 90% of the relevant studies (9 of 10) were included in the abbreviated search. In case study 2, the sensitivities substantially decreased during the single-reviewer screening. The screeners falsely excluded 3 to 5 relevant studies during the abstract screening.
The sensitivities of the combination of the abbreviated searches and single-reviewer screening were 40% (4 of 10), 50% (5 of 10), and 60% (6 of 10), respectively. Of the 53 manuscripts published on the 10 studies, the approach identified between 11% (6 of 53) and 13% (7 of 53). Nevertheless, when asked to base their conclusions on the evidence base of each of the three rapid reviews, the Cochrane review authors would have still drawn the same conclusions, albeit with less certainty (see Table A2 in Additional file 1). Figure 3 depicts the studies and number of respective publications that each of the three rapid reviews identified for case study 2 (see also Table A1 in Additional file 1).
Case study 3
For case study 3, the Cochrane review [17] focused on unconditional cash transfers to reduce poverty and vulnerabilities and included 21 studies (16 cluster RCTs, 4 controlled before–after studies, and 1 cohort study) summarized in 68 publications (see Table 2). Similar to the other case studies, the percentage of the abstracts screened by the rapid reviewers compared to the Cochrane reviewers was low (1% [387 of 30,453], 1% [394 of 30,453], and 1% [434 of 30453]). The reviewers spent a median of 22.6 hours on the screening and searching the reference lists.
The proportion of relevant studies that the abbreviated searches identified (33% [7 of 21], see Figure 4) was substantially lower for this topic than for the oncological topics. Searching the reference lists added 3 to 4 studies that had been missed by the abbreviated literature searches across the three rapid reviews (see Table 3).
During the single-reviewer screening, the investigators falsely excluded 1 to 2 studies across the rapid reviews. The sensitivity of the combination of the abbreviated search and single-reviewer literature screening on the study level was 38% (8 of 21), 43% (9 of 21), and 48% (10 of 21), respectively. Of the 68 manuscripts published on the 21 studies, the approach identified between 13% (9 of 68) and 22% (15 of 68). The Cochrane review authors would have drawn the same conclusions but with less certainty based on the evidence identified for one of the rapid reviews; however, they would have been unable to draw a conclusion at all had they relied on the evidence identified by the two other rapid reviews (see Table A2 in Additional file 1). Figure 4 depicts the studies and number of respective publications each of the three rapid reviews identified (see also Table A1 in Additional file 1).