We summarise the steps in the life cycle of a living systematic review in Figure 2. In Box 1, we summarise the methods that we used in our own living systematic review.(11) Frequently overlooked is the need to be realistic about the time needed to start and update the living systematic review and plan as many steps as possible in advance, taking into consideration that numbers of records to screen might continue to increase quickly, as has happened with covid-19 literature. Given the commitment required and the scale of the workload, reviewers should make sure that their living systematic review question has not already been addressed, or is being addressed, by searching the published literature, systematic review registries, such as the PROSPERO international register of systematic reviews,(4-6, 10) or the Open Science
Framework (OSF).(9, 11)
[Fig 2]
Box 1. Summary of a living systematic review of asymptomatic and presymptomatic SARS-CoV-2 infection.
Protocol: first published 1 April 2020, last updated 13 July 2022 (1)
Versions: 1) 29 April 2020 (preprint); 2) 24 May 2020 (preprint); 3) 28 July 2020 (preprint) 4) 22 Sept 2020 (peer-reviewed publication); 5) 30 Jan 2022 (preprint); 6) 26 May 2022 (peer-reviewed publication).
Research questions: 1) Among people who become infected with SARS-CoV-2, what proportion does not experience symptoms at all during their infection? 2) What is the infectiousness of people with asymptomatic and presymptomatic, compared with symptomatic SARS-CoV-2 infection? 3) What proportion of SARS-CoV-2 transmission is accounted for by people who are either asymptomatic throughout infection or presymptomatic?
Inclusion criteria: Studies of people with SARS-CoV-2 diagnosed by RT-PCR that documented follow-up and symptom status at the beginning and end of follow-up or investigated the contribution to SARS-CoV-2 transmission of asymptomatic or presymptomatic infection.
Exclusion criteria: Case series restricted to people already diagnosed; studies that did not report the number of people tested for SARS-CoV-2, from whom the study population was derived; case reports; contact investigations of single individuals or families; and any study without sufficient follow-up.
Articles screened and included in review: The searches for studies about asymptomatic or presymptomatic SARS-CoV-2, on 25 March, 20 April, and 10 June 2020 and 2 February and 6 July 2021 resulted in 89, 230, 688, 4,213, and 3,018 records for screening, respectively. The latest version included 146 studies in total. |
Setting up and managing a review team
The core team should coordinate the tasks of the review team, which includes anticipating and managing changes in workload, workflow and team composition. At least one core team member should have sufficient programming skills to automate steps in the workflow where possible. In our case study, a core review team of seven became overwhelmed when the number of new hits to be screened increased (Box 1). Growth in the team of researchers can be seen when following updates of other living systematic reviews.(9) Crowdsourcing is a valuable tool for large reviews (18) and can be mutually beneficial for the volunteers,(28) but the core team must weigh up the time spent training volunteers against the time saved. Some review teams have anticipated the workload and used crowdsourcing from the outset, increasing the size of the team to more than 100 volunteers to help with screening and data extraction (4) and other reviews mention the use of volunteers.(5, 8) We recruited twenty volunteers from April 2021, through the core team’s networks. All volunteers had previous experience with systematic reviews and agreed to spend at least three hours per month working on eligibility assessment, data extraction and/or risk of bias assessment. The core team provided online guidance materials (Additional file 3), individual feedback, and automated tools. Training for members of the crowd (Additional file 3) reduced potential disagreements in screening, extra work for the core team, and delays in the living systematic review.
Defining eligibility criteria for authorship is an essential task for the core team at the start of the review process, and should agree the policy in advance with the review team, including crowdsourced members. In our review, levels of contribution and availability changed during and between updates. Team members who fulfilled the criteria for authorship were co-authors of the relevant publication. We created lists of contributorship (29) in which people whose contributions no longer fulfilled criteria for authorship had their contributions acknowledged separately.
Publishing a protocol
A protocol for a living systematic review is also a living document, which should reduce potential biases and avoid posthoc decisions.(2, 19, 30) Publishing a protocol on PROSPERO,(4-6, 10, 16) on a preprint server, or public repositories like the OSF (9, 11) allows rapid sharing and updating of protocols. In a living systematic review, the review questions and scope and types of evidence included might evolve over time, so authors should document and justify changes to the protocol before starting an update, including decisions about the frequency of updating and about stopping the review. If protocol changes are needed, authors should note that the scope of a living systematic review can only become narrower over time without having to make changes to the original search strategy. Over the seven versions of the protocol for our review of asymptomatic SARS-CoV-2, the scope has narrowed over time.(1) In the first version, there were few publications and we included study populations in any setting. After our third version, we reduced the number of studies for data extraction by excluding small studies reporting on single family contact investigations and studies of hospitalised people, who were more likely to be symptomatic. The seventh version of the review has been restricted to SARS-CoV-2 variants of concern.
Study identification
Automatic alerts from bibliographic databases can notify researchers when new records are available.(18) For complex reviews, researchers with sufficient programming skills can set up automatic scripts to regularly search and collect results from search databases using programming languages, either with an application programme interface (API) (a software intermediary that communicates with websites from a third-party application) or by ‘web scraping’. Database aggregators are convenient, single sources for a topic of interest, information scientists develop, refine, automate, and update search strings in different electronic sources and de-duplicate the records. Database aggregators for covid-19 literature include the World Health Organization COVID-19 Database and the Cochrane COVID-19 Trials Register (https://covid-19.cochrane.org/). We used the COAP living evidence database, a database aggregator,(12) which we ran from March 2020 to March 2022.(11) We scheduled an automated R script (31) to search COAP weekly, using the task scheduler, Cron. Each week, the automatic search uploaded 100-200 new records from the COAP database for our living systematic review. We searched preprint servers and included preprints if they fulfilled eligibility criteria. In each update, we checked the status of preprints to see if they had been published in peer-reviewed journals and re-extracted data if the content had changed.
Electronic online databases to save and manage records support a secure and efficient workflow. Living systematic reviewers are using tools such as Evidence for Policy and Practice Information (EPPI)-reviewer,(32) Covidence,(7, 33, 34) or Microsoft Excel to organise records. New records in our review (Box 1) are saved in a Research Electronic Data Capture (REDCap) database,(35) a flexible and secure online system. A copy of the data is stored in a collaborative software repository.(36)
Study selection
Several software tools offer fast and user-friendly platforms to facilitate screening records.(33) Living systematic reviews on covid-19 have used REDCap surveys,(4, 35) EPPI-reviewer,(9, 32) and Covidence.(7, 34) The tools support multiple users, allocate tasks, record decisions, and produce automatic reports.(33) The open source R package revtools(37) supports the screening of titles and abstracts and deduplication. When specific features are desired, or if software licenses are unaffordable, building a custom application using open-source software might be more suitable. We constructed password-protected R Shiny applications to support the selection process (Figure 3).(11) The core team allocates records to the reviewing team via REDCap.(35) The applications included features to allow the team to train a machine learning algorithm (see below).
[Fig 3]
Semi-automated machine learning tools for the selection process can reduce the volume of studies that needs to be screened manually.(38) However, the tools may not perform as well for observational studies as for RCTs, for which accepted reporting guidelines and terminology facilitate reliable identification of reports.(38) Wynants et al. built a custom classification model to speed up the selection process in their living systematic review of prognostic models for covid-19.(9) They used the initial set of records that they screened to train an algorithm to recognise patterns in text to identify studies that are very unlikely to be relevant and automatically exclude them. For reporting the results of searching, selection and inclusion, a specific flow chart for living systematic reviews allows a logical way of updating.(39)
Data collection
Web applications may help to streamline manual data extraction by reviewers who are extracting information independently or verifying the information extracted by another reviewer, including using the use of online forms,(5, 8) REDCap surveys,(4) or standardised prespecified extraction forms.(7, 9, 10, 15) None of these living systematic reviews mentioned the use of automated tools for data collection. We used RShiny applications to facilitate both steps and save decisions in REDCap.(11, 35) While machine learning tools for data extraction exist, very few are publicly available.(40) The tools face challenges with variations in wording, missing information, and adaptability to a subject area on which the tool was not developed.(40)
Data synthesis
Manual checks on included studies are still needed before starting data synthesis, especially when a large crowd has contributed to selection of studies and extraction of data and rapid processes have been put in place. Routine checks include making sure that data have not been included from a preprint and a published version of the same study. There are many statistical software packages for conducting quantitative data synthesis for living systematic reviews, including Stata (41) and R.(5-7, 10, 11, 31) The use of an API to communicate between an online database and the statistical software allows reviewers to import the latest data and update the analysis when new data are available. Reviewers can generate tables and figures automatically using statistical software (e.g., RMarkdown,(31) Stata (41)).
There are issues associated with repeated updating of statistical analysis, which are particularly relevant in living systematic reviews. With each update, the analysis of data from RCTs or comparative effectiveness studies is more likely to generate a false statistically significant result.(17) Even when the aim of the living systematic review is to support a decision, e.g. to decide which intervention is more effective, statistical significance is rarely the only criterion guiding this decision. However, reviewers can employ methods that control the type I error if they want.(17, 42)
A substantial proportion of living systematic reviews rely on observational studies (Table 1), in which levels of between heterogeneity are often high,(43) and for which meta-analysis might not be appropriate. Although several living systematic reviews on covid-19 have conducted meta-analyses,(5, 7, 44) some did not, owing to high heterogeneity in included studies.(11, 15) In our living systematic review (Box 1), between-study heterogeneity has increased with each update, contrary to our expectation, and we could not explain most of the heterogeneity.(11) In the fifth and sixth versions of the review, we did not produce a summary estimate for the proportion of asymptomatic SARS-CoV-2 infection. Instead, we reported an interquartile range for the results from included studies, estimated a prediction interval (45) to show the range of values in a future hypothetical study.
Publishing a living systematic review
Living systematic reviews should be published in a way that explicitly cross-references different versions of the report as updates of the same review.(21) These links are needed to make sure that readers have access to the most recent update and that different versions of a living systematic review are not mistaken for redundant publications. Reviewers should consider contacting the editors of their target journal to find out whether they can submit a living systematic review and to find out how the journal handles updates. Editors of online publications, print publications, and preprint servers use different methods, and apply different rules about what they consider a ‘version of record’,(46) which refers to the version of an article that is considered final and is identified online with a digital object identifier (DOI). For living systematic reviews, the version of record is not defined consistently across journals.(23) Different publishers apply different rules to determine whether an update receives the same DOI as a previous version, or a new DOI. This decision can depend on whether the journal editors consider and update as minor or major. The BMJ assigns the same DOI to all versions of a living systematic review and adds a ‘reader’s note’ to the abstract, signalling the update number and how to find earlier updates.(9) Cochrane reviews have indexed updates for many years and assign a DOI that incorporates the same review number for all updates and includes an extension with the update number.(5) Newer online publishers, such as F1000, use the same principle as the Cochrane library, and also include the version number in all article titles.(40) The Public Library of Science (PLoS) publishes minor updates as online comments to the earlier version and assigns a new DOI if the editors consider it a major update.(11) Annals of Internal Medicine uses a similar approach, with minor updates published as letters.(15, 47) Preprints are not considered a version of record. The medRxiv server allows updates to a living systematic review to be uploaded under the same DOI (available in the history of the article) until the review is published in a peer-reviewed publication.(25, 48) After that, only a major update can be uploaded and that version receives a new DOI, again, until published. In our case study (Box 1), we have published both preprints (25, 48) and peer-reviewed articles.(11, 49)
Transparency is important when sharing results of living systematic reviews. Living systematic review and living guidelines teams who maintain dedicated websites can display updated results as soon as they are incorporated and include links to articles, protocols and datasets using FAIR principles (Findability, Accessibility, Interoperability, and Reuse of digital assets).(4-7, 9, 11, 23)
Stopping a living systematic review
An important feature of a living systematic review is knowing when to stop and the criteria for stopping should be part of the review protocol, updated if necessary. Covid-19 living systematic review teams have reported a predefined point at which they intend to stop: either a specific month (4, 5, 10) or when new evidence is unlikely to emerge.(7) In our review, we stated the following criteria for ending the review: when estimates are stable and unlikely to change or the question is no longer of importance.(11) Alternatively, publishers or available funding may determine the lifetime of a living review. The BMJ has set a duration of two years for a living systematic review, after which the editors and authors should assess the need for continuation.(24)