Development and Optimization of an mNGS Assay for Detection of Viral Respiratory Pathogens
We developed an mNGS assay for the detection of viral pathogens from respiratory secretions, including upper respiratory swab and bronchoalveolar lavage (BAL) fluid samples (Figure 1). We leveraged our 7-year experience running clinical mNGS assays for pathogen detection from cerebrospinal fluid20 by optimizing the sample preparation and bioinformatics analysis protocols to maximize sensitivity and decrease assay sample-to-result turnaround time. We tested different combinations of centrifugation, heat, and addition of a DNA/RNA stabilization medium prior to total nucleic acid extraction and found that centrifugation alone produced the highest yield of detected viral reads. To decrease turnaround times, we used a 15-minute protocol for human rRNA depletion and reduced incubation times for the reverse transcription and second-strand cDNA synthesis steps to 15 and 9 minutes, respectively. The final assay used 450 μL of sample input volume and consisted of the following steps: (1) centrifugation (~15 min), total nucleic acid extraction and DNase treatment for isolation of total RNA (~1 hr), (2) cDNA synthesis with ribosomal RNA (rRNA) depletion (~1 hr), (3) barcoded adapter ligation, library PCR amplification and purification on an automated instrument (~6.5 hr), (4) library pooling (~5 min), (5) Illumina (San Diego, CA) sequencing (5 or 13 hr, depending on whether a MiniSeq or NextSeq sequencer is used), and (6) bioinformatics analysis for viral detection and quantification using the SURPI+ pipeline (~1 hr). Overall sample-to-answer assay turnaround time was 14 - 24 hours. We used MS2 phage and External RNA Controls Consortium (ERCC) RNA Spike-In Mix (Invitrogen, Waltham, MA) added into each sample as internal qualitative and quantitative controls, respectively. The MS2 phage and ERCC sequencing results were also used to evaluate and interpret the background level in the sample, generally originating from the human host (Supplementary Tables 1 and 2).A commercial reference panel (Accuplex Panel, SeraCare, Milford, MA) consisting of quantified SARS-CoV-2, influenza A, influenza B, and respiratory syncytial virus (RSV) was spiked into pooled virus-negative nasopharyngeal swab matrix (see Methods for details) as an external positive control (PC) for the assay, with pooled virus-negative nasopharyngeal swabs from healthy uninfected donors as the negative matrix serving as an external negative control (NC).
The SURPI+ computational pipeline, run as a container on either a server or cloud, was used for the identification of viral respiratory pathogens from mNGS data21,22. Three enhancements were made (Figure 2A). First, we added the capability for viral load quantification using the PC and a standard curve generated for each sample from the ERCC reads. Second, “tagging” of Genbank accession numbers in the SURPI+ database was incorporated to allow inclusion of curated viral reference genomes, such as those deposited in the FDA-ARGOS database23, for virus identification by alignment and results reporting . Third, a custom algorithm consisting of de novo assembly of metagenomic reads and translated nucleotide, or amino acid, alignment of the reads to a viral protein database was developed to enable detection of novel, sequence-divergent viruse 23.
Following the review of clinical charts, we investigated the correlation between viral load concentration, quantified in copies per milliliter (cp/mL) (Figure 2B). The severity of the infection which was categorized on a scale ranging from asymptomatic to mild, moderate, and severe. We observed significant differences in median viral loads between patients with asymptomatic/mild and moderate/severe infections (P < 0.001) (Supplemental Fig. 5a). Further stratification of patients into asymptomatic, mild, moderate, and severe infections highlighted an increasing trend in viral load concentrations. Through pairwise comparisons, we noted significant differences between asymptomatic and moderate (P < 0.01), as well as between mild and moderate (P < 0.01) infections. Overall, differences in median viral loads across all severity levels were significant (P < 0.001) (Supplemental Fig. 5b).
Quality control metrics were based on those previously established for a validated cerebrospinal fluid mNGS assay21 and include a minimum of 5 million preprocessed reads per sample, >75% of data with quality score >30 (Q>30), and successful detection of the internal spiked MS2 phage control and all four respiratory viruses in the PC. A threshold criterion of ≥3 non-overlapping viral reads or contigs aligning to the target viral genome was considered a positive detection. Overall, 93% (156 of 167) of both positive (n= 111) and negative (n=56) nasopharyngeal swab samples met QC metrics, those that did not meet QC metrics were excluded from the analysis.
Analytical Sensitivity
We adopted Clinical and Laboratory Standards Institute (CLSI) guidelines for NGS-based infectious diseases testing (MM24)24 and validation of multiplex nucleic acid assays (MM17)25 to conduct a comprehensive evaluation of assay performance metrics (Table 1). To determine limits of detection (LoD), negative nasopharyngeal swab matrix was spiked with the Accuplex Verification Panel and diluted at concentrations ranging from 5,000 to 100 copies/mL, with 10 to 40 replicates at each concentration. By 95% probit analysis, the LoD was determined for each of the four representative organisms in the panel (SARS-CoV-2, Influenza A, Influenza B, and RSV). We found LoDs ranging from 439 to 706 copies/mL for the four respiratory viruses in the positive control (Figure 3). The achieved average LoD of 550 copies/mL was comparable within one log to reported LoDs from specific reverse transcription-polymerase chain reaction (RT-PCR) assays for detection of viral respiratory pathogens26.
Linearity
To evaluate the assay’s capability to accurately quantitate viral load for detected viruses, a linearity panel was generated using five log dilutions of a quantified high-titer SARS-CoV-2 positive nasal swab sample and compared to a commercially available AccuSpanTM HCV RNA Linearity Panel. For both panels, the calculated linearity was 100% after running duplicates or triplicate replicates across a minimum of four 10-fold dilutions (Supplementary Figure 1). The absolute log10 deviation of calculated from expected viral loads was <0.52 log10, which was favorable in comparison to the interquartile ranges for virus-specific qPCR assays between different laboratories27.
Precision
We measured intra-assay precision by testing two PC and two NC samples within the same run using different barcodes across 20 runs and inter-assay precision by testing 20 PC and 20 NC samples using different barcodes across 20 separate runs. Essential agreement (EA) was 100% and intra- and inter-assay precision were within our a priori established limits of <10% and <30% (log-transformed coefficients of variation in reads per million), respectively (Table 1).
Inclusivity and Exclusivity
To evaluate the ability of the mNGS assay to detect a wide range of targets (inclusivity), we obtained commercially available culture supernatants from 17 respiratory viruses representing different sublineages and subspecies. Viruses were spiked into negative control matrix at concentrations ranging from 1.3 x 103 to 1.2 x 107 50% tissue culture infective dose (TCID50) per mL in 1:10 ratio (Table 2).All 17 (100%) of 17 viruses in these contrived samples were correctly identified by mNGS assay at the sublineage or subspecies level. Additionally, we identified subtypes of rhinovirus and enterovirus from PCR-positive clinical samples that were not differentiated by multiplex RT-PCR (Supplementary Figure 2A). We also evaluated the ability of the mNGS assay to identify uncommon or rare viral pathogens associated with respiratory infections (n=8 virus-positive tracheal aspirate samples) or central nervous system (CNS) infections (n=4 cerebrospinal fluid samples) in severely ill hospitalized patients (Table 2, Supplementary Figure 2B). The assay detected 11 (100%) of 11 viruses in these samples. To assess the exclusivity of the mNGS assay, we spiked two mixtures of microorganisms, including a previously reported positive control mNGS panel consisting of 7 representative pathogens21 and a commercial reference panel consisting of 10 bacterial and fungal species, into negative nasopharyngeal swab matrix and analyzed multiple aliquots (Table 1 and Supplementary Table 3). Detected reads from non-viral pathogenic organisms did not result in any false-positive detections for viral pathogens.
Contamination,, Matrix Effect and Stability
We evaluated potential cross-contamination between nearby sample wells and carryover contamination across successive runs from 10 SARS-CoV-2 high-titer clinical samples and 24 controls (cycle threshold, or Ct = 16-20) loaded in a modified checkerboard pattern (with at least one space between samples) on a 96-well plate, to mimic a single run on the Illumina NextSeq instrument. Only one possible cross-contamination event was observed, with a single SARS-CoV-2 read detected in one of the negative control wells at a subthreshold reporting level. We also evaluated the effects of interference from human RNA, bacterial DNA, and potential interfering substances on mNGS assay performance. Hemolysis, lipids, bilirubin, and human genomic RNA spiked into PC matrix at concentrations of 0.1 – 100 µg/mL did not interfere with respiratory virus detection, but background DNA/RNA spiked into PC matrix at concentrations ³1 x 107 cells/mL resulted in failure to detect viruses due to high background. To evaluate the potential matrix effect from samples with high host background, we analyzed 14 PCR-positive highly mucoid bronchoalveolar lavage (BAL) samples obtained from lung transplant or cystic fibrosis patients undergoing surveillance bronchoscopy (Supplementary Table 4). All 14 samples had high host background, and 13 (92.9%) of 14 samples had very high host background. As a result, 6 (42.9%) of 14 samples had neither detection of the internal spiked MS2 phage control nor of a respiratory virus, and thus excluded from further analysis, as they not pass equencing quality control criteria (Supplementary Table 1). The respiratory viral pathogen was detected in all (100%) of the remaining 8 samples. We concluded that highly mucoid samples can inhibit the assay due to high host background. Finally, we evaluated mNGS assay stability; qualitative detection was not affected by keeping samples for up to 7 days at 4°C or subjecting the samples to 3 freeze/thaw cycles.
Accuracy
To evaluate accuracy, 191 residual samples after routine clinical testing were obtained from the UCSF Clinical Microbiology Laboratory, including 110 virus-positive samples (104 upper respiratory swab samples and 6 BAL fluids) from patients with acute respiratory infection (Supplementary Dataset 1), along with 81 virus-negative samples (52 upper respiratory swab samples and 29 BAL fluids) (Figure 4).As more than one target may be positive with mNGS and respiratory viral multiplex panel (RVP) testing using FDA-approved in vitro diagnostic (IVD) assays, sensitivity/specificity analyses were performed by assessing each result independently to assign true/false-positive/negative calls (see Methods for details). Compared to results from RVP RT-PCR testing, the mNGS assay exhibited 93.6% (103 of 110) sensitivity, 93.8% (76 of 81) specificity, and 93.7% (179 of 191) accuracy.
Discrepancy testing and clinical adjudication (DTCA) of 14 mNGS positive-RVP negative samples using blinded chart review by two board-certified infectious diseases physician (PB and CYC) and orthogonal assays run by the California Department of Public Health Viral and Rickettsial Disease Laboratory confirmed the presence of 9 respiratory viruses missed by RVP, allowing them to be reclassified as true positives (Supplementary Table 5). Viruses detected by mNGS but not targeted by RVP were not considered false-positive results. In one case, while the original RVP and orthogonal PCR testing returned negative results, mNGS identified rhinovirus C with high confidence. A review of the viral sequences revealed 12 non-overlapping reads across the human rhinovirus C genome (Supplementary Figure 3). Cross-contamination was ruled out, as no other sample in the sequencing batch tested positive for rhinovirus. A nucleotide BLAST (blastn) search confirmed sequences with high homology (95-98% identity) to known rhinovirus C strains (Supplementary Data 1). Although the exact primer binding sites for the clinical RT-PCR assays used in the current study are unknown, we identified, for the rhinovirus C sample, the presence of mismatches in primer and probe regions from previously reported RT-PCR assays targeting the 5’-untranslated region (UTR)28,29 (Supplementary Figure 3C), which explained the detection by mNGS despite negative RT-PCR results.
Similarly, DTCA was performed on the 7 mNGS negative / RVP positive samples along with repeating the RVP assay (if possible, on a different instrument). This reassessment resulted in 5.5 samples being reclassified as true negatives (1 sample harbored two organisms adjudicated as one true negative and one false negative) (Supplementary Table 6). Compared to a composite standard that incorporates discrepancy testing and clinical adjudication, positive, negative, and overall predictive agreements of the mNGS assay were 98.7% (110.5 of 113), 98.1% (76.5 of 78), and 97.9% (187 of 191), respectively.
Detection of divergent viruses
To benchmark the capability of the modified SURPI+ pipeline for detection of novel, highly divergent viruses in silico, we created a simulated sequencing output file containing many known human viral pathogens of clinical and public health significance, including those with pandemic potential (Figure 5, left). We then removed all viral reference sequences of the same type (for example, all human polyomviruses, coronaviruses, or parainfluenza viruses) or corresponding to the same genus or species from the SURPI+ 2019 reference database (Figure 5, middle). Next, we used the SURPI+ pipeline to analyze the simulated sequencing file against both the original and “filtered” reference databases. In this analysis, 98.6% (69 of 70) of human viruses were detected at a sequencing depth of 100 reads per million (RPM) and 100% (70 of 70) at 1000 RPM based on homology to known animal or plant viruses (Figure 5, right). Of note, bunyaviruses pathogenic to humans, which are among the most divergent viruses, were still identified by translated nucleotide (amino acid) alignment to plant viruses (for example, detection of Venezuelan equine encephalitis virus based on homology to vanilla latent virus in Figure 3).