Evaluating The Performance Characteristics of Five Lateral Flow Assays for The Detection of The SARS- CoV-2 Nucleocapsid Antigen

doi:10.21203/rs.3.rs-951445/v1

Introduction: In response to the COVID-19 pandemic, lateral flow assays (LFAs) for the detection of SARS-CoV-2 antigen have been proposed as a complementary option to the more costly and time consuming reverse-transcriptase polymerase chain reaction (RT-PCR). We assessed five commercially available SARS-CoV-2 antigen detecting LFAs (ASSUT EUROPE (Rome, Italy), Besthree (Taizhou, China), Encode (Zhuhai, China), Fortress (Antrim UK), and Hughes Medical (Buckinghamshire, UK), using samples collected from hospitalised individuals with COVID-19 and compared these results against established RT-PCR assays with the aim of estimating test performance characteristics.

Method: We performed a diagnostic accuracy study of the LFAs on 110 inpatients with confirmed COVID-19 and 75 COVID-19 negative control participants. Assay evaluation was performed using a modified version of each manufacturer’s protocol allowing for parallel testing of a single sample on multiple assays.

Results: Half (46%, 51) of the patient cohort were in the higher risk age category (>70 years old). Community acquisition of SARS-CoV-2 accounted for 77% (85) of our positive cohort with the remaining 23% (25) resulting from nosocomial transmission. The sensitivity of the LFAs ranged from 64% (95% CI 53-73) to 76% (95% CI 65-85). Of the assays tested, Fortress performed best with a sensitivity of 76% (95% CI 65-85). Specificity based on testing SARS-CoV-2 volunteers was high across all assays with four of the five achieving 100%

Conclusion: SARS-CoV-2 antigen detecting LFAs may complement RT-PCR testing to facilitate early diagnosis and provide community testing strategies for identification of patients with COVID-19, however we find suboptimal test performance characteristics across a range of commercially available manufacturers, below WHO and MHRA pre-set sensitivity performance thresholds. With such variation in sensitivity between LFAs and PCR testing and between assay brands, we advise caution in the deployment of LFAs outside of environments with clinical oversight.

Infectious Diseases

Virology

COVID-19

lateral flow assay

SARS-COV-2

In response to the COVID-19 pandemic, testing capacity for clinical diagnostics and for screening have been stretched at a global level. Increasing demand has meant that novel testing platforms and protocols are required, with an expectation that they will play a role in both individual healthcare and community testing programmes, national population studies, and international epidemiological guidance.

Reverse transcriptase polymerase chain reaction (RT-PCR) is considered the gold standard test for the detection of SARS-CoV-2 infection, but has a significant processing time requiring specific laboratory infrastructure.(2) Whilst some advances have been made in laboratory-free RT-PCR platforms, these are still not in widespread use in large scale community testing.(3, 4) A faster and more portable alternative is required and lateral flow assays (LFAs) for the detection of SARS-COV-2 antigen have been proposed as a complementary option.(5) During the rapid development of SARS-CoV-2 antibody lateral flow assays, significant inter-manufacturer variation was seen, with some performing particularly poorly (6–8). If antigen LFAs are to be used to complement PCR testing, evaluation for any similar inter-manufacturer variation must be analysed.

Whilst laboratory based assessment of these rapid antigen LFA assays has occurred, evaluation of assays in the setting of intended use with both sample and operator skill-sets who are representative of clinical and community testing sites remains unstudied. In the UK, testing has focused on one specific manufacturer, procured at a national level,(9, 10) whilst competitor assays also remain unstudied. In this analysis we assessed five commercially available SARS-CoV-2 antigen LFAs using samples collected from hospitalised individuals with COVID-19 and compared these results against established RT-PCR assays with the aim of estimating test performance characteristics.

Study Design

We performed a prospective diagnostic accuracy study, independent of manufacturers, to evaluate five commercially available SARS-CoV-2 antigen point-of-care lateral flow assays between November 2020 and February 2021. This was a single centre study, based in a large urban hospital, comprising 110 in-patients and 75 SARS-CoV-2 negative controls. Clinicians identified participants, collected clinical details and performed nasopharyngeal swabs for inoculation of the lateral flow assays.

SARS-CoV-2 antigen LFAs were identified through our procurement team as viable alternatives to the nationally procured LFA.(9, 10) Initial assessment was performed with all assays required to meet the following criteria: a cassette-based design with visual read-out result in less than 20 minutes, and manufacturer reported sensitivity of >95% and specificity of >95%, availability for purchase in the United Kingdom, and accreditation via a CE mark. Five assays were selected for in-depth evaluation; ASSUT EUROPE (Rome, Italy), Besthree (Taizhou, China), Encode (Zhuhai, China), Fortress (Antrim UK), and Hughes Medical (Buckinghamshire, UK). The target of all selected assays was the SARS-CoV-2 nucleocapsid protein antigen. All evaluated assays were approved for nasopharyngeal sampling. As all assays were CE marked, this service evaluation was undertaken as a validation of these in vitro diagnostic devices in line with the UK Standards for Microbiological Investigation structure.(11) The service evaluation was registered with the hospital Clinical Governance Department, and all participants provided informed consent for use of their samples for the purpose of assay validation.

SARS-CoV-2 Positive Cohort

Patients confirmed to have COVID-19 were identified via a centralised daily report of all SARS-CoV-2 results performed in the previous 24 hours within our Trust. Patients’ electronic health records were examined to assess for suitability according to the inclusion criteria; current inpatient at Chelsea and Westminster Hospital NHS Foundation Trust, positive RT-PCR test for SARS-CoV-2 within the previous 72 hours, symptom onset <14 days (in those with symptoms), ability to provide informed consent. Patients were excluded if; age <18 years old, unable to consent due to pre-existing medical condition or acute alteration in an individual’s condition, symptom onset > 14 days, PCR positive for SARS-CoV-2 > 72 hours, asymptomatic without a negative swab within the previous 14 days.

SARS-CoV-2 Negative Control Cohort

Negative controls were recruited from hospital staff, and students from Imperial College School of Medicine. In line with NHS England guided testing of healthcare professionals, all individuals recruited undertook routine twice weekly LFA testing using the aforementioned nationally procured Innova assay. Additionally, all recruited members from the Haematology, Oncology and HIV teams were undergoing weekly PCR testing in keeping with local policy on health care professionals working with immunocompromised patients. All individuals recruited as negative controls were asymptomatic at time of testing and had a negative Innova LFA or RT-PCR within the previous 72 hours.

Evaluation Protocol

Assay evaluation was performed using a modified version of each manufacturer’s protocol. This allowed for parallel testing of a single sample on multiple assays. These modifications included an increase in the quantity of extraction buffer used from 300µl to 600µl and the use of a single extraction solution with sodium azide 0.09% preservative. This protocol deviation was evaluated using five positive and five negative ACON Biotech5 (Hangzhou, China) control swabs for SARS-CoV-2 on each of the five assays. The results of this preliminary assessment were 100% sensitivity and 100% specificity on each assay against these laboratory controls, despite modifications in testing protocol.

The nasopharyngeal swabbing procedure followed manufacturer protocol and all samples were taken by a trained member of the research team to ensure consistent sampling technique. All assays with a visible control line were deemed valid. Chromatographic results on each assay were read by a single member of staff trained in the use of lateral flow assays including interpretation of results. In the case of an inconclusive result, a second member of the research team was consulted to reach a final decision.

Statistical Analysis

Each LFA result was compared to the RT-PCR result. Cycle threshold values were not available for all RT-PCR samples and thus absent results were imputed via multiple imputations using chained equations.(12) This technique facilitates the use of missing data whilst propagating the uncertainty in estimating missing data, thus minimising selection bias. The dataset is imputed 20 times, using differing random seeds, resulting in 20 different possible values for each missing value. Each imputed dataset’s performance is then aggregated, and the spread is calculated and displayed for the appropriate metric.

To understand the effects of the RT-PCR cycle threshold (CT) and temporal delay between swabbing on the accuracy of LFAs, receiver operating characteristic (ROC) curves were constructed for each assay.

Patient/negative control cohort

As detailed, the SARS-CoV-2 positive cohort consisted of 110 inpatients, with each assay tested against 75 of these individuals. The cohort was almost evenly split by gender, 52% (57) male and 48% (53) female and ages ranged from 26 -100 years old, with a mean age of 67 years. Almost half (46%, 51) of the cohort were in the higher risk age category (>70 years old). Community acquisition of SARS-CoV-2 accounted for 77% (85) of our positive cohort with the remaining 23% (25) resulting from nosocomial transmission. (See Table 1)

Table 1

Patient Cohort – Demographics and Data
Patient Cohort
Gender	No.	Age	No.	Acquisition Source	No.	Oxygen Delivery Method	No.	Outcome at data collection completion	No.
Male	57	25-34	6	Community	85	Room Air	55	Discharged	67
Female	53	35-44	6	Hospital	25	Nasal O2	28	Inpatient	17
		45-54	21			Non Rebreathe/ Venturi	14	Died	26
		55-64	13			NIV	13
		65-74	21
		75-84	21
		85-94	20
		95-100	2

Hypoxia requiring supplemental oxygen was a common symptomatic feature. At the point of LFA sampling, 50% (55) of patients were not requiring supplemental oxygen, 38% (42) were receiving oxygen via nasal cannula or by non-rebreathe and Venturi face masks, while 12% (13) were receiving positive pressure non-invasive ventilation. At the completion of data collection 61% (67) of the cohort had been discharged from hospital, 15% (17) remained in hospital undergoing rehabilitation, and 24% (26) of the cohort had died. (See Table 1)

In total, 75 negative control individuals were enrolled in the validation process. The cohort was 51% (38) male and 49% (37) female with an age range of 21 - 64 years (mean: 34 years of age). (See Table 2)

Table 2

Negative Control Cohort – Demographics
Negative Control Cohort
Gender	No.	Age	No.
Male	38	25-34	20
Female	37	35-44	23
		45-54	18
		55-64	12
		65-74	2
		75-84	0
		85-94	0
		95-100	0

LFA test performance characteristics

The sensitivity of the LFAs ranged from 64% (95% CI 53-73) to 76% (95% CI 65-85). Of the assays tested, Fortress performed best with a sensitivity of 76% (95% CI 65-85).

Specificity based on testing SARS-CoV-2 volunteers was high across all assays with four of the five achieving 100% (Assut test 100% (95% CI 95-100), Besthree 100% (95% CI 95-100), Fortress 100% (95% CI 95-100) and Hughes 100% (95% CI 95-100), with Encode specificity at 99% (95% CI 92-100). (See Table 3)

Table 3

– LFA Performance Characteristics
LFA Performance Characteristics
Brand	Sensitivity	95%CI	Specificity	95% CI
Fortress	76%	(65-85)	100%	(95-100)
Hughes	72%	(61-81)	100%	(95-100)
Encode	71%	(60-80)	99%	(92-100)
Besthree	68%	(57-77)	100%	(95-100)
Assut	64%	(53-73)	100%	(95-100)

Correlation of RT-PCR cycle threshold and LFA test performance characteristics

The sensitivity of each LFA was explored as a surrogate marker of the RT-PCR cycle threshold using ROC curves. No particular test demonstrated reliance on a specific RT-PCR cycle threshold to achieve positivity. (See supplementary figure 1)

Impact on time from RT-PCR to LFA test performance characteristics

Time from RT-PCR swab to LFA swab was examined with 46% (51) of participants sampled within 24 hours, 34% (38) within 48 hours and 20% (21) within 72 hours of a positive PCR test. No patients were sampled at greater than 72 hours of a confirmed RT-PCR result. None of the LFAs showed significant dependence on the time between PCR tests and LFA swab. (See supplementary figure 2)

Our study presents the largest independent inter-manufacturer evaluation to date of commercially available SARS-CoV-2 antigen lateral flow assays on individual patients, sampled in real time. While other studies examining antigen detecting LFAs have utilised recombinant nucleoproteins, viral culture samples and stored respiratory samples, our study demonstrates significant inter-manufacturer sensitivity (and particularly variability in time-from-PCR-positive) using clinical in situ methodology.

The range of manufacturer reported sensitivity values (78–99%) differed substantially from the sensitivity determined in our evaluation (64–76%), highlighting the importance of independent assessment of product performance in real clinical settings with representative clinician-user skill sets. Our findings are corroborated by results from a PHE Porton Down study which showed a drop in sensitivity of the UK government procured Innova antigen LFA(9, 13) when real-world samples, and operators, are used. The between-assay variation in sensitivity we demonstrated also suggests that interchangeability between different manufacturers could significantly impact accuracy in testing.

Based on our evaluation, none of the five assays meet the WHO and MHRA’s priority target product profiles for COVID-19 diagnostics, where acceptable sensitivity is deemed ≥ 80% and specificity ≥ 97%. Thus, none can be considered as a replacement for laboratory-based RT-PCR.(14, 15) They may, however, represent a useful diagnostic adjunct in the current SARS-CoV-2 pandemic, its potential future endemic presence, and in regions where RT-PCR testing remains limited. With any highly infectious condition, time sensitive decisions regarding patient placement in hospital are required and laboratory-based RT-PCR cannot always meet this demand. SARS-CoV-2 antigen LFAs may be best suited to situations where confirmatory RT-PCR testing can be performed once the patient is in a more secure environment.

There are several limitations to our study including deviation from the manufacturer protocol, although our verification with set laboratory controls indicated this had no effect. Secondly, we undertook non-contemporaneous RT-PCR and LFA sampling. This decision was pragmatic, as unlike antibody LFAs which can be verified using excess/waste serum, antigen LFAs require nasal or nasopharyngeal samples for which there is no excess. Tolerability of repeated nasopharyngeal swabs impacts the ability to perform head-to-head comparisons on individual patients. Variation in quantity and quality of sampling compared to the initial RT-PCR swabs may impact on the reported assay sensitivity, but we mitigated this by using clinicians experienced in nasopharyngeal swabbing, to obtain all samples. There remains a possibility that an individual’s antigen positive status may have waned after their initial RT-PCR positive result, but our analysis of the impact of time-between-sampling demonstrated minimal impact of this variable, implying that this is unlikely to have had a significant impact on the LFA poor sensitivity. Finally, RT-PCR testing was performed on a number validated platforms, and CT was not available to enable comparison for all assays (some were tested on the dnaNudge platform).(3) Reassuringly, our investigation of correlation between RT-PCR CT and LFA positivity demonstrated little effect here either, thus this is unlikely to have had a significant impact on the LFA poor sensitivity.

In conclusion, while SARS-CoV-2 antigen LFAs may complement PCR testing to facilitate early diagnosis and provide community testing strategies for identification of patients with COVID-19, we find suboptimal test performance characteristics across a range of commercially available manufacturers, below pre-set sensitivity performance thresholds.(14, 15) With limited access to healthcare facilities due to pandemic restrictions, the ability for patients to perform home sampling greatly reduces the demand on hospital services and the need for unecessary travel to, and footfall within, clinical settings. Demand for a rapid, self-administered point of care diagnostic tool remains paramount, however with such variation in sensitivity between LFAs and PCR testing and between assay brands, we advise caution in the deployment of LFAs outside of environments with clinical oversight

Funding

JH received funding in the form of a fellowship from the Joint Research Committee of CW+ and the Westminster Medical School Research Trust.

Authors’ contributions

All listed authors made substantial contributions to the conception or design of the work; or to the acquisition and analysis of data for the work; and drafting the work or revising it critically ahead of submission for publication. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

CRediT roles

JH – data curation, formal analysis, investigation, writing – original draft (including figures); SJCP – methodology, data curation, writing – review and editing; AAH – data analysis, statistical methodology, writing; NM – conceptualization, supervision; PR – conceptualization, data curation, laboratory analysis, writing; RJ – conceptualization, writing – review and editing; GWD – conceptualization, supervision, project administration, writing – review and editing; MR - conceptualization, data curation, writing – review and editing; LSPM – conceptualization, supervision, writing – review and editing, project administration.

Competing Interests Declaration

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare; JH received research funding from CW+ Charity and the Westminster Medical School Research Trust and received honoraria from Gilead (2020). SJCP has received a research grant from the Scientific Exploration Society/Viscount Gough, outside the submitted work. NM has received speaker fees from Beyer (2016) and Pfizer (2019) and received educational support from Eumedica (2016) and Baxter (2017). LSPM has consulted for or received speaker fees from bioMerieux (2013-2021), Pfizer (2018-2021), Eumedica (2016–2021), DNAelectronics (2015-18), Dairy Crest (2017–2018), Umovis Lab (2020-2021), Shionogi (2021), and Pulmocide (2021), and received research grants from the National Institute for Health Research (2013-2019), CW+ Charity (2018-2021) and LifeArc (2020-2021). RJ has received honoraria, speaker fees, travel support and/or research grant funding from Gilead, ViiV Healthcare, BMS, Abbvie, Janssen and Merck. All other authors have no competing interests to declare.

Ethical Approval

This evaluation was commissioned as a service evaluation by the COVID Testing Committee of Chelsea & Westminster NHS Foundation Trust, on 20 October 2020. The study was reviewed by the Chelsea & Westminster NHS Foundation Trust Research and Development Office and deemed a verification of a CE marked in vitro diagnostic test; therefore, informed consent was required to participate but this was not required to be in written format. Aggregated data was analysed under the Health Service Control of Patient Information Regulations (2002) general notice that patient data for a COVID-19 purposes may be used for research as stated by the UK Secretary of State for Health and Social Care. The study was conducted in accordance with relevant guidelines and regulations including the Declaration of Helsinki.

Data availability

The data analysed during the current study and further details on the assays are available from the corresponding author (LSPM; [email protected]) on reasonable request, as long as this meets local ethical and research governance criteria

WHO. WHO Coronavirus (COVID 19) Dashboard [Available from: https://covid19.who.int/.
WHO. Establishment of PCR in Developing Countries. https://apps.who.int/iris/bitstream/handle/10665/205020/B4745.pdf?sequence=1; 2011.
Gibani MM, Toumazou C, Sohbati M, Sahoo R, Karvela M, Hon TK, et al. Assessing a novel, lab-free, point-of-care test for SARS-CoV-2 (CovidNudge): a diagnostic accuracy study. Lancet Microbe. 2020;1(7):e300-e7.
Moore LSP. Near-patient SARS-CoV-2 molecular platforms: new-old tools for new-old problems. Lancet Respir Med. 2020;8(12):1161-3.
Dinnes J, Deeks JJ, Adriano A, Berhane S, Davenport C, Dittrich S, et al. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. 2020;8:CD013705.
Pallett SJC, Rayment M, Patel A, Fitzgerald-Smith SAM, Denny SJ, Charani E, et al. Point-of-care serological assays for delayed SARS-CoV-2 case identification among health-care workers in the UK: a prospective multicentre cohort study. Lancet Respir Med. 2020;8(9):885-94.
Moshe M, Daunt A, Flower B, Simmons B, Brown JC, Frise R, et al. SARS-CoV-2 lateral flow assays for possible use in national covid-19 seroprevalence surveys (React 2): diagnostic accuracy study. BMJ. 2021;372:n423.
Flower B, Brown JC, Simmons B, Moshe M, Frise R, Penn R, et al. Clinical and laboratory evaluation of SARS-CoV-2 lateral flow assays for use in a national COVID-19 seroprevalence survey. Thorax. 2020;75(12):1082-8.
University PO. Preliminary report from the Joint PHE Porton Down & University of Oxford SARS-CoV-2 test development and validation cell: Rapid evaluation of Lateral Flow Viral Antigen detection devices (LFDs) for mass community testing:. 2020.
Mahase E. Covid-19: Innova lateral flow test is not fit for "test and release" strategy, say experts. BMJ [Internet]. 2020.
PHE. UK Standards for Microbiology Investigations Q1: evaluations, validations and verifications of diagnostic tests. https://www.gov.uk/government/publications/smi-q-1-commercial-and-in-house-diagnostic-tests-evaluations-and-validations2017.
White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. 2011;30(4):377-99.
Iacobucci G. Covid-19: Mass population testing is rolled out in Liverpool. BMJ. 2020;371:m4268.
WHO. Target product profiles for priority diagnostics to support response to the COVID-19 pandemic v.1.0. https://cdn.who.int/media/docs/default-source/blue-print/who-rd-blueprint-diagnostics-tpp-final-v1-0-28-09-jc-ppc-final-cmp92616a80172344e4be0edf315b582021.pdf?sfvrsn=e3747f20_1&download=true; 2020.
MHRA. Target Product Profile: Point of Care SARS-CoV-2 Detection Tests. https://www.gov.uk/government/publications/how-tests-and-testing-kits-for-coronavirus-covid-19-work/target-product-profile-point-of-care-sars-cov-2-detection-tests; 2020.

No competing interests reported.

Evaluating The Performance Characteristics of Five Lateral Flow Assays for The Detection of The SARS- CoV-2 Nucleocapsid Antigen

Status:

Version 1

Abstract

Introduction

Methods

Study Design

SARS-CoV-2 Positive Cohort

SARS-CoV-2 Negative Control Cohort

Evaluation Protocol

Statistical Analysis

Results

Patient/negative control cohort

LFA test performance characteristics

Correlation of RT-PCR cycle threshold and LFA test performance characteristics

Impact on time from RT-PCR to LFA test performance characteristics

Discussion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1