Evaluation of diagnostic accuracy of 10 serological assays for detection of SARS-CoV-2 antibodies.

doi:10.21203/rs.3.rs-65054/v1

Background

Antibody detection is essential to establish exposure, infection and immunity to SARS-CoV-2, as well as to perform epidemiological studies. The worlwide urge for new diagnostic tools to control the pandemic has led to a quick in- corporation in clinical practice of the recently developed serological assays.

Methods

We evaluated the diagnostic accuracy to detect Ig G, Ig M+A and/or IgA anti SARS-CoV-2 of 10 different assays: 3 Lateral Flow card inmunoassays, 4 en- zyme-linked inmunoabsorbent assay (ELISA) and 3 chemiluminescent particle immunoassays (CMIA). Using PCR for COVID-19 as gold standard, sensitivity, specificity, PPV, and NPV were determined. Each assay was tested in 2 groups: Positive Controls, formed by 50 sera from 50 patients with SARS-CoV-2 pneu- monia with positive PCR; Negative Controls, formed by 50 sera from 50 pa- tients with respiratory infection non-COVID-19.

Results

Sensitivity range of the 10 assays evaluated for patients with positive COVID-19 PCR was 40-77% (65-81% considering IgG plus IgM). Specificity ranged 83-100%. VPP and VPN were respectively 81-100% and 61.6-81%.

Conclusions

Results obtained varied widely among the assays evaluated.

Highest diagnostic accuracy was obtained with ELISA and CMIAs, but they last much longer.

Virology

SARS-CoV-2

antibody assays

diagnostic accuracy.

A new Coronavirus from the betacoronavirus family (subgenus Sarbecovirus) has emerged in the last few months. It has been denominated Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) because of a high phylogenetic similarity to the SARS-CoV, first identified in China's Guangdong province in 2002¹. The disease caused by SARS-CoV-2 infection has been named by international consensus COVID-19 (Coronavirus Disease 2019)².

SARS-CoV-2 was described for the first time in December 2019 in Wuhan City, Hubei Province, China³. Due to high contagiousness, virulence and issues to identify infected people, an extremely fast worldwide spreading broke. On March 11, 2020, the World Health Organization (WHO) officially declared the pandemic situation⁴. In the first week of July 2020, there were 216 affected countries, 11,514,395 people infected worldwide, and 535,185 deaths from this virus⁵.

From the beginning, a significant effort has been done from laboratories worldwide to develop and commercialize specific diagnostic assays. Consequently, viral RNA polymerase chain reaction technique (PCR SARS-CoV-2) of nasopharyngeal exudate samples was available from the very first weeks of infection. Due to its high accuracy to identify genetic material, this technique was and is still considered the gold standard in the diagnosis of the infection⁶. Despite this, PCR has several limitations as it can only asses the presence of viral material in the body but not the biological viability of these rests. On the other side, negative result cannot rule out a previous infection, nor even asses the immune status of the individual against the infection. This last can only be reported with the detection of IgG, IgM and IgA antibodies against SARS-CoV-2. Theoretically, these data would allow us to know if the individual has been in contact with the virus and if there is a current infection. This information would be much more accurate to know the risk of developing and transmitting the disease. At the individual level, it would allow actions to prevent and treat the infection. At the community level, essential information would be obtained to control the pandemic, by allowing epidemiological studies to be carried out in the general population and in specifical sources of transmission (as health workers). For this reason, the development and commercialization of techniques to detect antibodies has constituted the second step in the laboratory diagnosis of SARS-CoV-2 infection. Due to its recent and progressive availability, information about the diagnostic accuracy of these assays is not enough. Because of this, the use of one over another in the different Microbiology departments is determined most of the times by accessibility rather than scientific evidence. The publication of comparative studies between the different assays is essential to determine their diagnostic precision and, therefore, its true usefulness.

AIM

To determine the sensitivity, specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of these ten assays (3 rapid and 7 ELISA / chemiluminescence) using the nasopharyngeal exudate PCR for COVID-19 as gold standard.

Patients: Descriptive study carried out between March and May 2020 in serum from patients admitted to or treated as outpatient at La Princesa University Hospital located at Madrid, Spain. During this period, frozen serum from 50 patients who had clinical criteria for pneumonia with an average evolution time of 10 days and a positive PCR for COVID-19 in the respiratory sample (Positive Controls). The Negative Controls group consisted of frozen serum from 50 patients. These were patients randomly obtained from among those with clinical criteria of respiratory infection, treated in the same hospital prior to the start of the SARS-CoV-2 pandemic (November 2019). Serum were frozen at -20ºC at the Microbiology Department.

Antibody detection assays evaluated/Antibodies testing: The Microbiology Department of La Princesa University Hospital has 2 types of antibody detection asssays at disposal. On one hand, Lateral Flow inmunoassays cards, whose main advantage is the speed in obtaining results. Our service has had 3 different ones, WONDFO®, SGTi-Flex® and Innovitta®. On the other hand, chemiluminescence or ELISA detect IgG Antibodies, and, in some cases, they also allow the detection of IgM, Ig A and / or Ig M+A. These assays theoretically have greater diagnostic precision due to their detection methodology, but they take considerably longer. Of these second ones, the following have been evaluated: VIRCLIA (IgG and IgM+A) and ELISA (IgG and IgM+A), both from VIRCELL®; EUROIMMUN® ELISA (IgG and IgA); and the ABBOTT® chemiluminescence technique, currently only available for the detection of IgG. All assays have CE marking for in vitro diagnosis. All have been carried out in the Microbiology Department of La Princesa University Hospital.

Lateral Flow inmunoassays card

3 assays were evaluated: Wondfo® SARS-CoV-2 Antibody Test (Guangzhou Wondfo Biotech Co., Ltd), SGTi-flex® COVID-19 IgM/IgG (Sugentech, Inc.), Innovita® 2019 n-CoV Ab Test Colloidal Gold (Biological Technology Co.). In each of them, the manufacturer's instructions regarding the volume of sample and buffer to be dispensed were followed. The results of each test were visibly evaluated after 15 minutes. No result was invalidated.

Chemiluminescence

3 assays were evaluated following the manufacturer's instructions. Cut offs were calculated according to the manufacturer. Values above the cut off were considered positive.

VIRCLIA IgG Monotest, VIRCLIA IgM+A Monotest (Vircell® Spain, S.L.):

Both assays use recombinant antigens from the spicule (protein S) and the nucleocapsid (protein N). Virclia processes 24 samples determining IgG and IgM+A simultaneously in 3 hours. It is the maximum number of samples that can be processed at the same time. The manufacturer reports a global IgG sensitivity of 56% and a specificity of 99%. And for IgM+A an overall sensitivity of 63% and a specificity of 99%.

SARS CoV-2 IgG Architect (Abbott®):

This assay uses the nucleocapsid protein (protein N) as antigen. Architect has a higher work performance and is able to manage a greater number of samples in less time. The manufacturer reports a sensitivity of 86.36% in patients with an evolution of 8-13 days after the onset of symptoms and 100% in those with more than 14 days of evolution and a specificity of 99.63%.

ELISA

4 assays were evaluated. All ELISAs were performed automatically on DS2 (Alere®), which automatically calculated the optical densities of the samples and measured at 450 nm. The cut offs were calculated according to the manufacturer. Values above the cut off were considered positive. Vircell® VIRCLIA and ELISA processed sera also required manufacturer's recommended 30 minutes at 56ºC for discomplementation.

COVID-19 ELISA IgG, COVID-19 ELISA IgM + IgA (Vircell® Spain, S.L.):

Both Vircell techniques use recombinant spike (protein S) and nucleocapsid (protein N) antigens. In 4 hours it is capable of evaluating 92 samples, determining both IgG and IgM+A simultaneously. It is the maximum number of samples that can be processed at the same time. The manufacturer reports an overall sensitivity for IgG of 58% and a specificity of 98%. For IgM+A an overall sensitivity of 66% and a specificity of 99%.

EUROIMMUN® ELISA Anti SARS-Co-V2 IgG, and Anti SARS-Co-V2 IgA (Medizinische Labordiagnostika AG):

They use a recombinant spike protein antigen (protein S). The manufacturer reports a sensitivity for IgG of 33% in patients with <10 days after the onset of symptoms and 80% in patients with > 10 days after the onset of symptoms and a specificity of 98.5%. And for IgA a sensitivity of 50% in patients with <10 days after the onset of symptoms and 100% in patients with> 10 days after the onset of symptoms and a specificity of 92.5%. For IgG, a maximum of 92 samples can be studied at the same time. Time until results are 3 hours. For Ig A, only 45 samples can be processed at a time. The manufacturer indicates if a larger number is processed, decreases in adsorption can happen. Time until the result are 3 hours.

Given the exceptional situation in which this pandemic is developing, the evaluation of the assays has been limited by their availability, as the development and commercialization of specific reagents for SARS-CoV-2 has not been simultaneous. It has been considered a priority to use the same serum when evaluating all the assays, since, in our opinion, this increases the validity of the results obtained. For this reason it was necessary to preserve the serum extracted from patients. In our case it was frozen, being defrosted with each assay examined. Due to the high number of them to be evaluated, this freezing and thawing process was carried out more than what, in our opinion, is desirable. We believe that this may limit the quality of the results obtained. The results are summarized in Table 1.

Immunochromatography Lateral Flow type:

Wondfo® SARS-CoV-2 Antibody: Total antibody sensitivity was 76%, with a specificity of 100%, PPV of 100% and NPV: 81%.

SGTi-flex® COVID-19 IgM / IgG, which separates the two antibodies: in the case of IgG, a sensitivity of 40%, specificity of 100%, PPV: 100%, NPV: 62.5% was obtained. For IgM, sensitivity 70%, specificity 90%, PPV: 87.5%, NPV: 75%. If we consider an Ig of the two positive (IgG or IgM), the sensitivity rises to 74%, the specificity to 90%, PPV of 88.1% and NPV of 77.6%.

Innovita® 2019 n-CoV Ab Test Colloidal Gold also discriminates between the two antibodies. The results obtained for IgG were: sensitivity of 44%, specificity of 98%, PPV: 95.7%, NPV: 63.6%. The results obtained for IgM were: sensitivity: 52%, specificity: 100%, PPV: 100%, NPV: 67.6%. The results obtained for IgG or IgM, considering one of the two Ig positive, were as follows: sensitivity: 58%, specificity: 98%, PPV: 96.7%, NPV: 70%.

Chemiluminescence

VIRCLIA IgG Monotest Vircell®: sensitivity of 48%, specificity of 96%, PPV: 92%, NPV: 65%.

VIRCLIA IgM+A Monotest Vircell®: sensitivity 63%, specificity 96%, PPV: 94%, NPV: 72%. If we consider a sample as positive if it has IgG and/or IgM+A, the results were as follows: 65% sensitivity, 94% specificity, PPV: 91%, NPV: 73%.

SARS CoV-2 IgG Architect Abbott®: It should be noted that, despite not being advised by the manufacturer, the technique was tested with discomplementated serum because it was the last assay to be available and previously the serum had been discomplemented as it was an essential requirement to analyze Vircell's ELISA and CLIA techniques. The results obtained for IgG were a sensitivity of 52%, specificity of 100%, PPV: 100%, NPV: 68%.

ELISA

COVID-19 ELISA IgG Vircell®: 65% sensitivity, 96% specificity, PPV: 94% and NPV: 73%.

COVID-19 ELISA IgM+A Vircell®: sensitivity 77%, specificity 83%, PPV: 82%, NPV: 78%. If we consider a sample as positive if it has ELISA IgG and/or IgM+A, the results were as follows: sensitivity: 81%, specificity: 81%, PPV: 81%, NPV: 81%.

EUROIMMUN® ELISA Anti SARS-Co-V2 IgG: sensitivity of 37.8%, specificity of 100%, PPV: 100%, NPV: 61.6%.

EUROIMMUN® ELISA Anti SARS-Co-V2 IgA: a fail dispensing the stopping solution during the performance of Negative Controls group, caused the invalidation of the assay. Therefore, only sensitivity data are presented for this antibody (IgA) and the IgG+IgA group. The sensitivity for IgA was 88.9%. If we consider a sample as positive if it has an ELISA IgG and/or IgA, as in the previous cases, it should be noted that only Positive Controls have been taken into account because Negative Controls we do not have IgA. In this case, the sensitivity is 88.9%. The IgA and IgG+IgA sensitivity results coincide because there is no patient who has positive IgG and negative IgA.

Current diagnosis of active SARS-CoV-2 infection is established with clinical data (imaging tests, symptoms, epidemiological context) and PCR of respiratory samples. Antibody detection assays are destined to assess the immunological situation against the virus of the individual and, by extension, of the community. As the development of acquired defenses against the virus provides security, the first target for these new techniques is to be highly specific. The consequences of a false positive are worse than those of a false negative, since the unfounded belief of presenting immunity carries an obviously much higher risk. Therefore, we believe that it is important to analyze the results obtained from this prism.

As expected, Lateral Flow immunoassays showed worse diagnostic accuracy than ELISA and chemiluminescence test, due to their lower sensitivity. In our study, the test that showed the best result was the one that detected total antibodies, Wondfo® SARS-CoV-2 Antibody, with a sensitivity of 76%, and a specificity of 100%, PPV of 100% and NPV of 81%. The rest presented similar results, although a little lower. Innovita® 2019 n-CoV Ab Test Colloidal Gold presented significantly worse values (sensitivity 58%, specificity 98%, PPV: 96.7%, NPV: 70%). These results are in line with previously published series⁷, in which, however, results of Innovita® 2019 n-CoV Ab Test Colloidal Gold are comparable to the others tests.

Finally, when evaluating this group of test 2 facts should be considered. They provide a quick result and, furthermore, they can be performed by less-experienced staff. Both advantages make them more useful in certain clinical situations. However, outside of these contexts or the lack of availability, we believe that their use is not justified against chemiluminescence and ELISAs, taking into account their higher diagnostic accuracy and larger capacity.

ELISA assays demonstrated specificity >95% in most cases, being higher than 99% for Wondfo, EUROIMMUN® ELISA Anti SARS-Co-V2 IgG, and SARS CoV-2 IgG Architect Abbott®. EUROIMMUN ELISA IgG and IgA was, in global terms, the ELISA with the highest sensitivity (88.9%) and specificity (100%). IgA values were high, in line with previously published results^8,9. However, the sensitivity for the isolated detection of IgG was very low (37.8%), in contrast to other studies (67- 93.8%)^8,9. Something similar happened with COVID-19 ELISA IgG Vircell®. Facing Kohmer et al.⁸ results (100% sensitivity, 95.2% specificity), in our study the sensitivity was 65%, maintaining a high specificity (96%) with 94% PPV.

Despite a high specificity, in general, the sensitivity of chemiluminescence assays was lower tan ELISAs. However, it should be considered that Architect's results were significantly limited by the discomplementation and freeze-thaw process. We expect this results to be better when performed in optimal conditions. In addition, it is important to consider that Architect is the equipment with the largest capacity (it processes the greatest number of samples in the least time). This, together with the fact that it does not need a discomplementation process, adds speed to the test and reduces the response time by the Microbiology Department. A last advantage is that it also permits other determinations to be carried out in parallel on the same sample.

When analyzing results obtained in this study, we believe it is important to take several aspects into account. On one hand, during the infection, the identification of antibodies against SARS-CoV-2 is time-dependent. From the seventh day after the onset of symptoms, they are detected in 40% of patients. This percentage rises significantly, to more than 90%, from day 14¹⁰. In our case, the sera of the Positive Controls were obtained from patients with an average clinical evolution of 10 days, which can mainly affect the sensitivity of the techniques. On the other hand, we are aware that the sample size is not especially large, what may limit the results obtained and may justify the differences observed with previous studies. Finally, the descriptive study methodology limits the interpretation of the observed differences.

In any case, as it has been highlighted in the introduction, there is a lack of evidence regarding the diagnostic accuracy of antibody detection assays. In the current pandemic context with high morbidity and mortality rates, scientific data obtained in a real clinical context regarding the real accuracy of these diagnostic tests is highly needed. Our study is, of those published, one which evaluates at the same time a larger number of tests. This provides a direct comparison of their results in the same sample. Furthermore they were evaluated in a real clinical context such as the one experienced in the last months, with the limitations that were found. We believe that this helps to reflect the real usefulness of these tests.

Regardless of these results and those obtained to date, larger comparative and randomized studies are now more than needed, as this pandemic seems to be a long way to go.

Compliance with Ethical Standards:

Funding:

The authors did not receive any fundings to do the study.

Conflict of Interest:

Authors do not have conflict of interest.

Ethical approval:

For validate a new technique to incorporate it into the laboratory normal routine, we are allowed to take samples from our laboratory sample collection. At Hospital Universitario La Princesa of Madrid, Spain, we do not need to pass under Ethic Comission.for this case either we do not need to ask for informed consent to realiza this type of study.

Zhu N Zhang D, Wang W, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med 2020; 382: 727-33.
World Health Organization. Coronavirus disease 2019 (COVID-19) Situation Report – 78 2020.
Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020; 395: 497-506.
Organización Mundial de la Salud. Alocución de apertura del Director General de la OMS en la rueda de prensa sobre COVID-19 celebrada el 11 de marzo de 2020; https://www.who.int/es/dg/speeches/detail/whodirector-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-20204.
Coronavirus Resource Center. John Hopkins University Medicine.
Rabi FA, Al Zoubi MS, Kasasbeth GA, et al. 2020 SARS-CoV-2 and Coronavirus Disease 2019:What we know so far. Patogens9:doi:10.3390/patogens9030231.
Whitman JD, Hiatt J, Mowery CT et al. Test performance evaluation of SARS-CoV-2 serological assays. Preliminary report.
Kohmer N, Westhaus S, Rühl C, et al. Clinical performance of SARS-CoV-2 IgG antibody and potential protective immunity. bioRxiv.http:doi.org/10.1101/2020.05.08.085506. May 10, 2020.
Lassaunière R, Frische A, Harboe ZB et al. Evaluation of nine commercial SARS-CoV-2 immunoassays. En: medRxiv preprint doi: https://doi.org/10.1101/2020.04.09.20056325.this version posted April 10, 2020.
Zhao J, Yuan Q, Wang H, Liu W, Liao X, Su Y, et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis (in press). doi:https://doi.org/10.1093/cid/ciaa344.

Table 1: Comparative table of serology techniques COVID-19 Microbiology Department

Técnica

Sensibilidad

Especificidad

VPP

VPN

Wondfo (n=100)

75

100

81

SGTi (n=100)

IgG

IgM

IgG+IgM

40

70

74

100

90

100

87.5

88.1

62.5

75

77.6

Innovita (n=100)

IgG

IgM

IgG+IgM

44

52

58

98

100

98

95.7

100

96.7

63.6

67.6

70

VIRCLIA Vircell (n=96)

IgG

IgA+M

IgG+IgA+M

48

63

65

96

94

92

94

91

65

72

73

ELISA Vircell (n=96)

IgG

IgA+M

IgG+IgA+M

65

87

81

96

83

81

94

82

81

73

78

81

ELISA Euroimmun (n=90)

IgG

IgA

IgG+IgA

37.8

88.9

100

-

100

-

61.6

-

IgG Architect (n=100)

IgG

52

100

68

Evaluation of diagnostic accuracy of 10 serological assays for detection of SARS-CoV-2 antibodies.

Status:

Journal Publication

Version 1

Abstract

Introduction

Methods

Results

Discussion

Declarations

References

Tables

Status:

Journal Publication

Version 1