Assessment of a Comparative Bayesian-Enhanced Population-Based Decision Model for COVID-19 Critical Care Prediction in Dominican Republic Social Security Affiliates

doi:10.21203/rs.3.rs-991050/v1

Introduction: The novel coronavirus disease 2019 is a major health concern worldwide. The objective was to develop a Bayesian model to predict critical outcomes in patients with COVID-19.

Methods: Sensitivity and specificity were obtained from previous meta-analysis. Using the IVC-COV2 index as pretest probability, likelihood ratios were integrated in a Fagan nomogram for posttest probabilities, generating IVC-COV2 + NEWS and CURB-65 scores values. Absolute and Relative Diagnostic Gains (ADG, RDG) were calculated.

Results: The IVC-COV2 index was derived from a population of 1,055,746 individuals and based on mortality divided into high (71.97%) Intermediate (26.11%) and low (1.91%) risk groups. Integrating the IVC-COV2 intermediate + NEWS≥5 and CURB-65 >2 score models found that the Number Needed to Diagnose demonstrated a slight improvement for the CURB-65 model [2.00 (2) vs 2.71(3)]. When comparing diagnostic gains, no statistical differences were found on the IVC-CoV2 NEWS model compared to the CURB-65 model in both LR+ (P=0.62) and LR- (P=0.95).

Conclusion: This mathematical model proposed that the combination of a IVC-COV2 Intermediate score plus NEWS or CURB-65 scores yields superior results and a greater predictive value for severity of illness. To our knowledge this is the first population-based/mathematical model for COVID-19 Critical Care decision making.

Critical Care & Emergency Medicine

COVID-19

population-based

risk

Intensive Care

ICU

NEWS

CURB-65

The novel coronavirus disease 2019 (COVID-19) caused by the severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), has become a major health concern worldwide at the time of this study with more than 1 million direct deaths according to the World Health Organization (1). Respiratory failure is the leading cause of mortality in patients with COVID-19 (2). Myocardial injury, kidney or liver injury, and multi-organ dysfunction are among the other complications leading to death (3). Several prognostic factors, such as older age, male gender, presence of comorbidities, and smoking, have been found to be associated with severe disease or death (4–7).

The National Institute for Health and Care Excellence (NICE) in its guidelines for the management of COVID-19 recommended the use of NEWS2 in critical care (8, 9). National Early Warning Score (NEWS) is a standardized clinical scoring system developed to improve detection of deterioration in acutely ill patients (Fig. 1) and it’s based on a logistic regression model designed to predict in-hospital patient mortality within 24 hours of a set of vital signs observation (10). Originally it consisted in evaluation of pulse rate, respiratory rate, blood pressure, temperature and oxygen saturation. NEWS-2 is the latest version of the NEWS score, which adds new onset of confusion to the parameters and then 2 points are added for people requiring supplemental oxygen to maintain their recommended oxygen saturation. A recent in-hospital study showed that NEWS2 did not appear to add predictive value over NEWS, even in patients with type 2 respiratory failure (11–14), for the purposes of our model we opted to integrate sensitivities and specificities of the original NEWS score.

Other scoring systems, like the CURB-65 (Fig-2) are widely used in predicting 30-day mortality in community-acquired pneumonia (15). CURB-65 has also been found to be useful in predicting 14-day mortality in hospital-acquired pneumonia (16). Recently a simple predictive tool for estimating the risk of 30-day mortality, and to stratify patients with COVID-19 was developed integrating CURB-65 (17).

The Complex Vulnerability Index (IVC-COV2 for its abbreviation in Spanish) (18) is a population-based index designed by Dominican health authorities which takes into account sex, age, and comorbidities in order to assess how much risk a specific patient has to suffer a critical outcome if infected with COVID-19, for the purposes of this study patients were stratified as low, intermediate, and high risk based on the DR-IVC-COV2 score. The use of clinical scoring systems to predict severe disease and mortality in patients with COVID-19 should be investigated further in larger prospective studies.

Bayesian statistics have been utilized to evaluate uncertainty using mathematical probability instruments. Our group has been studying Bayesian statistics in clinical/ medical decision making and as a “data recycling” tool and can be used to compare the diagnostic quality of different serum biomarkers, with a methodology that outputs the probability of an event based on criteria related to the specific event (19–26). Our group has developed a simple mathematical method for interpreting diagnostic impact called “Bayesian Diagnostic Gains (BDG)”, where relative diagnostic gain (RDG) and absolute diagnostic gain (ADG) were calculated based on the differences deducted from pre and posttest probabilities (ADG = post-test – pre-test) and (RDG = 100 × post-test – pre-test/Pre-test). This particular study is our first attempt at integrating BDGs in a COVID-19 Critical Care prediction multi-item model.

Objective

To develop a hybrid mathematical model that assists in predicting critical care disposition in patients with COVID-19 by means of Bayesian statistics assessing comparatively the IVC-COV2 score integrated with both NEWS and the CURB-65 score.

Calculations for Sensitivity and specificity were obtained from previous meta-analysis (11, 15). The IVC-COV2 Index is a population-based tool, developed by the Dominican Republic Health and Risk Superintendence (SISALRIL) as a tool to identify COVID-19 at risk individuals. The IVC-COV2 Index (18) was created from a captive insured population and derived by recursive partitioning from for dimensions (Table-1) 1) Co-morbidities 2) Age 3) Gender 4) Social/ Family by utilizing this formula 𝐼𝑉𝐶𝐶𝑜𝑉2= .3𝐷1+.3𝐷2+.3𝐷3+.1𝐷4. Using the IVC-COV2 index as pretest probability, we proceeded to calculate likelihood ratios and integrate in a Bayesian/Fagan nomogram to attain posttest probabilities, generating a sequential value with IVC-COV2 + NEWS score.

To quantify diagnostic impact, we developed a framework called “Bayesian Diagnostic Gains (BDG)”, where relative (RDG) and Absolute (ADG) diagnostic gains were calculated using differences between CURB 65 pretest results and post-test probabilities. Absolute gain was defined as the difference between pretest and post-test probability (ADG= Post- Pre). Relative gain was the percentage of absolute gain in relation to pretest probability (RDG= ADG/Pre x 100).

‘'Number Needed to’' metrics hold a more intuitive appeal for clinicians than standard diagnostic accuracy measures and these tools are being used for correctly treating, diagnose or predict disease in certain populations. The Number Needed to Treat (NNT) is the number of patients you need to treat to prevent one additional bad outcome. The NNT is the inverse of the absolute risk reduction (ARR). The ARR is the absolute difference in the rates of events between a given activity or treatment relative to a control activity or treatment, i.e. control event rate (CER) minus the experimental event rate (EER), or ARR = CER - EER. The NNTs are always rounded up to the nearest whole number and accompanied as standard by the 95% confidence interval. Example: if a drug reduces the risk of a bad outcome from 50–40%, the ARR = 0.5 - 0.4 = 0.1. Therefore, the NNT = 1/ARR = 10. The ideal NNT would be 1 - ie all patients treated will benefit.

On the basis of this concept, we used the ADG to create a formula for the Number Needed to Diagnose (NND) and called it Bayesian Number Needed to Diagnose (B-NND). For this tool we took the statistical basis of the formula used for the NNT, using ADG as a substitute for ARR. Our formula is as follows: NND = 1/ADG. Descriptive statistics with confidence intervals were used to represent group characteristics, Analysis of Variance was used to stablish comparative difference between models, statistical significance was set at the P<0.05 level.

The IVC-COV2 Score was derived from four dimensions (Table-1) in a captive insured population of 1,055, 745 Dominican social security affiliates. Table-2 highlights the results of the co-morbidities analysis, whereas Table-3 demonstrates the risk analysis breakdown, where 393,459 (were low risk 1.91), 395,172 (26.11%) intermediate risk and 267,114 (71.97%) high risk. We then proceeded to integrate the IVC-COV2 score with a Bayesian model that comparatively included the NEWS and CURB-65 scores likelihood ratios in order to obtain a post-test probability. Whereas table-4 shows the pooled sensitivity and specificity for both the NEWS and CURB-65 scores (11, 15).

Table-5 showcases the combination of low, intermediate, and high risk in the IVC-COV2 index with NEWS and CURB-65 low, intermediate and high results. When using low pretest probability (1.91%) attributed to IVC-COV2 LOW with NEWS score ≥5 our results for positive post-test probability was 15% (95% CI 13-17%). Negative post-test probability was 0% (95% CI 0-1%). For an intermediate pretest probability (26.11%) associated to IVC-COV2 Intermediate with NEWS score ≥5 we obtained a positive post-test probability of 76% (95%CI 72-78%). For negative post-test probability, we obtained 4% (95% CI 3-5%). Combining the high pretest probability (71.97%) of IVC-COV2 HIGH with NEWS score ≥5 gave us a positive post-test probability of 96% (95% CI 93-97%) and a negative post-test probability of 28% (95% CI 26-29%).

When assessing the CURB-65 >2 models (Table-5) we found post-test probability scores of positive likelihood ratios of: low 15% (95% CI 12-19%), Intermediate 76% (95% CI 72-80%) and High 96% (95% CI 94-97%). Whereas post-test probability score for negative likelihood rations were; low 0% (95% CI 0-1%), intermediate 5% (95% CI 4-7%), high 28% (24-32%).

Table-6 demonstrates the Bayesian, Number Needed to Diagnose analysis for both IVC-CoV2 NEWS and CURB-65 models. Demonstrating a slight increased improvement for the CURB-65 model [2.00 (2) vs 2.71(3)]. When assessing comparatively the absolute and relative diagnostic gains, no statistical differences were found on the IVC-CoV2 NEWS model compared to the CURB-65 model in both LR+ (P=0.62) and LR- (P=0.95).

Mathematical modeling can offer important solutions to probability and uncertainty in medicine. “Data recycling” is a concept we have traditionally seen in other statistical methodologies, such a Meta-Analysis. By using the best previously published data and integrating independent high-quality studies in a Bayesian model we propose that can generate previously un answered solutions to clinical decision and hypothesis generation, this is the unique and innovative ingredient to our modeling efforts.

There appear to be very few published studies on the use of NEWS/NEWS2 in the specific context of COVID-19, even in hospital settings. A recent publication from China during the early phase of COVID-19 pandemic offered an early warning score based on an adapted version of the NEWS2 score with age >65 (score 3 points) added to reflect emerging evidence that age is an independent risk factor for survival (11–13). One out of four patients hospitalized with COVID-19 had severe disease, and in-hospital mortality was 20% (13). NEWS score at emergency department admission predicted severe disease and in-hospital mortality and was superior to qSOFA and other clinical risk scores for this purpose. The CURB-65 score has been previously used in predicting pneumonia mortality and ICU disposition (15).

In our models, the biggest diagnostic gains were found when integrating NEWS and CURB-65 to the low and intermediate pre-test probability subgroups. According to both the IVC-COV2 index and the NEWS score, high risk patients are those that should be monitored more closely due to the higher risk of critical outcome. In clinical practice multiple factors (history, physical exam, laboratory, radiology etc...) are used to guide clinical decision making and unusually intermediate (moderate) scores in decision rules leave opportunities for clarity and enhancements. Rapidly producing data-driven results in public health emergency events is of the utmost importance so that authorities and clinicians can guide decisions with the immediately available evidence. The use of insurance population-based data combined with existing decision rules all within a strong mathematical model can be a very useful tool on critical-care decision making and resource allocation.

The integration of IVC-COV2 index with both NEWS and CURB-65 score demonstrated an improved post-test probability in both “rule-in” and “rule-out” subgroups. Thus, suggesting that this Bayesian/patient-centered clinical decision tool can be used to integrate independent clinical items in an effort to adequately predict severity of illness and eventual ICU resource utilization, the model proposes that these 2 scores independently integrated with the IVC-CoV2 can have a greater predictive value for ICU and wards admissions in patients with COVID-19. This decision format is aligned with the realities of clinical practice, where the specifics of one patient are inserted with the integrated value of various tests, prior to making a diagnostic and/or therapeutic decision. We believe that this approach can be especially helpful when guiding decision making in low resource settings and to non-physician clinical staff, particularly if integrated as a computer-based decision tool.

Traditionally, Number Needed to Diagnose (NND) is defined as the number of patients who need to be examined in order to correctly detect one person with the disease of interest in a study population of persons with and without the known disease. For diagnostic tests, low values of NND are desired with one being the closes to a perfect decision. This study reflects on this NND concept and integrates an novel Evidence-Based tool that we called “Bayesian Number Needed to Diagnose” or B-NND as another clinical tool that can aid in understanding the clinical application of Bayesian statistics, we feel it’s a more simple tool to visualize the impact of probability mathematics specially when comparing and measuring effectiveness of various mathematical models.

Based on our NND analysis and comparing all the results from our analysis we can discuss that having an IVC-COV2 index + NEWS /CURB-65 is more predictive of ICU disposition needs than having just IVC-COV2 index alone, the B-NND of 2 assigned to the integration of NEWS score suggests a higher diagnostic value yet this was not seen in the ANOVA analysis model. The most predictive of COVID-19 ICU needs was found in the low and Intermediate IVC-COV2 index integrated with either the NEWS or CURB-65 score. This study experiments with “data recycling” concepts, where by integrating within one probability model data findings from different studies, we can work on rapid turnover clinical decisions not otherwise studied, or even use these findings as hypothesis generating instruments, in the case of COVID-19 similar population-based hybrid math models can be potentially even used for vaccine campaign strategies. Limitations include the sample size, single country nature and the intrinsic qualities of the mathematical model and the absence of strong prospective studies looking at validation NEWS2 and CURB-65 in the care of COVID-19. The use of clinical scoring systems to predict severe disease and mortality in patients with COVID-19 is recommended to be investigated further in larger prospective studies.

This theoretical mathematical model proposed that the combination of a IVC-COV2 Intermediate score plus NEWS or CURB-65 scores yields superior results then other probability settings, suggesting a greater predictive value for severity of illness and admission level care needs, advocating for rapid deployment of decision support tools to be able to combine these various clinical items in a final pathway for the prediction of admissions in patients with COVID-19. No statistical significance was found when comparing the NEWS and CURB-65 score models, thus we recommend for institutions to use the scoring system most applicable to their setting. To the best of our knowledge this is the first hybrid population-based/mathematical model for COVID-19 Critical Care decision making.

Authors Contributions:

AAB: Lead investigator of the ACDC Research program and first author of this specific paper, with substantial contributions to the conception, design of the work, analysis and interpretation of data. As well as drafting the work or revising it critically for important intellectual content, final approval of the version to be published. And agrees be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated.

OL, PRS, LM are co-authors with substantial contributions to the conception, design of the work, analysis and interpretation of data. As well as drafting the work or revising it critically for important intellectual content, final approval of the version to be published. And agrees be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

MPM and PLC: are co-authors with substantial contributions to the conception, design of the work, analysis and interpretation of data. As well as drafting the work or revising it critically for important intellectual content, final approval of the version to be published. And agrees be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

All authors have read and approved the manuscript, and ensure that this is the case.

Acknowledgements: Not Applicable

Competing interests: The authors declare no competing interests

WHO. 2020. Novel coronavirus (2019-nCoV) situation report 132.https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200531-covid-19-sitrep-132.pdf?sfvrsn=d9c2eaef_2
Ruan Q., Yang K., Wang W., Jiang L., Song J. Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China. Intensive Care Med. 2020;46:846–848. Springer. PMID: 31092526
Zhang L., Yan X., Fan Q., Liu H., Liu X., Liu Z. D-dimer levels on admission to predict in-hospital mortality in patients with Covid-19. J Thromb Haemost. 2020;18(6):1324–1329.
Yan L., Zhang H.-T., Goncalves J., Xiao Y., Wang M., Guo Y. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell. 2020;2(May (5)):283–288.
Aziz M., Fatima R., Assaly R. Elevated interleukin-6 and severe COVID-19: a meta analysis [published online ahead of print, 2020 Apr 28] J Med Virol. 2020 NLM (Medline)
Du R.H., Liang L.R., Yang C.Q., Wang W., Cao T.Z., Li M. Predictors of mortality for patients with COVID-19 pneumonia caused by SARSCoV- 2: a prospective cohort study. Eur Respir J. 2020;55(May (5))
Liu Y., Du X., Chen J., Jin Y., Peng L., Wang H.H.X. Neutrophil-to-lymphocyte ratio as an independent risk factor for mortality in hospitalized patients with COVID-19. J Infect. 2020;81(1):e6–e12.
Smith GB, Redfern OC, Pimentel MA, Gerry S, Collins GS, Malycha J, Prytherch D, Schmidt PE, Watkinson PJ. The National Early Warning Score 2 (NEWS2). Clin Med (Lond). 2019 May;19(3):260. doi: 10.7861/clinmedicine.19-3-260. PMID: 31092526
Myrstad M, Ihle-Hansen H, Tveita AA, Andersen EL, Nygård S, Tveit A, Berge T. National Early Warning Score 2 (NEWS2) on admission predicts severe disease and in-hospital mortality from Covid-19 - a prospective cohort study. Scand J Trauma Resusc Emerg Med. 2020 Jul 13;28(1):66. doi: 10.1186/s13049-020-00764-3. PMID: 32660623
Pimentel MAF, Redfern OC, Gerry S, Collins GS, Malycha J, Prytherch D, Schmidt PE, Smith GB, Watkinson PJ. A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study. Resuscitation. 2019 Jan;134:147-156. doi: 10.1016/j.resuscitation.2018.09.026. Epub 2018 Oct 1. PMID: 30287355
Gidari A, De Socio GV, Sabbatini S, Francisci D. Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection. Infect Dis (Lond). 2020 Oct;52(10):698–704. doi:10.1080/23744235.2020.1784457. Epub 2020 Jun 25. PMID: 32584161
Jang JG, Hur J, Hong KS, Lee W, Ahn JH. Prognostic Accuracy of the SIRS, qSOFA, and NEWS for Early Detection of Clinical Deterioration in SARS-CoV-2 Infected Patients. J Korean Med Sci. 2020;35(25):e234. Published 2020 Jun 29. doi:10.3346/jkms.2020.35.e234
Gerry S, Bonnici T, Birks J, Kirtley S, Virdee PS, Watkinson PJ, Collins GS. Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ. 2020 May 20;369:m1501. doi: 10.1136/bmj.m1501. PMID: 32434791; PMCID: PMC7238890.
Physicians RCo. National Early Warning Score (NEWS) 2: Standardizing the assessment of acute-illness severity in the NHS. Updated report of a working party. London: RCP, 20172017 [Available from: https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-NEWS accessed 24 Apr 2020.
Satici C, Demirkol MA, Sargin Altunok E, et al. Performance of pneumonia severity index and CURB-65 in predicting 30-day mortality in patients with COVID-19. Int J Infect Dis. 2020;98:84–89. doi:10.1016/j.ijid.2020.06.038
Shah B.A., Ahmed W., Dhobi G.N., Shah N.N., Khursheed S.Q., Haq I. Validity of pneumonia severity index and CURB-65 severity scoring systems in community acquired pneumonia in an Indian setting. Indian J Chest Dis Allied Sci. 2010;52(1):9–17.
Oktariani, Pitoyo C.W., Singh G., Mansjoer A. CURB 65 score as a predictor of early mortality in hospital-acquired pneumonia. Egypt J Chest Dis Tuberc. 2019;68(2):231.
SISALRIL. Retreived form the web Dic 18th 2020 from: http://sisalril.gov.do/transparencia/phocadownload/Publicaciones/memorias_institucionales/Memoria_2020.pdf
Cochon L, McIntyre K, Nicolás JM, Baez AA. Incremental diagnostic quality gain of CTA over V/Q scan in the assessment of pulmonary embolism by means of a Wells score Bayesian model: results from the ACDC collaboration. Emerg Radiol. 2017 Aug;24(4):355–9.
Baez AA, Cochon L. The acute care diagnostics collaboration: Performance assessment of contrast-enhanced ultrasound compared to abdominal computed tomography and conventional ultrasound in an emergency trauma score bayesian clinical decision scheme. Int J Crit Illn Inj Sci. 2018 Jul-Sep;8(3):154–159.
Farook N, Cochon L, Bode AD, Langer BP, Baez AA. HEART Score and Stress Test Emergency Department Bayesian Decision Scheme: Results from the Acute Care Diagnostic Collaboration. J Emerg Med. 2018 Feb;54(2):147–155.
Cochon L, Smith J, Baez AA. Bayesian comparative assessment of diagnostic accuracy of low-dose CT scan and ultrasonography in the diagnosis of urolithiasis after the application of the STONE score. Emerg Radiol. 2017 Apr;24(2):177–182.
Baez AA, Cochon L. Improved rule-out diagnostic gain with a combined aortic dissection detection risk score and D-dimer Bayesian decision support scheme. J Crit Care. 2017 Feb;37:56–59.
Baez AA, Cochon L. Acute Care Diagnostics Collaboration: Assessment of a Bayesian clinical decision model integrating the Prehospital Sepsis Score and point-of-care lactate. Am J Emerg Med. 2016 Feb;34(2):193–6.
Baez AA, Cochon L, Nicolas JM.Bayesian decision support sequential model for severity of illness predictors and intensive care admissions in pneumonia. BMC Med Inform Decis Mak. 2019 Dec 30;19(1):284. doi: 10.1186/s12911-019-1015-5. PMID: 31888590
Báez AA, López O, Martínez MDP, Libell N, Cochón L, Nicolás JM. Clinical validation demonstrates concordance of qSOFA and POC lactate Bayesian model: Results from the ACDC Phase-2 program. Am J Emerg Med. 2020 Oct 1:S0735-6757(20)
30876-7
. doi: 10.1016/j.ajem.2020.09.080. PMID: 33046312

Table-1 IVC Cov2 COVID-19 Index Dimensions

Dimension #1 (D1)

Comorbidities

Dimension #2 (D2)

Age

Dimnesion#3 (D3)

Gender

Dimension #4 (D4)

Family/Social

Table-2 Population-Based Dimension # (D1) assessment of risk factors associated with death
Risk Factor	Percentage% (N=157)
Diabetes	33.12%
Hypertension	61.15%
Emphysema	15.92%
Peripheral Vascular Disease	1.91%
Cancer	14.64%
Chronic Renal Failure	13.38%
Cardiovascular disease	16.56%

Table 3 IVC-COV2 Index Population Assessment
IVC-COV 2 Risk	Vulnerable Population	Alive	Deceased	Percentage of mortality according to IVC-COV2 risk
Total	1,055,745	1,055,588	157	100%
Low	393,459	393,456	3	1.91%
Intermediate	395,172	395,131	41	26.11%
High	267,001	267,001	113	71.97%

Table-4 NEWS and CURB-65 Scores Sensitivity, Specificity, Positive (+) and Negative (-) Likelihood Ratios

Table 6 - Bayesian Number Needed to Diagnose
Pre-test Probability (Intermediate only)	Post- test LR (+)	ADG (%)	NND (Rounded)
COVID-19 VI (26.11)	NEWS2 58	49.89	2.00 (2)
COVID-19 VI (26.11)	CURB 65 63	36.89	2.71 (3)
* LR (+): Positive Likelihood Ratio * ADG: Absolute Diagnosis Gain * NND: Number Needed to Diagnose

Assessment of a Comparative Bayesian-Enhanced Population-Based Decision Model for COVID-19 Critical Care Prediction in Dominican Republic Social Security Affiliates

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Results

Discussion

Conclusion

Declarations

References

Tables

Status:

Version 1