As of March 2022, the coronavirus has caused five global peaks in the number of patients and deaths from COVID-19 through different strains. It is critical to monitor and allocate patients to increase the efficacy of the health system. The high capabilities of artificial intelligence and machine learning algorithms in information processing can help us improve patient management. In this study, we worked intimately with healthcare professionals to provide a tool that can solve real-world needs. For this, we developed a model to predict the mortality risk of COVID-19 inpatients at admission using clinical and laboratory data. In addition, a set of eight clinical and ten cheap, available laboratories were selected in our model. Furthermore, an imputation tool is used to impute the not-available labs, and a ternary outcome classification (low, high, and very high risk) was proposed as healthcare experts' suggestion which is helpful during peaks of disease.
The results of this study are promising and applicable for managing COVID-19 inpatients with the current and upcoming COVID-19 variants. The internal validation, validation with 20% missing laboratories, and external validation showed promising results (AUC > 80). Validation with 20% missing data indicates the approved potential of our model in cases when extracting some of the patient's data is not feasible and needs to be imputed. Moreover, the model's generalization was investigated using data from the fourth hospital in a different province. The AUC of 82.8% was achieved in external validation, which further confirmed the model performance for global application.
Finally, we selected a deep neural network model trained on features determined by the lasso regression method as our proposed model based on its performance on the external dataset (AUC = 83.4%). Despite the susceptibility of neural networks to overfitting, our neural network models performed well on the external validation dataset due to feature selection methods and large sample sizes. Several studies have developed machine learning models to predict COVID-19 patients' mortality risk. However, as demonstrated in Table 3, models with high AUC scores are most likely trained on a small dataset or the data gathered from a single medical center which can indicate that these models may not be generalized and their performance can drop in a dataset from a different center (7–12). Furthermore, our proposed model performed relatively better when compared to models trained on a larger multicentral dataset. This higher performance may be due to the large number of input features, which can simultaneously analyze different aspects of a patient's health (13–16).
Table 3
Current studies with external validation in the literature predicting prognosis of COVID-19 using clinical and laboratory retrieved from search in PubMed and Scopus databases and review articles (29, 30).
Author, Publish Date, | Training dataset sources, Country | Number of patients for model development | Variable for prediction | Outcome | Proposed Model | Internal (In) and External (Ex) Validation AUROC (95% CI) |
Our Model, Iran | 3 centers | 5320 | 27 clinical (history and examination) and 10 laboratory variables | In-hospital mortality | Deep neural network, LASSO | In: 83.8% Ex: 82.8% |
Singh et al, Dec 2021, (13) | 3 centers | 8,427 | 10 markers selected from 57 laboratory, clinical, and demographic variables | Disease severity* | minimum redundance maximum relevance, hybrid feature selection | In: 78% Ex: 74% |
Noy et al, Feb 2022(7) | 1 centers, Israel | 417 | Static and dynamic features including demographics, background disease, vital signs and lab measurements | deterioration within the next 7–30 h | CatBoost (ensemble decision tree) | In: 84% Ex: 74% |
Chen et al, Apr 2021, (14) | 7 centers, China | 6415 | 4 Clincal and 4 Laboratory Variables | In-hospital Mortality | Random forrest, LASSO | In: 90% Ex: 89%, 90%, 81% |
Clift et al, Oct 2020, (15) | 910 practices, UK | 6,083,102 | age, ethnicity, deprivation, body mass index, and a range of comorbidities | In-hospital Mortality | regression coefficients, LASSO | AUROC is not reported, R squared = 73.1% |
Vaid et al, Sep 2020, (8) | 1 center, USA | 1514 | Age and 8 laboratory markers | In-hospital Mortality (following 1,3,5,7 days) | XGBoost, LASSO | In: 89% at 3 days, 85% at 5 and 7 days Ex: 80% at 3 days, 79% at 5 days, 80% at 7 days |
Ko et al, Nov 2020, (9) | 1 center, China | 361 | Age, gender, and 28 blood biomarkers | In-hospital mortality | deep neural network and random forest models | In: accuracy = 93% Ex: accuracy = 92% |
Gao et al, Oct 2020 (10) | 2 centers, China | 1506 | 6 clinical and 2 laboratory biomarkers | mortality risk stratification | Logistic Regression, Support Vector Machine, Gradient Boosted Decision Tree, and Neural Network | In: 92.4%, Ex: 95.5%, 87.9% |
Bertsimas et al, Dec 2020, (16) | 33 centers | 3,927 | Age and 9 laboratory biomarkers | In-hospital mortality | XGBoos | In: 90% Ex: 87%, 92%, 80% |
Guan et al, Jan 2021 (11) | 2 centers, China | 1270 | 2 clinical and 4 laboratory features | In-hospital mortality | Simple-tree XGBoost | In:99.1% Ex: 99.7% |
Hu et al, Sep 2020 (12) | 1 center, China | 183 | Age and 4 laboratory variables | In-hospital mortality | Logistic Regression | int:89.5% Ex: 88.1% |
Footnote: AUROC: ;LASSO: ; * Severity level 0 (no respiratory problem) to level 4 (in-hospital ≤ 30-day mortality) |
The application of machine learning models in the clinic depends on the features based on which the machine predicts. Ease of access and the possibility of easy measurement of these features to predict with high accuracy at the right time is of great importance at the bedside of patients. Predictors were selected by the use of 2 different feature selection methods and their further comparison. Selected features in the present study include 18 factors: age, history of myalgia, loss of consciousness, vertigo and vomiting, dramatic lesions, alcohol consumption, history of GI problems, rheumatoid arthritis, neurologic disorders, leukocytosis, thrombocytopenia, low hemoglobin level, high CRP, low HCO3, high CPK level, low O₂ saturation, pulse rate, and respiratory rate at the time of admission. Previous studies included many of our selected features for prognosis prediction (7–9, 13, 14).
Our predictors are easily accessible and routinely checked with a simple history, physical examination, and blood test in all hospitals. Our results can better explain COVID-19 poor prognostics since all these factors are associated with high mortality risk. Multicollinearity may bring about redundancy in the model performance. Feature selection methods dismissed parameters with a high level of correlations and collinearity. In previous studies, laboratory markers, patient demographics, medical history, and vital signs have been used as effective features in predicting the mortality of patients with COVID-19 (7, 13, 17–22). Some studies used factors including different inflammatory cytokines (23–26), which are not part of patients' routine admission measurements and cannot be obtained in settings with congested resources in contrast to our predictors. Our model may prompt individualized treatments due to distinguishing patients' prognoses based on their different clinical characteristics. This can lead to optimal decision-making of physicians.
There are some limitations to this work that should be noted. First, even though we had a relatively large patient population, our study was retrospective. Prospective validation of our study is required to ascertain the results. The hospitals in our study are all in a developing country (Iran). The scarcity of medical resources in hospitals may bring about inadequate service allocated to patients. This condition can thereby increase the mortality rate in such countries in contrast to countries with effective medical systems. Additionally, the current model does not encompass imaging, microbiological, and histological data, which could contribute to a more precise prognosis prediction despite the inconvenience. Socioeconomic and racial differences, which were investigated in some studies (27, 28), might as well play a role in prognosis.
In conclusion, this study shows that using machine learning methods can predict the mortality risk of COVID-19 patients on admission. This confirms the potential of ML methods for use in clinical practice as a decision-support system. However, effective machine learning models should satisfy the real-world needs of healthcare experts to increase the chance of implementation in practice. Further studies are suggested to investigate the current barriers to implementing ML in practice.