Prediction of the risk of adverse clinical outcomes with machine learning techniques in patients with chronic no communicable diseases

doi:10.21203/rs.3.rs-4843892/v1

Download PDF

Research Article

Prediction of the risk of adverse clinical outcomes with machine learning techniques in patients with chronic no communicable diseases

https://doi.org/10.21203/rs.3.rs-4843892/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

Decision-making in chronic diseases guided by clinical decision support systems that use models including multiple variables based on artificial intelligence requires scientific validation in different populations to optimize the use of limited human, financial, and clinical resources in healthcare systems worldwide.

Methods

In this cohort study, a prediction model was derived by evaluating two algorithms, XGBoost and Elastic Net logistic regression, for three outcomes - mortality, hospitalization, and emergency department visits - to build a clinical decision support system for patients with non-communicable chronic diseases at the Alma Mater Hospital complex in Medellin, Colombia.

Results

We collected 4845 electronic medical record entries from 5000 patients included in the study. The median age was 71.83 years, with 63.8% women and 29.7% receiving home care. The most prevalent medical conditions were diabetes (52.9%), hypertension (67.2%), dyslipidemia (57.3%), and COPD (19.4%). For the mortality outcome, the Elastic Net logistic regression model had an AUCROC of 0.88 (95% CI, 0.8032 to 0.9032), and the XGBoost model had an AUCROC of 0.912 (95% CI, 0.8802 to 0.9437). For the hospitalization outcome, the Elastic Net logistic regression model had an AUCROC of 0.967 (95% CI, 0.957 to 0.9763), while the XGBoost model had an AUCROC of 0.976 (95% CI, 0.9661 to 0.985). For the emergency department visit outcome, the Elastic Net logistic regression model had an AUCROC of 0.930 (95% CI, 0.9158 to 0.945), while the XGBoost model had an AUCROC of 0.982 (95% CI, 0.9755 to 0.9891). We created a dashboard as to interact with the model, segmenting risk in the cohort.

Conclusions

A clinical decision support system based on artificial intelligence using electronic medical records possibly can help segmenting the risk in populations with chronic diseases for effective decision-making.

Clinical decision support system

predictive models

Mortality

Emergency Consultation

Hospitalization

Artificial Intelligence.

The increase in demand for health services from people with chronic noncommunicable diseases represents a challenge for the health systems of many countries and ours is no exception. In Colombia, the high costs of chronic diseases are reflected in their diagnosis and treatment, which are characterized by being prolonged, complex and affecting the economically active population. Giventhat in many cases their diagnosis and intervention are late, in addition to the costs for the system, a burden is generated for the patient's health and the stability of his family[1]

An appropriate strategy for this chronic disease problem may be to use a model of care based on risk stratification. Stratification is defined as "the identification and/ or grouping of patients according to risk or severity classification", which serves to define in advance interventions that are tailored to their future health care needs (2). To carry out the stratification there are several systems of classification of patients, among which are: Adjusted Clinical Groups (ACG), a system that assigns each person in an exclusive category based on clinical criteria, with the aim of predicting health costs, pharmaceutical expenses and hospitalization risks [2]; Diagnostic Cost Groups (DCG), which use information from all diagnoses and prescriptions to form Clinical Risk Groups (CRG); which, together with demographic characteristics, manage to predict health costs in a year by classifying individuals according to the severity of their health status and according to their chronicity (4); and the Diagnosis-Related Groups (DRGs), which allow relating the different types of patients treated in a hospital, with the cost of their management. The use of DRGs is recommended by the World Health Organization (WHO) and several Latin American governments are exploring their implementation. However, this requires having information systems that have a high quality of the data, to guarantee the accuracy of the classifications and therefore the decisions that are made. In the case of Colombia, there has been evidence of low quality in the Individual Health Service Delivery Registries (RIPS) for the DRM system, as well as the absence of policies that promote and promote comparable health risk [3] (5).

The need to accurately direct the finite resources, both human and economic, of health systems towards a subpopulation at higher risk of adverse outcomes can be realized under the stratification of individual risk taken to population terms, which allows to reach a coordination of the level of care according to the presence of "clusters" or risk profiles in chronic diseases[4] Clinicaldecision support systems (CDS) can help clinicians make informed decisions if they are properly integrated into the treatment process, if they are easy to use and understand, and if they use standards that enable interoperability with other systems. If these CDS systems are designed and implemented with user needs in mind, they have the potential to improve medical decisions, streamline physicians' work, and improve patient outcomes[5]

The hospital in which this research was carried out has developed a "Sermás" care model based on integrated and continuous care, promoting synergies in the health services network and co-management of health risk between the hospital and the insurer[6] Therefore, the aim of this study is to retrospectively derive a real-time risk prediction methodology for adverse clinical outcomes with machine learning techniques and big data analytics in patients with chronic noncommunicable diseases. The final goal is the creation of a prescriptive analysis dashboard as a clinical decision support system, which allows real-time interaction with predictions based on clinical and epidemiological characteristics of patients in the cohort.

Source of data

A retrospective cohort study was conducted on electronic medical record records for the derivation of 2 prediction models of 3 outcomes: mortality, hospitalization and emergency room visit. Patient collection was conducted from April 1, 2017 to December 31, 2020 and outcome assessment from January 1 to December 31, 2020. This study was approved by the ethics committee of the Alma Mater Hospital in Antioquia (INS 2022-08). The data was always managed within the Hospital with security control and passwords in the work ecosystems to protect the identity of the patients. We followed the TRIPOD-AI consensus for reporting prognostic models in health with artificial intelligence.[7]

Participants

The study was conducted in a highly complex hospital complex in the city of Medellín, Antioquia (Hospital Alma Mater de Antioquia) that has an outpatient care headquarters, a home care headquarters and a hospital headquarters. The inclusion criteria were to be at least 18 years old and to have at least one morbidity indicating chronic disease according to ICD-10 code [8] (Table 1 of supplementary appendix.). Patients without clinical data recorded in their electronic medical record were excluded due to lack of medical care due to non-compliance with medical appointments for admission to the program and loss of follow-up of the cohort. The treatment of patients was carried out according to the protocol of the "SerMás" care model, which is a comprehensive health model that involves the coordination of efforts between health providers, insurers and other actors of the health system under international guidelines for chronic disease care.

Table 1

Clinical characteristics of patients included in the study cohort.
Variable	Category	Data loss n(%)	Total n = 4845 (100%)
Age, Average (SD)			71.8 (13.0)
Gender, n (%)	F		3104 (64.1)
Gender, n (%)	M		1741 (35.9)
Emergency room, n (%)			1029 (21.2)
Inpatients, n (%)			918 (18.9)
Surgery, n (%)			370 (7.6)
General practitioner, n (%)			2832 (58.5)
Specialized medicine, n (%)			3464 (71.5)
General wards Days, Media (DS)			7.8 (9.7)
Special Care Days, Media (DS)			4.5 (3.5)
Intensive Care ICU Days, Media (DS)			7.2 (7.7)
Depression and mood disturbances, n (%)			990 (20.4)
COPD, n (%)			940 (19.4)
Thyroid diseases, n (%)			808 (16.7)
Somatoform, n (%)			808 (16.7)
Osteoarthritis, n (%)			798 (16.5)
Ischemic heart disease, n (%)			789 (16.3)
Chronic kidney disease, n (%)			660 (13.6)
Obesity, n (%)			566 (11.7)
Heart failure, n (%)			545 (11.2)
Cerebrovascular, n (%)			472 (9.7)
Dementia, n (%)			443 (9.1)
Osteoporosis, n (%)			424 (8.8)
Atrial fibrillation, n (%)			350 (7.2)
Sleep disorders, n (%)			345 (7.1)
Hypertension, n (%)			3267 (67.4)
Vertigo and hearing impairment, n (%)			286 (5.9)
Other genitourinary ones, n (%)			282 (5.8)
Venous and lymphatic disease, n (%)			282 (5.8)
Peripheral neuropathies, n (%)			271 (5.6)
Upper gastrointestinal diseases, n (%)			268 (5.5)
Migraine and painful facial syndromes, n (%)			251 (5.2)
colitis and lower gastrointestinal, n (%)			248 (5.1)
prostate diseases, n (%)			222 (4.6)
epilepsy, n (%)			213 (4.4)
diabetes, n (%)			2123 (43.8)
dyslipidemia, n (%)			2057 (42.5)
Anemia, n (%)			136 (2.8)
Weight, Medium (SD)		617(12.71%)	68.1 (15.0)
Height, Medium (DS)		617(12.71%)	156.2 (9.3)
Thigh circumference, Mean (DS)		617(12.71%)	48.3 (94.2)
Waist circumference, Mean (DS)		617(12.71%)	95.6 (17.5)
triceps fold measurement, Medium (DS)		617(12.71%)	17.7 (15.4)
Abdomen Fold measure, Medium (DS)		617(12.71%)	26.5 (86.6)
Thigh fold, Medium (DS)		617(12.71%)	21.7 (15.2)
Systolic blood pressure, Mean (SD)		617(12.71%)	129.8 (20.4)
Ddiastolic blood pressure, Mean (SD)		617(12.71%)	73.4 (11.2)
Resting heart rate, Mean (SD)		617(12.71%)	76.0 (11.9)
self-rated exercise level, n (%)	1.0	617(12.71%)	4144 (98.0)
	2.0		24 (0.6)
	3.0		16 (0.4)
	4.0		15 (0.4)
	5.0		29 (0.7)
METS metabolic rate, Mean (SD)		617(12.71%)	4.8 (2.5)
VO2 at maximum oxygen, Mean (SD)		617(12.71%)	17.0 (8.6)
	Fragile		1968 (40.6)
Gröningen fragility index, n (%)	Not qualified		617 (12.7)
	Normal		2260 (46.6)
Monopodial time, Mean (DS)		617(12.71%)	7.8 (10.3)
Ankle-brachial index, n (%)	0.41 to 0. mild to moderate		70 (1.4)
	0.91 a 1.30 Normal		1574 (32.5)
	< 0.4 EAP Grave		33 (0.7)
	Not qualified		3168 (65.4)
Blood glucose, Mean (SD)		944(19,45%)	102.3 (118.6)
Glycated Hemoglobin HbA1c, Media (DS)		944(19,45%)	5.0 (3.3)
LDL, Media (DS)		944(19,45%)	47.2 (48.0)
HDL, Media (DS)		944(19,45%)	41.6 (159.8)
Total Cholesterol, Mean (SD)		944(19,45%)	134.0 (68.3)
Triglycerides, Medium (DS)		944(19,45%)	125.3 (92.2)
Framingham Cardiovascular Risk adjusted to Colombia, n (%)	High risk		1710 (35.3)
	Low risk		2177 (44.9)
	Not rated		958 (19.8)
Glomerular Filtration Rate (GFR), Mean (SD)		944(19,45%)	59.9 (39.7)
Stage of kidney disease, n (%)	Stage 0		1511 (31.2)
	Stage 1		466 (9.6)
	Stage 2		1029 (21.2)
	Stage 3A		796 (16.4)
	Stage 3B		720 (14.9)
	Stage 4		253 (5.2)
	Stage 5		70 (1.4)
Urinary Albumin to Creatinine Ratio, Media (DS)		944(19,45%)	44.3 (271.4)
TSH, Media (DS)		944(19,45%)	2.6 (6.6)
“SerMás” functional classification, n (%)	Functional class 1		39 (0.8)
	Functional class 2A		1482 (30.6)
	Functional class 2B		814 (16.8)
	Functional class 3		145 (3.0)
	Functional class 4		1421 (29.3)
	Not rated		944 (19.5)
Additional results on Hartigan immersion test, normality test and Tukey's test are described in Supplementary appendix
The calibration curves of the Elastic Net and XGBoost models are shown for the outcomes of (A) Hospitalization, (B) Mortality and (C) Emergency Room Consultation with 95% CI for intercept and slope. The graph also shows the values from C to statistic for XGBoost models along with their respective 95% confidence intervals.

Outcomes

Hospital and out-of-hospital mortality: data were obtained from the GHIPS system (2024 "ALMA MATER HOSPITAL DE ANTIOQUIA" Version: 31.2.20221216 to 37) and out-of-hospital mortality was confirmed by the health insurer and the RUAF (©information system that consolidates the affiliations reported by the entities and administrators of the Social Protection System in Colombia).

Hospitalization: data were obtained on the number of times the patient consulted the assigned referral hospital or other hospitals the data was reported to the hospital when the patient was hospitalized elsewhere.

Use of emergency only in the reference hospital.

Predictor variables

We included 164 variables which were obtained from the GHIPS system, a web application that works as an electronic medical record system of the hospital in the outpatient and hospital settings. Clinical, laboratory and billing variables were extracted (Table 2 of supplementary appendix). In addition, the outcomes of mortality, hospitalization and emergency use during the evaluation year were obtained.

Table 2

Statistical analysis of XGBoost and Elastic net Logistic Regression models for mortality, hospitalization, and emergency room outcomes.
	Model	Sensitivity (recall)	Specificity (selectivity)	Negative predictive value	Positive predictive value (precision)	AUCROC	Intercept	Slope	DeLong's test
Mortality	Elastic net logistic regression	38.85% (IC95% 30.71–47.48%)	93.50% (IC95% 92.02–94.77%)	93.50% (IC95%. 92.64–94.26%)	93.50% (IC95%. 92.64–94.26%)	0.88 (IC95%. 0.8032 to 0.9032)	-0.03 (IC95%. -0.25 to 0.19)	0.85 (IC95%. 0.72 to 0.96)	p = 0.04414
Mortality	XGBoost	47.48% (IC95% 38.95–56.12%)	99.19% (IC95% 98.52–99.61%)	94.39% (IC95% 93.49–95.17%)	86.84% (IC95%. 77.65–92.61%)	0.912 (IC95%. 0.8802 to 0.9437)	0.17 (IC95%. -0.06 to 0.41)	1.14 (IC95%. 0.99 to 1.29)	p = 0.04414
Hospitalization	Elastic net logistic regression	80.53% (IC95% 75.61–84.83%)	95.90% (95% 94.54–97.01%)	94.58% (IC95% 93.28–95.64%)	84.72% (IC95%. 80.51–88.16%)	0.967 (IC95%. 0.957 to 0.9763)	0.16 (IC95%. -0.41 to 0.08)	0.84 (IC95%. 0.73 to 0.94)	p = 0.02082
Hospitalization	XGBoost	84.49% (IC95% 79.91–88.37%)	96.55% (IC95% 28–97.56%)	95.66% (IC95% 94.43–96.63%)	87.37% (IC95%. 83.40–90.50%)	0.976 (IC95%. 0.9661 to 0.985)	0.01 (IC95%. -0.24 to 0.25)	1.06 (IC95%. 0.93 to 1.19)	p = 0.02082
Emergency consultation	Elastic net logistic regression	5.21% (IC95% 3.07–8.22%)	99.81% (IC95%. 99.31–99.98%)	77.25% (IC95%. 76.79–77.69%)	89.47% (IC95%. 66.38–97.34%)	0.930 (IC95%. 0.9158 to 0.945)	-0.01 (IC95%. -0.14 to 0.11)	12.23 (IC95%. 10.64 to 13.83)	p = 6.981
Emergency consultation	XGBoost	92.33% (IC95% 88.89–94.98%)	95.62% (IC95% 94.20–96.78%)	97.57% (IC95% 96.50–98.32%)	86.74% (IC95%. 83.12–89.69%)	0.982 (IC95%. 0.9755 to 0.9891)	0.01 (IC95%. -0.23 to 0.25)	1.2 (IC95%. 1.07 to 1.34)	p = 6.981
*Sensitivity (Sens) is the ability of the model to correctly rule out patients with the outcome of interest. ^Specificity (Spec) is the ability of the model to correctly detect patients without the outcome of interest. ̈Negative predictive value (NPV) is the probability that a patient will not have the outcome of interest if the model classifies it as such. +Positive predictive value (PPV) is the probability that a patient will have the outcome of interest if the model classifies it as such.
AUCROC (Area under the ROC curve) is a numerical measure that evaluates the model's ability to differentiate between patients with and without the outcome of interest. 95% CI calculated by DeLong DeLong's Test: Test to define differences between areas under the curve

Sample size calculation.

A formal sample size calculation was not made because there was a fixed cohort of users.

Imputation of missing data

A data imputation process was performed to variables with less than 20% data losses using the K Nearby Neighbors (KNN) algorithm, an imputation technique that uses information from existing data to estimate missing values. The algorithm selects a number k of observations closest to the observation with the missing value (neighbors) in the complete dataset. Then use the mean of those k neighbors to estimate the missing value. [9]

Statistical analysis

For distributed variables, the mean and standard deviation are usually shown, while variables that are not normally distributed are reported with the median and interquartile range. The Hartigan immersion test was applied to describe possible multimodal distributions and the Tukey test to describe variables with distant outliers[10] DeLong's Test, was used to define differences between model’s areas under the curve. All statistical measures were calculated with accompanying 95% confidence intervals (CIs) and w used a p-value threshold of 0.05

Models

Two supervised learning models were used: the Elastic Net logistic regression model and the XGBoost algorithm. For the 3 outcomes, the database was divided into 2 parts: 85% for model training and 15% for a test dataset (Internal Validation). Before entering the data into the machine learning models, the numerical variables were centered at a mean of zero and then scaled to ensure that they all have a variance of 1. For the nominal variables, Dummies (indicator) variables (12) [11] with the ICD reference category 10 described in Table 1 of the supplementary appendix.

The Elastic Net logistic regression model is an extension of the traditional logistic regression model that uses regularization techniques to reduce the risk of overfitting, and uses a combination of the vector norm L1 (the sum of the absolute value of the elements of the vector) and L2 (Euclidean norm is the square root of the sum of the squares of the elements of the vector) for regularization, what is known as Lasso (L1), Ridge (L2) and Elastic Net (L1 and L2) regularization, respectively, to automatically select the most important characteristics in the data and avoid overfitting[12]

The XGBoost algorithm is an implementation of gradient boosting with decision trees. Gradient boosting is a machine learning technique that consists of training a set of decision trees sequentially, where each tree is trained to correct the errors of the previous tree. XGBoost uses a gradient optimization technique called "stochastic gradient regularization" to adjust the parameters of individual decision trees[13]

For the two models all the "hyperparameters" were initialized randomly, a set of fitting data of these "hyperparameters" was not used and instead the 10-fold cross-validation technique was performed for each of the two models with the three outcomes, to find the best parameters between the training dataset and the validation dataset[14] The metrics under the curve (AUCROC), sensitivity, specificity, negative predictive value,positive predictive value and calibration curves were determined with the calculation of the slope and intercept for each outcome. For each metric, the 95% confidence interval was calculated and a maximum alpha error of 0.05 was accepted.

Models were selected for each outcome with better discrimination in AUCROC and no statistically significant differences in slope and intercept in calibration curves. After training the XGBoost and Elastic Net regression models, their results were compared using the DeLong test for differences in the AUCROC of each of the outcomes[15]

The R programming language (version 4.2.2 Copyright (C) 2022 The R Foundation for Statistical Computing) and Python (Python Software Foundation (2021) were used. Python Language Reference, version 3.10.) to process the data and derive the model.

Risk groups.

Risk groups were created through a demo dashboard in Tableau software (Tableau. (2021). Tableau 2021.2 [Software]), in which the AI model was connected to make predictions and visualize the results in histogram form in the cohort. This allows patients to be displayed and filtered in the context of their prediction, to support decisions about the use of clinical resources and prioritization according to their risk. The dashboard creates histograms to predict the 3 outcomes with the probability extracted from the model on the X-axis and the number of patients on the Y-axis. This dashboard is the input for the end user to interact with the predictions (Fig. 5)

Data were collected from January 2020 to December 2020 for a total of 5000 eligible patients and 4845 finally analyzed (Fig. 1). The cohort had a mean age of 71.8 years (standard deviation of 13.0) with 64.1% (n = 3104) women. 21.2% (n = 1029) of patients presented to the emergency department and 18.9% (n = 918) were hospitalized. 58.5% (n = 2832) consulted a general practitioner and 71.5% (n = 3464) consulted a specialist physician. The most common comorbidities were hypertension (67.4%), diabetes (43.8%) and dyslipidemia (42.5%). 19.4% of patients had chronic obstructive pulmonary disease (COPD), 16.7% had thyroid disease and 11.2% heart failure. The total mean value of billing in Colombian pesos was COP 5,468,904 per patient in the year (standard deviation of 7,376,458) (Table 1.) The distribution of chronic disease categories is presented in sTable 3 of the Supplementary Appendix.

Models

For the mortality outcome, the Elastic Net logistic regression model had an AUCROC d e 0.88 (95% CI 0.8032 to 0.9032), while for the XGBoost model the AUCROC was 0.912 (95% CI 0.8802 to 0.9437). For the hospitalization outcome, the Elastic Net logistic regression model had an AUCROC of 0.967 (95% CI 0.957 to 0.9763) and in the XGBoost the AUCROC was 0.976 (95% CI 0.9661 to 0.985). For the emergency room consultation outcome, the AUCROC values for the logistic regression model and the XGBoost were 0.930 (95% CI 0.9158 to 0.945) and 0.982 (95% CI 0.9755 to 0.9891), respectively. (Fig. 2.)

In Table 2 we show the summary of all metrics performance of the two prediction models for the three selected outcomes.

The predictive performance metrics for Mortality prediction was for Elastic net logistic regression sensitivity of 38.85%, specificity of 93.50%, positive predictive value of 93.50%, and negative predictive value of 93.50%. For XGBoost exhibited a sensitivity of 47.48%, specificity of 99.19%, positive predictive value of 86.84%, and negative predictive value of 94.39%. For Hospitalization prediction: Elastic net logistic regression achieved a sensitivity of 80.53%, specificity of 95.90%, positive predictive value of 84.72%, and negative predictive value of 94.58%.

XGBoost demonstrated a sensitivity of 84.49%, specificity of 96.55%, positive predictive value of 87.37%, and negative predictive value of 95.66%, and for Emergency consultation prediction Elastic net logistic regression had a sensitivity of 5.21%, specificity of 99.81%, positive predictive value of 89.47%, and negative predictive value of 77.25%.

XGBoost showed a sensitivity of 92.33%, specificity of 95.62%, positive predictive value of 86.74%, and negative predictive value of 97.57%. Figure 3. shows a spider plot for the two models.

The models were calibrated against the outcomes Fig. 4. shows the calibration plots for the two models.

The weighting of the main variables by each model for the three outcomes is shown in Supplementary Appendix

There are several limitations in the study that must be considered. First, the retrospective observational design, since the study represents the first step in the derivation of a risk model for the creation of a clinical decision support system within the framework of the DECIDE AI consensus[16] Therefore, more prospective research with intervention studies is needed to validate the model in different populations and healthcare settings before it can be used in clinical practice. Second, the study was conducted in a single highly complex reference hospital with an elderly population with multiple chronic diseases, without adequate representation of young patients. Although the hospital had different settings of home, outpatient and hospital care, there were differences in the balance of demographic characteristics such as the marked difference between women and men. No race information was collected, which could limit generalizability of results to other hospitals or healthcare settings. Third, the sample was very small relative to the amount recommended in studies employing machine learning approaches[17] and included a large number of predictive variables, making the assembly process with other types of medical history software technically difficult.

In this study, two prediction models were derived from Elastic Net Logistic Regression and XGBoost for three outcomes: mortality, hospitalization and emergency room visit. Statistically significant differences were found when comparing AUCROC in favor of XGBoost over Elastic-Net regression in the mortality models (DeLong test p = 0.04414) and hospitalization (DeLong test p = 0.02082). On the other hand, for the emergency room consultation model there were no statistically significant differences (DeLong test p = 6.981). In the calibration of this last outcome, although both models overestimate, the slope is much higher for the Elastic Net regression model (12.23; CI95% 10.64 to 13.83) than for the XGBoost model (1.2; CI95% 1.07 to 1.34). With respect to the calibration of hospitalization outcome, Elastic logistic regression does not significantly underestimate risk. In the other calibration comparisons, there were no significant differences. Previously, in the same cohort, a functional scale for predicting mortality (C-statistical of 0.721 95% CI: 0.660–0.780), emergency room (C-statistical of 0.570 95% CI: 0.500–0.640) and hospitalization (C-statistical of 0.609 CI95%: 0.570–0.650) was developed and validated, so this study presents a predictive approach of greater discrimination of adverse outcomes of the SerMás cohort (8).[16]

While the models performed well overall in our study, XGBoost performed better. This same finding has been observed in another research. Forrest IS et al derived and validated a model of random decision trees to predict coronary heart disease with an AUROC of 0.95 (95% CI 0.94 to 0.95), a sensitivity of 0.94 (95% CI 0.94 to 0.95) and a specificity of 0.82 (95% CI 0.81 to 0.83) (19). Li et al evaluated the ability of XGBoost and logistic regression and other algorithms to predict mortality in heart failure patients admitted to the ICU. The results showed that XGBoost and logistic regression Las so L1 with AUCROC of 0.8416 (95% CI 0.7864 to 0.8967) had a superior performance compared to the risk score model "The American Heart Association Get With The Guidelines a Heart Failure GWTG - HF", which exhibited an AUCROC of 0.7856 (95% CI 0.7183 to 0.8470). However, the XGBoost showed a wide net profit threshold range (> 0.1) above the other two models [18] Another study to predict ICU admission from the emergency room found that XGBoost performed well compared to deep neural networks (DNNs). The XGBoost model obtained an AUCROC of 0.861 (95% CI 0.848 to 0.874) with a higher discriminative yield than the DNN model with an AUCROC of 0.833 (95% CI 0.819 to 0.848) [19] Khera et al, compared the performance of some artificial intelligence algorithms, including XGBoost against logistic regression, in predicting mortality in patients with acute myocardial infarction. It found that the XGBoost model reclassified 32,393 of 121,839 patients (27%) at moderate to high risk of death, considered to be low risk in the logistic regression model[20]

An interesting finding of this study was the predictive capacity of billing administrative variables in different outcomes. MacKay et al developed and validated a risk model with machine learning techniques, based on administrative and clinical data, to predict mortality and different adverse clinical outcomes in medical and surgical patient populations. The XGBoost algorithm achieved an AUROC of 0.88 compared to logistic regression of 0.84 for predicting 30-day mortality. In addition, they present an interface to interact with risk that shows the weight of the coefficients in order to guide physicians to manage risk as a clinical decision support system ([21] a similar way to the interface that was constructed in our study (Fig. 1 of supplementary appendix 2).

In conclusion, the XGBoost model presented a better performance than the logistic regression and Elastic Net in the prediction of mortality and hospitalization, while for the consultation to the emergency room both models had a similar performance. Overall, the results indicate that the XGBoost model has the potential to be a tool for building clinical decision support systems that function as useful prognostic models for decision-making in patients with chronic noncommunicable diseases. These types of tools should be evaluated and validated in future experimental studies for safe implementation in clinical flowcharts.

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

The authors declare that there was no funding.

The study was approved by the Ethics Committee on Human Research at the University Research Headquarters (CBE-SIU), Universidad de Antioquia, under number 20-114-922. The Committee deemed the retrospective observational study ethically valid and posing no risk to participants. The Committee had ongoing access to study data while maintaining confidentiality and approved the measures for participant protection and the informed consent process. The study was conducted in accordance with Resolution 008430 of October 4, 1993, of the Colombian Ministry of Health, which established the scientific, technical, and administrative standards for health research; the principles of the World Medical Assembly as stated in the Declaration of Helsinki of 1964, last updated in 2013; the Code of Federal Regulations, Title 45, Part 46, for the protection of human subjects by the U.S. Department of Health and Human Services and the National Institutes of Health (June 18, 1991); and Resolution 2378 of 2008 of the Ministry of Social Protection of Colombia, which adopts Good Clinical Practices for institutions conducting research with medications in humans.

Funding:

This research was conducted without external funding. All resources were provided by the authors.

Author Contribution

A.H.A. (Alejandro Hernández-Arango) was responsible for the design and conceptualization of the study. M.I.A. (María Isabel Arias) and V.P. (Viviana Pérez) conducted the data collection. Data analysis and interpretation were performed by A.H.A. and F.J. (Fabian Jaimes). A.H.A. and F.J. drafted and revised the manuscript. F.J. approved the final version of the manuscript. No external funding was received for this study.All authors reviewed and approved the final manuscript.

Data Availability

The data that support the findings of this study are available from the authors, but restrictions apply to the availability of these data, which were used under ethical approval from Hospital Alma Máter de Antioquia for the current study, and so are not publicly available. Data are, however, available from the authors upon reasonable request and with permission from the Hospital Alma Máter de Antioquia.

K. Gallardo-Solarte K., F. P. Benavides-Acosta F.P., and R. Rosales-Jiménez R., “Costos de la enfermedad crónica no transmisible: la realidad colombiana,” Rev. Cienc. Salud, vol. 14, no. 1, pp. 103–114, Feb. 2016, doi: 10.12804/revsalud14.01.2016.09.
J. F. Orueta Mendia, A. García-Álvarez, E. Alonso-Morán, and R. Nuño-Solinis, “Desarrollo de un modelo de predicción de riesgo de hospitalizaciones no programadas en el País Vasco,” Rev. Esp. Salud Publica, vol. 88, no. 2, pp. 251–260, Apr. 2014, doi: 10.4321/s1135-57272014000200007.
I. Gorbanev, A. E. Cortés Martínez, S. Agudelo Londoño, and F. J. Yepes Lujan, “Grupos relacionados por el diagnóstico: experiencia en tres hospitales de alta complejidad en Colombia,” Univ. Médica, vol. 57, no. 2, pp. 171–181, Jul. 2016, doi: 10.11144/javeriana.umed57-2.grde.
E. Nolte, World Health Organization: Regional Office for Europe, and C. Knai, Assessing chronic disease management in European health systems. Europe, UK: WHO Regional Office for Europe, 2015.
B. C. Stagg et al., “Special Commentary: Using Clinical Decision Support Systems to Bring Predictive Models to the Glaucoma Clinic,” Ophthalmol Glaucoma, vol. 4, no. 1, pp. 5–9, Jan-Feb 2021, doi: 10.1016/j.ogla.2020.08.006.
V. García-Arango, J. Osorio-Ciro, D. Aguirre-Acevedo, C. Vanegas-Vargas, C. Clavijo-Usuga, and J. Gallo-Villegas, “Validación predictiva de un método de clasificación funcional en adultos mayores,” Rev. Panam. Salud Publica, vol. 45, p. e15, Apr. 2021, doi: 10.26633/rpsp.2021.15.
G. S. Collins et al., “TRIPOD + AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods,” BMJ, vol. 385, p. e078378, Apr. 2024, doi: 10.1136/bmj-2023-078378.
A. Calderón-Larrañaga et al., “Assessing and measuring chronic multimorbidity in the older population: A proposal for its operationalization,” J. Gerontol. A Biol. Sci. Med. Sci., p. glw233, Dec. 2016, doi: 10.1093/gerona/glw233.
S. Faisal and G. Tutz, “Multiple imputation using nearest neighbor methods,” Inf. Sci. (Ny), vol. 570, pp. 500–516, Sep. 2021, doi: 10.1016/j.ins.2021.04.009.
T. J. Pollard, A. E. W. Johnson, J. D. Raffa, and R. G. Mark, “tableone: An open source Python package for producing summary statistics for research papers,” JAMIA Open, vol. 1, no. 1, pp. 26–31, Jul. 2018, doi: 10.1093/jamiaopen/ooy012.
J. Gareth, W. Daniela, H. Trevor, and T. Robert, An introduction to statistical learning: with applications in R. Spinger, 2013. [Online]. Available: https://dspace.agu.edu.vn/handle/agu_library/13322
J. Friedman, T. Hastie, and R. Tibshirani, “Regularization Paths for Generalized Linear Models via Coordinate Descent,” J. Stat. Softw., vol. 33, no. 1, pp. 1–22, 2010, doi: 10.1109/TPAMI.2005.127.
T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, in KDD ’16. New York, NY, USA: Association for Computing Machinery, Aug. 2016, pp. 785–794. doi: 10.1145/2939672.2939785.
Y. A. Ali, E. M. Awwad, M. Al-Razgan, and A. Maarouf, “Hyperparameter search for machine learning algorithms for optimizing the computational complexity,” Processes (Basel), vol. 11, no. 2, p. 349, Jan. 2023, doi: 10.3390/pr11020349.
E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson, “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach,” Biometrics, vol. 44, no. 3, pp. 837–845, Sep. 1988, doi: 10.2307/2531595.
B. Vasey et al., “Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI,” Nat. Med., vol. 28, no. 5, pp. 924–933, May 2022, doi: 10.1038/s41591-022-01772-9.
M. A. Gianfrancesco, S. Tamang, J. Yazdany, and G. Schmajuk, “Potential biases in machine learning algorithms using electronic health record data,” JAMA Intern. Med., vol. 178, no. 11, pp. 1544–1547, Nov. 2018, doi: 10.1001/jamainternmed.2018.3763.
F. Li, H. Xin, J. Zhang, M. Fu, J. Zhou, and Z. Lian, “Prediction model of in-hospital mortality in intensive care unit patients with heart failure: machine learning-based, retrospective analysis of the MIMIC-III database,” BMJ Open, vol. 11, no. 7, p. e044779, Jul. 2021, doi: 10.1136/bmjopen-2020-044779.
S. W. Choi, T. Ko, K. J. Hong, and K. H. Kim, “Machine Learning-Based Prediction of Korean Triage and Acuity Scale Level in Emergency Department Patients,” Healthc. Inform. Res., vol. 25, no. 4, pp. 305–312, Oct. 2019, doi: 10.4258/hir.2019.25.4.305.
R. Khera et al., “Use of Machine Learning Models to Predict Death After Acute Myocardial Infarction,” JAMA Cardiol, vol. 6, no. 6, pp. 633–641, Jun. 2021, doi: 10.1001/jamacardio.2021.0122.
E. J. MacKay et al., “Application of machine learning approaches to administrative claims data to predict clinical outcomes in medical and surgical patient populations,” PLoS One, vol. 16, no. 6, p. e0252585, Jun. 2021, doi: 10.1371/journal.pone.0252585.

No competing interests reported.

09082024AHASupplementaryAppendixPredictionoftheriskofadverseclinicaloutcomeswithmachinelearningtechniquesinpatientswithchronicnocommunicablediseases.docx

Download PDF

Reviews received at journal
24 Sep, 2024
Reviewers agreed at journal
26 Aug, 2024
Reviewers invited by journal
21 Aug, 2024
Editor assigned by journal
15 Aug, 2024
Submission checks completed at journal
12 Aug, 2024
First submitted to journal
01 Aug, 2024

You are reading this latest preprint version

Prediction of the risk of adverse clinical outcomes with machine learning techniques in patients with chronic no communicable diseases

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Introduction

Methods

Source of data

Participants

Imputation of missing data

Statistical analysis

Models

Results

Models

Discussion

Conclusions

Declarations

Funding:

Author Contribution

Data Availability

References

Additional Declarations

Supplementary Files

Status:

Version 1