A machine-learning approach for dynamic prediction of sepsis-induced coagulopathy in critically ill patients with sepsis: an integrated analysis of the MIMIC-IV and eICU-CRD databases

doi:10.21203/rs.3.rs-125438/v1

Download PDF

Research

A machine-learning approach for dynamic prediction of sepsis-induced coagulopathy in critically ill patients with sepsis: an integrated analysis of the MIMIC-IV and eICU-CRD databases

https://doi.org/10.21203/rs.3.rs-125438/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Sepsis-induced coagulopathy (SIC) denotes an increased mortality rate and poorer prognosis in septic patients.

Methods

Machine-learning models were developed based on septic patients who were older than 18 years and stayed in intensive care units (ICUs) for more than 24 hours in Medical Information Mart for Intensive Care (MIMIC)-IV. Eighty-eight potential predictors were extracted, and 15 various machine-learning models assessed the daily risk of SIC. The most potent model was selected based on its accuracy and Area Under the receiver operating characteristic Curve (AUC), followed by fine-grained hyperparameter adjustment using the Bayesian Optimization Algorithm. The effects of features on prediction scores were measured using the SHapley Additive exPlanations (SHAP) values. A compact model was developed, based on 15 features selected according to their importance and clinical availability. Two models were compared with Logistic Regression and SIC scores in terms of SIC prediction. Additionally, an external validation was performed in the eICU Collaborative Research Database (eICU-CRD).

Results

Of 11362 patients in MIMIC-IV included in the final cohort, a total of 6744 (59%) patients had SIC during sepsis, and 16183 samples were extracted. The model named Categorical Boosting (CatBoost) had the greatest AUC in our study (0.869 [0.850, 0.886]). Coagulation profile and renal function indicators are the most important features to predict SIC. A compact model was developed with the AUC of 0.854 [0.832, 0.872], while the AUCs of Logistic Regression and SIC scores were 0.746 [0.735, 0.755] and 0.709 [0.687, 0.733], respectively. A cohort of 35252 septic patients in eICU-CRD was analyzed. The AUCs of the full and the compact models in external validation were 0.842 [0.837, 0.846] and 0.803 [0.798, 0.809], respectively, which were still larger than those of Logistic Regression (0.660 [0.653, 0.667]) and SIC scores (0.752 [0.747, 0.757]). Prediction results can be illustrated by using SHAP values in the instance level, which makes our models clinically interpretable.

Conclusions

We developed two models which were able to dynamically predict the risk of SIC in septic patients better than conventional Logistic Regression and SIC scores. Prediction results of our two models can be interpreted by using SHAP values.

Critical Care & Emergency Medicine

Agricultural Engineering

Sepsis-induced coagulopathy

early prediction

critical care

machine learning

Categorical Boosting algorithm

Logistic Regression

external validation

model interpretation

Sepsis, defined as life-threatening organ dysfunction caused by a dysregulated host response to infection, remains the first leading cause of mortality in critically ill patients [1, 2]. Coagulopathy is one of the main complications of sepsis, leading to a higher risk of bleeding, the deterioration of organ failure, and an increased mortality rate [3–5]. However, the effectiveness of anticoagulant therapies was not observed in septic patients [6, 7]. Recent studies and subgroup analysis in large-scale randomized controlled trials reported that anticoagulant therapies brought a significant reduction of mortality risk and improved outcome in the septic patients with coagulopathy [8–11]. In contrast, anticoagulant therapies on non-coagulopathy patients were supposed to be avoided because of the increased risk of bleeding with no survival benefit [10, 12]. These study results have heightened the need for early identification of coagulopathy in septic patients in a timely way.

Sepsis-induced coagulopathy (SIC) criteria were developed by members of the Scientific and Standardization Committee (SSC) on Disseminated Intravascular Coagulation (DIC) of the International Society of Thrombosis and Haemostasis (ISTH) in 2017 [13] (Additional File 1: Table S1). The criteria are a scoring system designed to identify patients with "sepsis and coagulation disorders". SIC is defined as a score ≥ 4. It was found that the mortality rate increased as SIC score elevated and exceeded 30% at a score of 4 [13]. Compared with DIC, a significant cause of organ failure in sepsis, SIC is more relevant for the updated Sepsis‐3 criteria [1, 14]. Observational evidence has shown that SIC preceded DIC in most cases and that mortality rates of SIC and DIC cohorts were relatively high and comparable [15, 16]. As a result, the new guideline in 2019 recommended that septic patients with thrombocytopenia (platelet count < 150 × 10⁹/L) should be screened, first by using SIC diagnostic criteria and then by using ISTH DIC diagnostic criteria [14].

However, there is still a lack of predictive tools for coagulopathy in sepsis. The current SIC criteria serve as a diagnostic tool rather than a predictor of SIC. In our study, daily SIC scores were assessed to predict the SIC risk of the next day; it was demonstrated that the scoring system was outperformed by our predictive models in both internal and external validations. Furthermore, several new biomarkers have been found for the early detection of coagulopathy and DIC in sepsis or septic shock [17, 18]. However, these promising results are not ready for large-scale clinical practice due to the high cost and complicated test procedures [19].

Machine learning is a field of artificial intelligence that learns from data based on computational modeling. Advanced machine-learning models can fit high-order relationships between covariates and outcomes, and therefore, they excel in the analysis of complex signals in data-rich environments [20]. The aim of this study was to develop and validate machine-learning models for the early prediction of SIC, and to assess the feature importance in SIC prediction by interpreting the final model.

Source of data

We conducted our retrospective study based on two sizeable critical care databases named Medical Information Mart for Intensive Care (MIMIC)-IV [21] and the eICU Collaborative Research Database (eICU-CRD) [22]. The MIMIV-IV database is an updated version of MIMIC-III. A number of improvements have been made, including simplifying the structure, adding new data elements, and improving the usability of previous data elements. Currently, MIMIC-IV contains comprehensive and high-quality data of patients admitted to intensive care units (ICUs) at the Beth Israel Deaconess Medical Center between 2008 and 2019. The other database, eICU-CRD, is a multicenter database comprising de-identified health data associated with over 200,000 admissions to ICUs across the United States between 2014 and 2015. One author (QZ) obtained access to the two databases and was responsible for data extraction. The study was reported according to the recommendations of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [23].

Selection of participants

In MIMIC-IV, patients who fulfilled the definition of sepsis between 2008 and 2019 were included. According to the sepsis-3 criteria, sepsis was defined as a suspected infection combined with an acute increase in Sequential Organ Failure Assessment (SOFA) score ≥ 2 [1]. Patients with prescriptions of antibiotics and sampling of bodily fluids for microbiological culture were considered to have suspected infection. In line with previous research, when the antibiotic was given first, the microbiological sample must have been collected within 24 h; when the microbiological sampling occurred first, the antibiotic must have been administered within 72 h [24]. Hourly SOFA was assessed based on the clinical and laboratory data. In eICU-CRD, microbiology data was not well populated due to the limited availability of microbiology interfaces; instead, infection was identified according to documented diagnosis.

We only included patients who were older than 18 years old and spent more than 24 hours in ICU. No patients were excluded because of missing values. We made no attempt to estimate the sample size of the study; instead, all eligible patients in MIMIC-IV and eICU-CRD were included to maximize the statistical power of the predictive model.

Outcome (SIC)

Septic patients with coagulation disorders were identified according to the SIC criteria, as recommended [14]. The worst daily values of SIC-related indicators were extracted when sepsis definition was fulfilled. Then daily repeated scoring was performed. A patient was defined as SIC if he or she has a score ≥ 4 on that day.

Predictors of SIC

Clinical and laboratory variables were extracted during sepsis by using Structured Query Language (SQL). For the prediction of SIC, 88 variables were collected (Additional File 2: Table S2), including patient characteristics (age, gender, ethnicity, admission type), vital signs (respiratory rate, blood pressure, heart rate, spo2, and temperature), laboratory data (blood gas, blood routine, liver function, renal function, and coagulation profile), transfusion (red blood cells, platelet, and fresh frozen plasma) and urine output. Comorbidities were also collected based on the recorded International Classification of Diseases (ICD)-9 and ICD-10 codes, including hypertension, diabetes mellitus, chronic obstructive pulmonary disease, congestive heart failure, myocardial infarction, chronic kidney disease, leukemia, strokes, cancer, and liver disease. In addition, medications such as heparin, antibiotics and vasopressor, continuous renal replacement therapy (CRRT), and mechanical ventilation (MV) were collected. Last, the length of hospital stays, the length of ICU stays, and 28-day mortality were also analyzed but were not used for prediction.

Statistical analysis

Variable values on the first sepsis day were compared between SIC and non-SIC groups in MIMIC-IV. Values were presented as the means [standard deviations] (if normal) or medians [interquartile ranges] (if non-normal) for continuous variables, and total numbers [percentages] for categorical variables. Comparisons were made using the Student t-test or rank-sum test for continuous variables, and the Chi-square test or Fisher's exact test for categorical variables, as appropriate.

As shown in Fig. 1.A, our model generated a continuous prediction score on each day when patients were diagnosed with sepsis. The scores assessed the SIC risk of the next day. Prediction was not performed if SIC criteria were fulfilled on that day; when the patients recovered from SIC, our model restarted to predict. None of imputation methods was used for advanced boosting machine-learning methods because they can automatedly handle missing values; in contrast, missing values were imputed by the median values for continuous variables or mode values for categorical values when training other models. As shown in Fig. 1.B, we preliminarily compared the prediction performance of 15 algorithms using the PyCaret package, an open-sourced, automated machine-learning workflow. The assessment process was performed using 10-fold cross-validation. Accuracy and Area Under the receiver operating characteristic Curve (AUC) were calculated on each fold and pooled to evaluate each model. The most potential algorithm with the highest accuracy and the largest AUC was selected. Then, we performed fine-grained hyperparameter adjustment for the potential model using Bayesian Optimization Algorithm, an efficient constrained global optimization tool [25]. Hyperparameter search domains were listed in Additional File 1 (Table S3). The optimized model was believed as the best model for SIC prediction and was defined as the full model.

The effects of features on prediction scores were measured using the SHapley Additive exPlanations (SHAP) values, which assessed the importance of each feature using a game-theoretic approach based on the validation set [26]. We selected 15 features which had great importance and as easy as possible to collect in the clinical setting (Additional File 2: Table S2). Then, a compact model was trained for SIC prediction based on the selected features. Although this model was not so accurate as the full model, it is considered more practical in clinical settings.

External validation for the full and the compact models was performed in eICU-CRD. The median and 95% confidence intervals of AUC were calculated using the Bootstrap Resampling technique with 1000 times of iteration. Conventional Logistic Regression and SIC scoring system were assessed to predict SIC risk and were compared with our models in both internal and external validations. Additionally, performances of our models in different patient cohorts were assessed. Samples of the validation set were split into different groups, based on Acute Physiology and Chronic Health Evaluation (APACHE)-IV, age, the region of the United States, ethnicity, time since sepsis onset, and unit type. Two models were validated in each sub-cohort separately.

All analyses were performed using Python (Version 3.6), and p < 0.01 was considered statistically significant.

Baseline characteristics

As shown (Fig. 2), of 12381 septic patients in MIMIC-IV, 11362 were included in the final cohort. A total of 6744 patients had SIC during sepsis, and 4618 patients had not. A cohort of 35252 septic patients in eICU-CRD was included, and 111002 samples were derived.

Variables values on the first day of sepsis in MIMIC-IV were analyzed; the differences in characteristics were compared in Table 1. Patients in SIC group were significantly younger than those in the non-SIC group (67.9 [57.0, 78.7] vs. 69.1 [56.9,80.8]; p < 0.001). The SIC group had a higher rate of comorbidities, higher SAPS-II scores (44 [35, 54] vs. 37 [30, 45]; p < 0.001), higher SOFA scores (6 [4, 9] vs. 4 [3, 5]; p < 0.001), longer prothrombin time (PT) (16.9 [14.3, 21.8] vs. 13.0 [11.9, 14.1]; p < 0.001), less urine output (790 [300, 1545] vs. 1205 [605, 2015]; p < 0.001), higher rates of linezolid (2.9% vs. 1.7%; p < 0.001), vancomycin (55.6% vs. 46.0%; p < 0.001), CRRT (5.0% vs. 0.6%; p < 0.001), vasopressors (46.8% vs. 23.2%; p < 0.001) and MV (50.3% vs. 40.6%; p < 0.001), and higher 28-day mortality (27.0% vs. 10.8%; p < 0.001) than the non-SIC group. The length of hospital stays was also longer in SIC group than this in non-SIC group (14.4 [7.9, 26.7] vs. 10.9 [6.5, 19.5], p < 0.001).

Table 1

Baseline characteristics between SIC and non-SIC group
Variables		SIC (n = 6744)	non-SIC (n = 4618)	P-Value
Demographic variables
	Age, median [Q1,Q3]	67.87 [57.02,78.66]	69.05 [56.92,80.76]	< 0.001
	Male, n (%)	3770 (55.90)	2148 (46.51)	< 0.001
	BMI, mean (SD)	29.50 (8.35)	29.39 (9.82)	0.592
Admission type, n (%)				< 0.001
	Elective	96 (1.42)	62 (1.34)
	Emergency	4186 (62.07)	3070 (66.48)
	Urgent	1560 (23.13)	812 (17.58)
	Other	902 (13.37)	674 (14.60)
Ethnicity, n (%)				0.243
	Asian	167 (2.48)	143 (3.10)
	Black	739 (10.96)	525 (11.37)
	Hispanic	271 (4.02)	175 (3.79)
	White	4551 (67.48)	3062 (66.31)
	Other/Unknown	1016 (15.07)	713 (15.44)
Comorbidities, n (%)
	Hypertension	3837 (56.90)	2724 (58.99)	0.028
	DM	1405 (20.83)	820 (17.76)	< 0.001
	COPD	818 (12.13)	576 (12.47)	0.604
	CHF	2199 (32.61)	1148 (24.86)	< 0.001
	MI	748 (11.09)	341 (7.38)	< 0.001
	CKD	1948 (28.88)	961 (20.81)	< 0.001
	Leukemia	172 (2.55)	35 (0.76)	< 0.001
	Strokes	478 (7.09)	627 (13.58)	< 0.001
	Cancer	1432 (21.23)	747 (16.18)	< 0.001
	Liver disease	2033 (30.15)	296 (6.41)	< 0.001
Severity of illness, median [Q1,Q3]
	SAPS-II (ICU admission)	44 [35,54]	37 [30,45]	< 0.001
	SOFA (1st 24 h)	6 [4,9]	4 [3,5]	< 0.001
	GCS (1st 24 h)	15 [14,15]	14 [13,15]	< 0.001
Vital signs
	Heart rate (/min), mean (SD)	92.04 (18.15)	87.92 (16.23)	< 0.001
	Respiratory rate (/min), mean (SD)	20.96 (4.65)	20.41 (4.38)	< 0.001
	MAP (mmHg), mean (SD)	74.19 (10.59)	78.29 (12.20)	< 0.001
	Temperature (°C), mean (SD)	36.93 (0.75)	37.07 (0.64)	< 0.001
Laboratory tests
	Albumin (g/dl), mean (SD)	2.89 (0.68)	3.12 (0.64)	< 0.001
	BUN, median [Q1,Q3]	32.00 [19.33,51.00]	23.00 [14.00,37.00]	< 0.001
	Creatinine (mmol/L), median [Q1,Q3]	1.40 [0.90,2.40]	1.00 [0.70,1.55]	< 0.001
	INR, median [Q1,Q3]	1.55 [1.30,2.00]	1.20 [1.10,1.30]	< 0.001
	PT (s), median [Q1,Q3]	16.85 [14.30,21.81]	13.00 [11.90,14.10]	< 0.001
	Platelet (× 10⁹/L), median [Q1,Q3]	149.00 [92.33,234.10]	243.00 [187.50,324.00]	< 0.001
	RDW (%), mean (SD)	16.71 (2.76)	15.32 (2.14)	< 0.001
	WBC (× 10⁹/L), median [Q1,Q3]	11.85 [7.70,17.60]	11.95 [8.65,16.35]	0.091
	pH, mean (SD)	7.36 (0.09)	7.39 (0.08)	< 0.001
	PaO₂ (mmHg), median [Q1,Q3]	113.00 [87.00,159.43]	109.00 [83.90,155.00]	0.005
	PaCO₂ (mmHg), median [Q1,Q3]	39.00 [34.00,45.25]	41.00 [35.63,48.75]	< 0.001
	FiO₂ (%), median [Q1,Q3]	54.37 [45.00,75.00]	50.00 [40.00,67.92]	< 0.001
	PaO₂/FiO₂, median [Q1,Q3]	226.21 [156.21,308.00]	234.00 [166.54,313.75]	0.007
	BE (mEq/L), mean (SD)	-2.12 (5.88)	0.68 (5.41)	< 0.001
	Bicarbonate (mmol/L), mean (SD)	22.38 (5.35)	24.54 (5.15)	< 0.001
	Total CO₂ (mEq/L), mean (SD)	23.82 (6.42)	26.92 (6.58)	< 0.001
	Hematocrit (%), mean (SD)	30.23 (5.85)	31.73 (5.76)	< 0.001
	Hemoglobin (g/dl), mean (SD)	9.82 (1.94)	10.33 (1.98)	< 0.001
	Chloride (mmol/L), mean (SD)	103.22 (7.28)	103.41 (6.92)	0.164
	Calcium (mmol/L), mean (SD)	8.11 (1.26)	8.29 (1.13)	< 0.001
	Potassium (mmol/L), mean (SD)	4.20 (0.67)	4.17 (0.66)	0.008
	Lactate (mmol/L), median [Q1,Q3]	2.00 [1.35,3.27]	1.30 [1.00,1.90]	< 0.001
	Glucose (mmol/L), mean (SD)	147.31 (67.94)	150.29 (73.69)	0.030
Fluid balance
	Urine output (ml), median [Q1,Q3]	790 [300,1545]	1205 [605,2015]	< 0.001
Drug (1st 24 h), n (%)
	Heparin	2195 (32.55)	1999 (43.29)	< 0.001
	Linezolid	192 (2.85)	79 (1.71)	< 0.001
	Vancomycin	3751 (55.62)	2122 (45.95)	< 0.001
Num of antibiotic types (1st 24 h), n (%)				< 0.001
	0	1188 (17.62)	1111 (24.06)
	1	1459 (21.63)	1219 (26.40)
	2	1814 (26.90)	1179 (25.53)
	3	1338 (19.84)	718 (15.55)
	>=4	945 (14.01)	391 (8.47)
Support (1st 24 h), n (%)
	CRRT (1st 24 h)	334 (4.95)	28 (0.61)	< 0.001
	Vaso (1st 24 h)	3158 (46.83)	1070 (23.17)	< 0.001
	MV (1st 24 h)	3390 (50.27)	1874 (40.58)	< 0.001
Outcome
	Hosp. LOS (day), median [Q1,Q3]	14.44 [7.88,26.74]	10.89 [6.49,19.52]	< 0.001
	ICU LOS (day), median [Q1,Q3]	6.34 [2.94,13.03]	4.00 [2.10,8.94]	< 0.001
	28-day mortality, n (%)	1823 (27.03)	497 (10.76)	< 0.001
Abbreviations: SIC sepsis-induced coagulopathy, BMI body mass index, SD standard deviation, DM diabetes mellitus, COPD chronic obstructive pulmonary disease, CHF congestive heart failure, MI myocardial infarction, CKD chronic kidney disease, ICU intensive care unit, SAPS-II simplified acute physiology score II, SOFA sequential organ failure assessment, GCS Glasgow Coma Scale, MAP mean arterial pressure, BUN blood urea nitrogen, INR international normalized ratio, PT prothrombin time, RDW red cell distribution width, WBC white blood cell count, pH potential hydrogen, PaO₂ partial pressure of oxygen, PaCO₂ partial pressure of carbon dioxide in arterial blood, FiO₂ fraction of inspiration O₂, BE base excess, CRRT continuous renal replacement therapy, Vaso vasopressor, MV mechanical ventilation, Hosp. hospital, LOS length of stay.

Comparison of 15 models

Daily data were extracted, and 16183 samples for prediction in MIMIC-IV were created at last. Of them, 1489 were labeled as positive (SIC in the next day), 14694 were labeled as negative (still non-SIC in the next day). The prediction performances are listed in Table 2. As seen, Logistic Regression had an acceptable performance (Accuracy: 0.908; AUC: 0.746). Ensemble learning algorithms had better accuracy and larger AUC than others, such as Categorical Boosting (CatBoost) (Accuracy: 0.913; AUC: 0.841), Light Gradient Boosting (Accuracy: 0.912; AUC: 0.835) and Random Forest Classifier (Accuracy: 0.909; AUC: 0.760). The CatBoost model had the most powerful discrimination to predict SIC risk, and we optimized it in the next step.

Table 2

Performance of different models in internal validation
	Model	Accuracy	AUC
1	CatBoost Classifier	0.913 (± 0.004)	0.841 (± 0.025)
2	Light Gradient Boosting	0.912 (± 0.005)	0.835 (± 0.024)
3	Extreme Gradient Boosting	0.912 (± 0.004)	0.837 (± 0.025)
4	Gradient Boosting Classifier	0.911 (± 0.005)	0.832 (± 0.023)
5	Extra Trees Classifier	0.911 (± 0.002)	0.819 (± 0.032)
6	Random Forest Classifier	0.909 (± 0.002)	0.760 (± 0.022)
7	Ridge Classifier	0.908 (± 0.003)	0.753 (± 0.031)
8	Logistic Regression	0.908 (± 0.002)	0.746 (± 0.030)
9	K Neighbors Classifier	0.904 (± 0.001)	0.611 (± 0.040)
10	Ada Boost Classifier	0.902 (± 0.003)	0.804 (± 0.029)
11	Linear Discriminant Analysis	0.902 (± 0.003)	0.796 (± 0.027)
12	Multi-Level Perceptron	0.883 (± 0.004)	0.754 (± 0.022)
13	Decision Tree Classifier	0.861 (± 0.003)	0.593 (± 0.019)
14	SVM – RBF Kernel	0.859 (± 0.004)	0.777 (± 0.015)
15	Naive Bayes	0.805 (± 0.005)	0.756 (± 0.031)
Abbreviations: AUC area under receiver operating characteristic curve, CatBoost Categorical Boosting, SVM support vector machine, RBF Radial Basis Function.

Full and Compact Models

Fifteen iterations of optimization were performed. The optimized CatBoost model reached the greatest AUC in our study (0.869 [0.850, 0.886]). The final hyperparameters settings were listed in Table S3 in Additional File 1. SHAP values were calculated and plotted in Fig. 3. The summary plot sorts features by the sum of SHAP value magnitudes over all samples and shows the distribution of the impacts that each feature has on the full model output. As seen, the coagulation profile (platelet, International Normalized Ratio, PT) and renal function indicators (urine output, creatinine) are the most important features to distinguish SIC and non-SIC groups. Fifteen features were selected based on their SHAP values and clinical availability. The compact CatBoost model was built based on the selected features. It had a slightly smaller AUC (0.854 [0.832, 0.872]) but is considered more practical in clinical practice. The medians and 95% confidence intervals of AUCs were plotted in Fig. 4 for comparing the discrimination of different methods in MIMIC-IV. As shown, our two models outperformed the conventional Logistic Regression (0.746 [0.735, 0.755]) and SIC scoring system (0.709 [0.687, 0.733]) in terms of SIC prediction.

Prediction performance in eICU-CRD

Results of external validation were shown in Fig. 4 (0.842 [0.837, 0.846] for the full model, and 0.803 [0.798, 0.809] for the compact model). As seen, SIC scoring system had a better predictive power (0.752 [0.747, 0.757]) than in MIMIC-IV but was still worse than our two models (p < 0.001), while Logistic Regression had the poorest generalization ability (0.660 [0.653, 0.667]). The average performances of four predictive methods were summarized in Table 3.

Table 3

Performance of the final models and SIC scores in internal and external validations
	Internal Validation (MIMIC-IV)				External Validation (eICU-CRD)
Model	AUC	Youden	Sensitivity	Specificity	AUC	Youden	Sensitivity	Specificity
The full model	0.869	0.577	0.820	0.757	0.842	0.54	0.8	0.741
The compact model	0.854	0.564	0.848	0.716	0.803	0.477	0.745	0.732
Logistic Regression	0.746	0.433	0.753	0.680	0.660	0.230	0.582	0.648
SIC scores	0.709	0.368	0.707	0.661	0.752	0.448	0.655	0.793
The discrimination of three models (the full model, the compact model and Logistic Regression) and SIC scores were compared in internal and external validations. The full and the compact models were developed in MIMIC-IV, based on all or selected features, respectively. Logistic Regression was developed based on all features. Besides, the current SIC score was used to predict patient’s SIC risk of the next day. Youden Index, defined as Sensitivity + Specificity – 1, and AUC assessed the performance of different models. All statistics were the median values in 1000 iterations of the Bootstrap Resampling technique.
Abbreviations: SIC Sepsis-induced coagulopathy; AUC area under receiver operating characteristic curve; MIMIC Medical Information Mart for Intensive Care; eICU-CRD eICU Collaborative Research Database.

Model performance in different patient cohorts in eICU-CRD was shown in Fig. 5. As seen, two models had the greatest AUC for patients who had APACHE-IV scores between 81 and 100, whose ages were younger than 65 years old, or who were admitted in NICU and SICU. The two models maintained good performance over four regions of the United States. Besides, the two models had better discrimination when sepsis lasted for several days. A similar sub-cohort analysis was also performed in MIMIC-IV (Additional File 1: Fig. S1).

Model interpretation

The summary plot of SHAP in Fig. 3 provides an overview of feature impacts on the final models. Additionally, the prediction results of two specific instances were explained in Fig. 6. The bars in red and blue represent risk factors and protective factors, respectively; longer bars mean greater feature importance. For the instance in Fig. 6.A, although her coagulation profile was still normal, she was in poor circulatory status with high lactate and vasopressor administration. The model successfully predicted she would have SIC the next day. For the instance in Fig. 6.B, his condition was more moderate, and our model predicted a low-risk value.

Website-based tool

A website-based tool was established for clinicians to use the compact model, http://www.aimedicallab.com/tool/sic_risk.html. The SIC risk of the next day can be assessed by using this tool, and interpretation of the prediction result in the instance level will be shown to the user.

Our study analyzed and compared data of patients who would and who would not have SIC on the first sepsis day. Two variants of machine-learning models were developed (the full and the compact models), and could dynamically predict SIC with significantly improved accuracy. The relationships between clinical variables and SIC were analyzed based on model interpretation.

Our study compared the differences in characteristics between SIC and non-SIC groups on the onset of sepsis. As shown in Table 1, SIC patients were significantly younger but had worse physiological status (higher Simplified Acute Physiology Score [SAPS]-II, SOFA, and rate of support treatment) than those who were non-SIC. More types of antibiotics and a lower rate of heparin were administered to the SIC group on the first day. Interestingly, linezolid and vancomycin were administered to a higher rate of SIC patients. This was probably because patients with SIC had more severe infection. On the other hand, the administration of two antibiotics could cause a decrease in platelet and exacerbate clotting abnormalities [27, 28]. Additionally, the SIC group had a significantly higher mortality rate and longer length of hospital/ICU stays than the non-SCI group, consistent with the previous research [13].

Currently, there is a lack of reliable tools for the early prediction of coagulopathy in septic patients. Our study has demonstrated that advanced machine-learning algorithms can predict SIC with high accuracy and excellent AUC. They outperformed conventional Logistic Regression and SIC scores in both internal and external validations. CatBoost, an open-sourced gradient boosting algorithm, has not been widely adopted in critical care research. Gradient boosting is a powerful machine-learning technique that iteratively trains a weak classifier (e.g., decision tree) to fit residuals of previous models. Among these models, CatBoost successfully handles categorical features and takes advantage of dealing with them during training instead of preprocessing time [29]. That means categorial features no longer need to be encoded, and a CatBoost model can be successfully developed based on raw data. Another advantage of the algorithm is that it uses a new schema to calculate leaf values when selecting the tree structure. The schema helps to reduce overfitting, the major problem that constrains the generalization ability of machine-learning models [29].

In this study, we developed two variants of CatBoost models that can identify patients with a high risk of SIC and provide clinical decision-makers with more information. Generally, based on more valuable variables, models have better discrimination but worse clinical usability. Therefore, in our study, two model variants were developed for different application scenarios. The full model predicted SIC based on 88 clinical variables and reached the greatest AUC in this study. In external validation, the full model maintained good discrimination and only had a slight reduction in AUC. However, it is tough to collect 88 variables and apply this model. As a result, the full model is recommended to the hospitals with a well-designed clinical data system. By contrast, the compact model was trained based on 15 selected variables. Under the condition of ensuring necessary accuracy, it achieved practicality as far as possible. As shown in Fig. 5, our models had great and comparable AUC in different patient cohorts, demonstrating that machine-learning models based on big data have good generalization capability. Besides, a website tool was developed to help clinicians to use the compact model in clinical practice. By logging on the website and entering the values of 15 variables, our compact model will give the prediction results, and interpretation of the prediction result will be shown to the user.

By interpreting the full model, it was found that many clinical variables can help to indicate the risk of SIC. In this study, renal function indicators (urine output and creatinine) were important variables next only to coagulopathy profile. As shown in Fig. 3, patients with poorer renal function (less urine output and higher serum creatinine) tended to have a higher risk of SIC. Also, body mass index (BMI), vital signs (heart rate and mean arterial pressure), laboratory tests (such as lactate and white blood cell count), the use of MV and vasopressor, and SAPS-II scores can help assess the risk of SIC. In addition, prediction results can be interpreted in the instance level, as shown in Fig. 6, which makes our model clinically explainable.

Several limitations of this study should be considered. First, only septic adults in critical care were included, whereas hospitalized sepsis cases were not analyzed. Besides, considering the immaturity of the coagulation system in children, especially newborns, more research is needed for SIC in children with sepsis. Second, our models screen out patients with high risks of SIC but do not indicate who will benefit from the anticoagulant therapy. It is still up to clinicians to decide whether to administrate anticoagulant agents. However, the process from sepsis to severe coagulopathy is a continuous condition arising from coagulation disorder. Early and accurate prediction of SIC can provide more time for clinical workers to adjust treatment strategies, and also help to study the potential effect of anticoagulant therapy in early stage. Third, this is a retrospective observational study. Data missing and input errors exist, despite the very high quality of the MIMIC-IV and eICU-CRD databases. Therefore, prospective validation is still needed in the future. Compared with septic shock, for which advances have been made in recent years, giving rise to significant survival improvements, there is still a long way to go for diagnosis and management of sepsis-associated coagulopathy.

In conclusion, the present study developed two variants of the CatBoost model, which can differentiate septic patients who would and who would not have SIC. The full model had the highest accuracy, whereas the compact model is more practical in clinical settings. Two models outperformed conventional Logistic Regression and SIC scores in terms of SIC prediction in both internal and external validations. Prediction results can be illustrated by using SHAP values, making our models clinically interpretable.

Additional files

SIC: Sepsis-induced coagulopathy; ICU:intensive care unit; MIMIC:Medical Information Mart for Intensive Care; eICU-CRD:eICU Collaborative Research Database; CatBoost:Categorical Boosting; SHAP:SHapley Additive exPlanations; SSC:the Scientific and Standardization Committee; DIC:Disseminated Intravascular Coagulation; ISTH:the International Society of Thrombosis and Haemostasis; TRIPOD:Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis; SOFA:Sequential Organ Failure Assessment; SAPS-II:Simplified Acute Physiology Score II; SQL:Structured Query Language; ICD:International Classification of Diseases; CRRT:Continuous renal replacement therapy; MV:Mechanical ventilation; AUC:Area under the receiver operating characteristic curve; PT:Prothrombin time; BMI:body mass index; APACHE:Acute Physiology and Chronic Health Evaluation-IV.

Acknowledgments

We would like to thank the Massachusetts Institute of Technology and the Beth Israel Deaconess Medical Center for the MIMIC project. We also would like to thank the Philips eICU Research Institute and Philips Healthcare for their contribution to the eICU-CRD project.

Funding

This article was supported by Research Funds of Shanghai Municipal Health Commission (2019ZB0105), Natural Science Foundation of Shanghai (20ZR1411100), Program of Shanghai Academic/Technology Research Leader (20XD1421000), National Natural Science Foundation of China (82070085).

Availability of data and materials

The MIMIC-IV data were available on the project website at https://mimic-iv.mit.edu/, while the eICU-CRD data were available at https://eicu-crd.mit.edu/.

Authors' contributions

(I) Conception and design: Qin-Yu Zhao, Le-Ping Liu, Jing-Chao Luo

(II) Administrative support: Rong Gui, Guo-Wei Tu, Zhe Luo

(III) Collection and assembly of data: Qin-Yu Zhao

(IV) Data analysis and interpretation: Qin-Yu Zhao, Le-Ping Liu

(V) Manuscript writing: All authors

(VI) Final approval of manuscript: All authors

Ethics approval and consent to participate

The study was an analysis of two third-party anonymized publicly available databases with pre-existing institutional review board (IRB) approval.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche JD, Coopersmith CM et al: The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016, 315(8):801-810.
Martin GS, Mannino DM, Eaton S, Moss M: The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med 2003, 348(16):1546-1554.
Lyons PG, Micek ST, Hampton N, Kollef MH: Sepsis-Associated Coagulopathy Severity Predicts Hospital Mortality. Crit Care Med 2018, 46(5):736-742.
Levi M, van der Poll T: Coagulation and sepsis. Thromb Res 2017, 149:38-44.
Levi M, Ten Cate H: Disseminated intravascular coagulation. N Engl J Med 1999, 341(8):586-592.
Allingstrup M, Wetterslev J, Ravn FB, Moller AM, Afshari A: Antithrombin III for critically ill patients. Cochrane Database Syst Rev 2016, 2:CD005370.
Warren BL, Eid A, Singer P, Pillay SS, Carl P, Novak I, Chalupa P, Atherstone A, Penzes I, Kubler A et al: Caring for the critically ill patient. High-dose antithrombin III in severe sepsis: a randomized controlled trial. JAMA 2001, 286(15):1869-1878.
Dhainaut JF, Yan SB, Joyce DE, Pettila V, Basson B, Brandt JT, Sundin DP, Levi M: Treatment effects of drotrecogin alfa (activated) in patients with severe sepsis with or without overt disseminated intravascular coagulation. J Thromb Haemost 2004, 2(11):1924-1933.
Iba T, Gando S, Thachil J: Anticoagulant therapy for sepsis-associated disseminated intravascular coagulation: the view from Japan. J Thromb Haemost 2014, 12(7):1010-1019.
Kienast J, Juers M, Wiedermann CJ, Hoffmann JN, Ostermann H, Strauss R, Keinecke HO, Warren BL, Opal SM, KyberSept i: Treatment effects of high-dose antithrombin without concomitant heparin in patients with severe sepsis with or without disseminated intravascular coagulation. J Thromb Haemost 2006, 4(1):90-97.
Umemura Y, Yamakawa K, Ogura H, Yuhara H, Fujimi S: Efficacy and safety of anticoagulant therapy in three specific populations with sepsis: a meta-analysis of randomized controlled trials. J Thromb Haemost 2016, 14(3):518-530.
Umemura Y, Yamakawa K: Optimal patient selection for anticoagulant therapy in sepsis: an evidence-based proposal from Japan. J Thromb Haemost 2018, 16(3):462-464.
Iba T, Nisio MD, Levy JH, Kitamura N, Thachil J: New criteria for sepsis-induced coagulopathy (SIC) following the revised sepsis definition: a retrospective analysis of a nationwide survey. BMJ Open 2017, 7(9):e017046.
Iba T, Levy JH, Warkentin TE, Thachil J, van der Poll T, Levi M, Scientific, Standardization Committee on DIC, the S, Standardization Committee on P et al: Diagnosis and management of sepsis-induced coagulopathy and disseminated intravascular coagulation. J Thromb Haemost 2019, 17(11):1989-1994.
Iba T, Arakawa M, Di Nisio M, Gando S, Anan H, Sato K, Ueki Y, Levy JH, Thachil J: Newly Proposed Sepsis-Induced Coagulopathy Precedes International Society on Thrombosis and Haemostasis Overt-Disseminated Intravascular Coagulation and Predicts High Mortality. J Intensive Care Med 2020, 35(7):643-649.
Iba T, Arakawa M, Levy JH, Yamakawa K, Koami H, Hifumi T, Sato K: Sepsis-Induced Coagulopathy and Japanese Association for Acute Medicine DIC in Coagulopathic Patients with Decreased Antithrombin and Treated by Antithrombin. Clin Appl Thromb Hemost 2018, 24(7):1020-1026.
Abrams ST, Morton B, Alhamdi Y, Alsabani M, Lane S, Welters ID, Wang G, Toh CH: A Novel Assay for Neutrophil Extracellular Trap Formation Independently Predicts Disseminated Intravascular Coagulation and Mortality in Critically Ill Patients. Am J Respir Crit Care Med 2019, 200(7):869-880.
Delabranche X, Quenot JP, Lavigne T, Mercier E, Francois B, Severac F, Grunebaum L, Mehdi M, Zobairi F, Toti F et al: Early Detection of Disseminated Intravascular Coagulation During Septic Shock: A Multicenter Prospective Study. Crit Care Med 2016, 44(10):e930-939.
Iba T, Umemura Y, Watanabe E, Wada T, Hayashida K, Kushimoto S, Japanese Surviving Sepsis Campaign Guideline Working Group for disseminated intravascular c: Diagnosis of sepsis-induced disseminated intravascular coagulation and coagulopathy. Acute Med Surg 2019, 6(3):223-232.
Beam AL, Kohane IS: Big Data and Machine Learning in Health Care. JAMA 2018, 319(13):1317-1318.
Johnson A, Bulgarelli L, Pollard T, Horng S, Celi LA, Mark R: MIMIC-IV (version 0.4). PhysioNet 2020.
Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O: The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data 2018, 5:180178.
Collins GS, Reitsma JB, Altman DG, Moons KG: Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 2015, 13:1.
Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, Rubenfeld G, Kahn JM, Shankar-Hari M, Singer M et al: Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016, 315(8):762-774.
Snoek J, Larochelle H, Adams RP: Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems: 2012; 2012: 2951-2959.
Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, Katz R, Himmelfarb J, Bansal N, Lee SI: From Local Explanations to Global Understanding with Explainable AI for Trees. Nat Mach Intell 2020, 2(1):56-67.
Kishor K, Dhasmana N, Kamble SS, Sahu RK: Linezolid Induced Adverse Drug Reactions - An Update. Curr Drug Metab 2015, 16(7):553-559.
Mohammadi M, Jahangard-Rafsanjani Z, Sarayani A, Hadjibabaei M, Taghizadeh-Ghehi M: Vancomycin-Induced Thrombocytopenia: A Narrative Review. Drug Saf 2017, 40(1):49-59.
Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A: CatBoost: unbiased boosting with categorical features. In: Advances in neural information processing systems: 2018; 2018: 6638-6648.

Additional File 1:

Table S1. Sepsis-induced coagulopathy (SIC) criteria

Table S3. Hyperparameter search domain in Bayesian Optimization and final settings

Table S4. Results of Logistic Regression

Figure S1. Model performance in different patient cohorts in MIMIC-IV

Figure S2. Model interpretation of the full model in eICU-CRD

Figure S3. Model interpretation of the compact model in eICU-CRD

Additional File 2:

Table S2. Predictors extracted in MIMIC-IV and eICU-CRD

Download PDF

Version 1

posted

You are reading this latest preprint version

A machine-learning approach for dynamic prediction of sepsis-induced coagulopathy in critically ill patients with sepsis: an integrated analysis of the MIMIC-IV and eICU-CRD databases

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Source of data

Selection of participants

Outcome (SIC)

Predictors of SIC

Statistical analysis

Results

Baseline characteristics

Comparison of 15 models

Full and Compact Models

Prediction performance in eICU-CRD

Model interpretation

Website-based tool

Discussion

Conclusions

Additional files

Abbreviations

Declarations

References

Additional files

Supplementary Files

Status:

Version 1