Predicting Outcomes in Older ED Patients with Influenza in Real Time Using a Big Data-Driven and Machine Learning Approach to the Hospital Information System

doi:10.21203/rs.3.rs-55908/v1

Download PDF

Research article

Predicting Outcomes in Older ED Patients with Influenza in Real Time Using a Big Data-Driven and Machine Learning Approach to the Hospital Information System

https://doi.org/10.21203/rs.3.rs-55908/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Predicting outcomes in older patients with influenza in the emergency department (ED) by machine learning (ML) has never been implemented. Therefore, we conducted this study to clarify the clinical utility of implementing ML.

Methods: We recruited 5508 older ED patients (≥65 years old) in three hospitals between 2009 and 2018. Patients were randomized into a 70%/30% split for ML model training and testing. Using 10 clinical variables from their electronic health records, a random forest model using the synthetic minority oversampling technique preprocessing algorithm was constructed to predict five outcomes.

Results: The areas under the curves of predicting outcomes using the random forest model were 0.807 for hospitalizations, 0.974 for pneumonia, 0.994 for sepsis or septic shock, 0.981 for intensive care unit admission, and 0.851 for death in the testing data. The predictive model was further applied in the hospital information system to assist physicians’ decisions in real time.

Conclusions: ML is a promising way to assist physicians in predicting outcomes in older ED patients with influenza in real time. Evaluations of the effectiveness and impact are needed in the future.

Critical Care & Emergency Medicine

emergency department

death

influenza

hospital information system

machine learning

mortality

older

prediction

random forest

The rapidly aging population is one of the most important issues worldwide. In the United States, older adults (≥65 years old) were 15.2% of the total population in 2016, are projected to be 20% in 2030, and 23.5% in 2060 [1]. Taiwan is one of the rapidly aging countries in the world. In 2018, the number of deaths was nearly equal to that of births [2]. Older adults represented 14% of the total population in 2018, and are projected to be 20% in 2025 [2].

Influenza is a life-threatening disease for the older population. An Asian study revealed that older adults contributed to 70%-90% of total deaths [3]. The mortality rate of influenza in older adults was nearly 39-fold that of the population aged 40-64 years old [3]. Influenza-related complications, including cardiorespiratory diseases, pneumonia, chronic obstructive pulmonary disease (COPD), and ischemic heart diseases, are the common causes of death [3].

Because of limited medical resources during the influenza season, predicting outcomes in older adults with influenza and their subsequent disposition becomes a critical issue. In our previous study, we recruited 409 older patients with influenza for developing a Geriatric Influenza Death Score (GID score) [4]. This study identified five independent mortality predictors: severe coma (Glasgow Coma Scale [GCS] ≤8), past histories of malignancy and coronary artery disease (CAD), elevated C-reactive protein (CRP) levels (>10 mg/dl), and bandemia (>10% band cells) [4]. Three mortality risk and disposition groups were formed according to five predictors: (1) low risk (1.1%; 95% confidence interval [CI], 0.5%–3.0%); (2) moderate risk (16.7%; 95% CI, 9.3%–28.0%); and (3) high risk (40%; 95% CI, 19.8%–64.2%). The GID score has an area under the receiver operating characteristic curve of 0.86, and Hosmer-Lemeshow goodness of fit of 0.578 [4].

Although the GID score is a potentially good clinical decision rule (CDR) in older adults with influenza, it has the limitations of the small size of derivation sample and lacks both automation and feedback in real time to clinicians [5]. In recent years, a great deal of evidence showed that artificial intelligence (AI), including machine learning (ML) techniques, could handle more variables that are already available through electronic health records (EMRs) and may better predict patient outcomes [5]. We performed searches on Google Scholar and PubMed using the keywords “AI,” “death,” “influenza,” “machine learning,” “mortality older adult,” and “outcome,” but we did not find any AI application in this field. Therefore, we conducted the present study for clarifying the issue and applying it in the hospital information system (HIS) to assist decision making in real time.

2.1 Study design, setting, and participants

We established a multi-disciplinary team, including emergency physicians, data scientists, information engineers, nurse practitioners, and quality managers for this project (Figure 1). After our literature review, we decided to use the previous study about predicting mortality in older ED patients with influenza as the main reference [4]. We identified all older patients (≥65 years old) with influenza who visited the ED between January 1, 2009, and December 31, 2018, from the EMRs of three hospitals: Chi Mei Medical Center, Chi Mei Hospital, Liouying, and Chi Mei Hospital, Chiali. The present study hospitals are not the hospitals for developing the GID score. The criteria of influenza are defined as the diagnosis of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) of 487 or 488 or a prescription of Oseltamivir, Peramivir, or Relenza in the index ED visit.

2.2 Definitions of variables

We included age, sex, vital signs, and past histories of hypertension (ICD-9: 401-405), diabetes (ICD-9-CM: 250), COPD (ICD-9-CM: 496), CAD (ICD-9-CM: 410-414), stroke (ICD-9: 436-438), malignancy (ICD-9: 140-208), congestive heart failure (CHF, ICD-9-CM: 428), dementia (ICD-9: 290), bedridden, feeding with a nasogastric tube, and nursing home resident, laboratory data including white blood cell count (WBC), bandemia, hemoglobin, platelet, serum creatinine, CRP, procalcitonin, glucose, Na, K, GOT, and GPT for this study. We adopted 10 potential predictors proposed in the previous study as the feature variables for the ML [4]: (1) tachypnea (respiratory rate >20/min); (2) severe coma (GCS ≤8); (3) history of hypertension; (4) history of CAD; (5) history of malignancy; (6) bedridden; (7) leukocytosis (WBC >12,000 cells/mm); (8) bandemia (>10% band cells); (9) anemia (hemoglobin <12 mg/dL); and (10) elevated CRP (>10 mg/dL). The patients who did not have a record of subsequent follow-up were excluded. Missing laboratory data were treated as the normal values (i.e., respiratory rate: 12/min, GCS: 15, WBC: 7000 cells/mm, band form: 0%, hemoglobin: 12 mg/dL, and CRP: 2.5 mg/dL).

2.3 Outcome measurements

The outcome measurements were binary coded as the follows: (1) hospitalization; (2) complications with pneumonia (ICD-9-CM: 480-486): (3) complications with sepsis or septic shock (ICD-9-CM: 038, 790.7, 995.91, 995.91, 785.52); (4) admitted to intensive care unit (ICU); and (5) death.

2.4 Ethical statement

The present study was approved by the institutional review board in the Chi Mei Medical Center. Informed consent from the participants was waived because this study is retrospective, and it contains de-identified information, which does not affect the rights and welfare of the participants.

2.5 Data processing, comparison, and application in the HIS

First, we extracted, transformed, and validated the data from the HIS into a data mart. Missing and ambiguous data were carefully processed at this step. Second, we used the synthetic minority oversampling technique (SMOTE) preprocessing the algorithm for the model training and testing because of imbalanced samples. Third, we compared accuracy, precision, sensitivity, specificity, positive predictive value, negative predictive value, F1, and the area under the curve (AUC) among the analyses of the random forest, logistic regression, support vector machine (SVM), K-nearest neighbors (KNN), and light gradient boosting machine (LightGBM) and decided to use the random forest algorithm for its best model evaluation in most outcomes. All four models were trained and tested on a randomly partitioned 70%/30% split of the data. Fourth, we deployed the model in the AI web service and integrated it with the HIS in the ED. After two-months of pilot testing and validating, we launched the condition prediction application in the HIS to assist physicians for decision making in real time.

2.6 Patient and public involvement

Patients and the public were not be involved in this study.

In total, we recruited 5508 older ED patients into the present study. The mean ± standard deviation (SD) of age was 76.61 ± 7.44 years old, and the female proportion was 50.67% (Table 1). The proportion of the three age subgroups were young elderly (43.06%), moderately elderly (40.56%), and old elderly (16.38%). The mean ± SD of respiratory rate and GCS were 19.16 ± 3.94 breaths/min and 14.41 ± 1.84, respectively. The histories of ED patients were hypertension (56.05%), CAD (19.64%), malignancy (14.32%), and bedridden (31.94%). The mean ± SD of WBC, hemoglobin, and CRP were 8670.00 ± 4220.00 cells/mm³, 12.42 ± 1.95 mg/dL, and 42.06 ± 50.98 mg/dL, respectively. The proportions of patient outcomes were hospitalization (47.33%), complications with pneumonia (37.71%), complications with sepsis or septic shock (5.57%), admitted to ICU (1.07%), and death (2.20%).

The random forest model with the SMOTE preprocessing algorithm showed that the prediction of sepsis or septic shock had the best AUC of 0.994, followed by ICU admission (0.982), pneumonia (0.974), death (0.851), and hospitalization (0.807) (Table 2). Feature importance according to a random forest model for predicting the five outcomes was also reported (Supplementary Figure 1).

Comparisons of predictive accuracies among the random forest, logistic regression, SVM, KNN, and LightGBM revealed that the random forest model had the best AUC, accuracy, precision, recall, and F1 than did other models except ICU admission (random forest vs. LightGBM = 0.982 v.s 0.984) (Table 3 and Supplementary Figure 2). We applied the random forest for predicting outcomes in older ED patients in the HIS to assist decision making in real time (Supplementary Figure 3). The time of generation of the prediction results was within one second.

The present study revealed that the random forest model with the SMOTE preprocessing algorithm had the highest predictive accuracy for predicting five outcomes in older ED patients with influenza. Predictive accuracies by the random forest were better than the traditional analytic model (i.e., logistic regression) and other ML models, including SVM, KNN, and LightGBM except in the outcome of ICU admission. The predictions are very fast, in real time, and actionable, which provide prognostic information to assist in decision making, including disposition and outcome explanation.

The random forest is superior to the traditional model for developing CDR. One possible reason for the lower predictive accuracies of logistic regression is that it lacks external validation [5]. Traditional CDRs, including the GID score, are typically developed by gathering data at one or more hospitals, and then using both to derive and validate a model from a chosen set of predictors. The developed CDRs are then used in other hospitals, different from the original study hospital [6]. A recent study reviewed 127 new prediction models and showed that external independent validation was uncommon in predictive models [7]. Predictive performance in external validation tends to be worse than the original study [7]. In contrast to the GID score derived from other hospitals, we used local real-world big data in multi-centers to make predictions about the local population, which improves accuracy over the traditionally derived model. The variables used in the present study are structured data from the local EMR without being subjected to ambiguous clinical definitions or biases of data collection.

The random forest model is an ensemble learning method for classification and regression [8, 9]. It combines many binary decision trees, which are built by several bootstrapped learning samples, and chooses a subset of variables randomly at each node [8, 9]. Each tree in the random forest will vote for some input x, then the voting majority of trees will determine the output of the classifier [10]. The random forest can use a large number of trees in the ensemble to handle high dimensional data [10]. The random forest is a common method adopted for predicting outcomes and selecting predictors in the ED. A study about predicting in-hospital mortality in ED patients with sepsis revealed that the AUC of the random forest was 0.86, superior to the CART (classification and regression tree) model (0.69); logistic regression model (0.76); CURB-65 (Confusion, Urea, Respiratory rate, Blood pressure plus age ≥ 65 years old) (0.73); MEDS (mortality in emergency department sepsis) (0.71); and mREMS (modified rapid emergency medicine score) (0.72) [5]. A study used the random forest to select the most relevant variables for major adverse cardiac events in ED patients with chest pain [8]. They found that the selection predictor by the random forest is promising in discovering a few relevant and significant predictors [8].

The SMOTE adopted in the present study is the most common and effective method of oversampling for adjusting imbalanced data [11]. SMOTE solves the problems of both high-class skew and high sparsity and works in the ‘‘feature space’’ rather than ‘‘data space’’ [12]. By taking each minority class sample and the K-nearest neighbors, SMOTE creates synthetic samples for effectively forcing the decision region of the minority class to become more general [12]. Without duplicating the data, SMOTE increases the data space and amplifies the features of the minority class [12]. Studies with SMOTE preprocessing in health care are also acceptable [13, 14].

According to our literature review, the present study has the strength of being the first real-time prediction application in the HIS using ML for older ED patients with influenza. The limitations are as follows. First, ML has the problems of interpretability and inferences about variables. Second, we did not compare the predictive accuracy between this model and the physician’s judgment. Further studies about this issue, as well as the impact of this model, are warranted. Third, the application may not be generalized to other hospitals because it needs building an infrastructure to make real-time predictive analytics a reality.

We developed the first real-time prediction application in the HIS for predicting outcomes in older ED patients with influenza using a big data-driven and machine learning approach. This real-time prediction is a promising way to assist the physician’s decision making and explanations to patients and their families. Further studies about the predictive accuracy between this model and both the physician’s judgment and impact of the application are needed.

ED: emergency department

ML: machine learning

COPD: chronic obstructive pulmonary disease

GID: Geriatric Influenza Death Score

GCS: Glasgow Coma Scale

CAD: coronary artery disease

CRP: C-reactive protein

CI: confidence interval

CDR: clinical decision rule

AI: artificial intelligence

EMRs: electronic medical records

HIS: hospital information system

ICD-9-CM: International Classification of Diseases, Ninth Revision, Clinical Modification

ICU: intensive care unit

SMOTE: synthetic minority oversampling technique

AUC: area under the curve

SVM: support vector machine

KNN: K-nearest neighbors

LightGBM: light gradient boosting machine

SD: standard deviation

CART: classification and regression tree

CURB-65: Confusion, Urea, Respiratory rate, Blood pressure plus age ≥ 65

MEDS: mortality in emergency department sepsis

mREMS: modified rapid emergency medicine score

Ethics approval and consent to participate

Consent for publication

Not applicable.

Availability of data and materials

The datasets used during the current study are available from the corresponding author on reasonable request.

Competing interests

All authors denied any financial and non-financial competing interests.

Funding

This work was supported by Chi Mei Medical Center, [Grant number CMFHR108119].

Authors’ contributions

CC Hsu, TH Tan, CF Liu, and CC Huang designed and conceived this study and wrote the manuscript. CJ Chen and TL Liu performed the data processing, deployment in AI web service, integration with HIS, testing the application, and launching the application in the HIS. TL Liu performed model training and testing and statistical analysis. SL Hsu, HJL, and JJ Wang provided professional discussions and suggestions and wrote the manuscript. All authors read and approved the final manuscript.

Acknowledgment

We thank Miss Yu-Shan Ma for statistical assistance and Enago for the English revision.

An Aging Nation: Projected Number of Children and Older Adults [https://www.census.gov/library/visualizations/2018/comm/historic-first.html]
Population Projections for R.O.C. (Taiwan): 2016~2060 [http://www.ndc.gov.tw/en/cp.aspx?n=2E5DCB04C64512CC&s=002ABF0E676F4DB5]
Wong CM, Chan KP, Hedley AJ, Peiris JS: Influenza-associated mortality in Hong Kong. Clin Infect Dis 2004, 39(11):1611-1617.
Chung JY, Hsu CC, Chen JH, Chen WL, Lin HJ, Guo HR, Huang CC: Geriatric influenza death (GID) score: a new tool for predicting mortality in older people with influenza in the emergency department. Sci Rep 2018, 8(1):9312.
Taylor RA, Pare JR, Venkatesh AK, Mowafi H, Melnick ER, Fleischman W, Hall MK: Prediction of In-hospital Mortality in Emergency Department Patients With Sepsis: A Local Big Data-Driven, Machine Learning Approach. Acad Emerg Med 2016, 23(3):269-278.
Toll DB, Janssen KJ, Vergouwe Y, Moons KG: Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008, 61(11):1085-1094.
Siontis GC, Tzoulaki I, Castaldi PJ, Ioannidis JP: External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015, 68(1):25-34.
Liu N, Koh ZX, Goh J, Lin Z, Haaland B, Ting BP, Ong ME: Prediction of adverse cardiac events in emergency department patients with chest pain using machine learning for variable selection. BMC Med Inform Decis Mak 2014, 14:75.
Genuer R, Poggi, J.M., Tuleau-Malot, C.: Variable selection using random forests. Pattern Recogn Lett 2010, 31:2225–2236.
Khalilia M, Chakraborty S, Popescu M: Predicting disease risks from highly imbalanced data using random forest. BMC Med Inform Decis Mak 2011, 11:51.
Wang Y, Wu X, Chen Z, Ren F, Feng L, Du Q: Optimizing the Predictive Ability of Machine Learning Methods for Landslide Susceptibility Mapping Using SMOTE for Lishui City in Zhejiang Province, China. Int J Environ Res Public Health 2019, 16(3).
Taft LM, Evans RS, Shyu CR, Egger MJ, Chawla N, Mitchell JA, Thornton SN, Bray B, Varner M: Countering imbalanced datasets to improve adverse drug event predictive models in labor and delivery. J Biomed Inform 2009, 42(2):356-364.
Sui Y, Wei Y, Zhao D: Computer-aided lung nodule recognition by SVM classifier based on combination of random undersampling and SMOTE. Comput Math Methods Med 2015, 2015:368674.
Kuo CY, Yu LC, Chen HC, Chan CL: Comparison of Models for the Prediction of Medical Costs of Spinal Fusion in Taiwan Diagnosis-Related Groups by Machine Learning Algorithms. Healthc Inform Res 2018, 24(1):29-37.

Table 1. Characteristics of older ED patients with influenza in this study

Variable	Total patients (n = 5508)
Age (years)	76.61 ± 7.44
Age subgroup (%)
Young elderly (65-74)	43.06
Moderately elderly (75-84)	40.56
Old elderly (≥85)	16.38
Sex, %
Female	50.67
Male	49.33
Triage vital signs
GCS	14.41 ± 1.84
SBP (mm Hg)	142.88 ± 32.77
Heart rate (beats/min)	93.38 ± 24.24
Respiratory rate (breaths/min)	19.16 ± 3.94
Body temperature (°C)	37.53 ± 6.64
Past histories (%)
Hypertension	56.05
Diabetes	32.37
COPD	12.87
CAD	19.64
CVA	18.77
Malignancy	14.32
CHF	11.27
Dementia	10.62
Bedridden	31.94
Laboratory data
WBC (cells/mm³)	8670.00 ± 4220.00
Bandemia (%)	4.10 ± 5.24
Hemoglobin (mg/dL)	12.42 ± 1.95
Platelet (10³/mm³)	187.36 ± 72.39
Creatinine (mg/dL)	1.29 ± 1.52
hs-CRP (mg/dL)	42.06 ± 50.98
Sodium (mEq/L)	134.68 ± 4.86
Potassium (mmol/L)	3.76 ± 0.52
GOT (U/L)	51.55 ± 172.64
GPT (U/L)	31.79 ± 64.43
Outcomes (%)
Hospitalization	47.33
Pneumonia	37.71
Sepsis or septic shock	5.57
ICU admission	1.07
Death	2.20

Data are presented as mean ± SD or percent. ED, emergency department; GCS, Glasgow coma scale; SBP, systolic blood pressure; COPD, chronic obstructive pulmonary disease; CAD, coronary artery disease; CVA, cerebrovascular accident; CHF, congestive heart failure; WBC, White blood cell count; hs-CRP, high sensitivity C-reactive protein; GOT, glutamic oxaloacetic transaminase; GPT, glutamate pyruvate transaminase; ICU, intensive care unit; SD, standard deviation.

Table 2. Evaluation report using the random forest model with the SMOTE preprocessing algorithm on the outcomes of older ED patients with influenza

Outcome	Number	Negative outcome	Positive outcome	Accuracy	Precision	Sensitivity	Specificity	PPV	NPV	F1	AUC
Hospitalization	5508	3431	2077	0.808	0.808	0.832	0.782	0.802	0.815	0.807	0.807
Pneumonia	5508	5201	307	0.974	0.974	0.968	0.980	0.978	0.971	0.974	0.974
Sepsis or septic shock	5508	5449	59	0.994	0.994	0.995	0.993	0.993	0.995	0.994	0.994
ICU admission	5508	5387	121	0.981	0.981	0.984	0.979	0.978	0.985	0.981	0.982
Death	5508	4426	1082	0.852	0.853	0.822	0.881	0.871	0.835	0.851	0.851

SMOTE, synthetic minority oversampling technique; ED, emergency department; PPV, positive predictive value; NPV, negative predictive value; F1, 2 x (precision x recall/precision + recall); AUC, area under the curve; ICU, intensive care unit.

Table 3. Comparisons of predictive accuracies among logistic regression, random forest, SVC, KNN, and LightGBM in the outcomes of older ED patients with influenza

Outcomes and predictive models	Accuracy	Precision	Sensitivity	Specificity	PPV	NPV	F1	AUC
Hospitalization
Random forest	0.808	0.808	0.832	0.782	0.802	0.815	0.807	0.807
Logistic regression	0.742	0.743	0.776	0.706	0.737	0.706	0.741	0.741
SVM	0.653	0.658	0.578	0.732	0.695	0.621	0.652	0.655
KNN	0.724	0.726	0.784	0.661	0.710	0.743	0.722	0.722
LightGBM	0.789	0.789	0.799	0.779	0.710	0.743	0.789	0.789
Pneumonia
Random forest	0.974	0.974	0.968	0.980	0.978	0.971	0.974	0.974
Logistic regression	0.757	0.759	0.793	0.723	0.727	0.791	0.757	0.758
SVM	0.748	0.748	0.706	0.786	0.753	0.743	0.746	0.746
KNN	0.902	0.912	0.989	0.822	0.837	0.987	0.902	0.905
LightGBM	0.971	0.971	0.964	0.977	0.837	0.987	0.970	0.970
Sepsis or septic shock
Random forest	0.994	0.994	0.995	0.993	0.993	0.995	0.994	0.994
Logistic regression	0.868	0.876	0.940	0.797	0.820	0.931	0.868	0.869
SVM	0.831	0.833	0.787	0.874	0.860	0.807	0.830	0.831
KNN	0.948	0.951	0.990	0.907	0.913	0.989	0.948	0.949
LightGBM	0.994	0.994	0.994	0.993	0.913	0.989	0.994	0.994
ICU admission
Random forest	0.981	0.981	0.984	0.979	0.978	0.985	0.981	0.982
Logistic regression	0.820	0.823	0.867	0.775	0.785	0.860	0.820	0.821
SVM	0.776	0.778	0.729	0.822	0.795	0.761	0.775	0.775
KNN	0.933	0.938	0.994	0.874	0.882	0.994	0.932	0.934
LightGBM	0.984	0.984	0.983	0.984	0.882	0.994	0.984	0.984
Death
Random forest	0.852	0.853	0.822	0.881	0.871	0.835	0.851	0.851
Logistic regression	0.642	0.654	0.774	0.513	0.608	0.700	0.637	0.644
SVM	0.600	0.603	0.668	0.533	0.583	0.622	0.599	0.601
KNN	0.744	0.768	0.893	0.599	0.685	0.851	0.739	0.746
LightGBM	0.822	0.827	0.762	0.881	0.685	0.851	0.821	0.822

SVC, support-vector clustering; KNN, K-nearest neighbors; LightGBM, light gradient boosting machine; ED, emergency department; PPV, positive predictive value; NPV, negative predictive value; F1, 2 x (precision x recall/precision + recall); AUC, area under the curve.

3SupplementaryfiguresAIingeriatricFlu.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Predicting Outcomes in Older ED Patients with Influenza in Real Time Using a Big Data-Driven and Machine Learning Approach to the Hospital Information System

Status:

Version 1

Abstract

Figures

1. Background

2. Methods

3. Results

4. Discussion

5. Conclusions

Abbreviations

Declarations

References

Tables

Supplementary Files

Status:

Version 1