Efficacy of early warning score systems as prediction tools for detecting critically ill patients in an outpatient setting

doi:10.21203/rs.3.rs-3127212/v1

No systematic methods exist for triaging outpatients with serious conditions. Our previous pilot study showed that the National Early Warning Score (NEWS) could predict admissions and unexpected intensive care unit (ICU) transfers in rapid response system-activated outpatients. The Visensia Score Index (VSI) is another artificial intelligence-based Early Warning Score Systems that automatically collates and analyzes data from bedside monitors. This single-center retrospective cohort study aimed to investigate and compare efficacy of NEWS and VSI as a prediction tool among whole first-visit patients of our internal medicine clinic. From June 1, 2018 to November 30, 2018 at a 350-bed teaching community hospital in Japan. Patient age and sex, and physiological measurements, NEWS, VSI as well as disposition and outcomes were collected.

This study included 3301 patients. There were 108 (3.3%), 16 (0.5%), and 5 (0.2%) patients admitted to the general ward, high dependency unit (HDU), and ICU, respectively. The areas under the curve (AUCs) of the NEWS for hospital admission, HDU or ICU admission, and ICU admission were 0.71 (95% CI, 0.66–0.76), 0.88 (95% CI, 0.80–0.97), and 1.00 (95% CI, 0.996–1.0), respectively. The AUCs of the VSI for admission, HDU or ICU admission, and ICU admission were 0.66 (95% CI, 0.60–0.71), 0.82 (95% CI, 0.71–0.93), and 0.97 (95% CI, 0.96–0.98), respectively. The AUC of the NEWS was significantly superior to that of the VSI for hospital (p = 0.03) and ICU admission (p < 0.01). The NEWS could triage patients with serious conditions in an outpatient setting.

Rapid response system

early warning score system

outpatient

national early warning score

medical emergency team

Failure to recognize and respond to patient deterioration may lead to increased in-hospital mortality, morbidity, and length of hospital stay. Considerable interest has grown in developing strategies for the detection of critically ill patients.¹ A rapid response system (RRS) has been developed and adopted internationally for timely identification and intervention in clinically deteriorating patients. The implementation of an RRS is associated with a significant decrease in cardiac arrest and overall mortality in inpatients.^2–4 An RRS comprises both an afferent limb to notice patients at risk and activate the system and an efferent limb to evaluate patients and perform the necessary interventions by the medical emergency team (MET) or rapid response team.⁵

The Early Warning Score System (EWSS) was developed as a more accurate trigger of RRS activation than the conventional threshold. It was designed to allocate scores for a variety of vital signs and mental statuses and to assesses the sum of each score. In 2012, a standardized EWSS for the National Health Service was developed by the Royal College of Physicians, known as the National EWS (NEWS).⁶ The NEWS has been validated in the general ward, ER, and prehospital settings, and it was shown to be useful in predicting unplanned intensive care unit (ICU) transfer, cardiac arrest, and short-term mortality within 48 h.^7–9 However, knowledge of the efficacy of RRS and EWSS among outpatients is very limited.^10–13

Our previous report suggested that NEWS might be an effective tool for triaging in out-patient settings.¹⁴ However, our pilot study had several limitations: the NEWS has 7 components of vital signs, and complete data gathering was difficult. Consequently, the NEWS was often an incomplete database and difficult to analyze. On the other hand, the Visensia Safety Index (VSI) is another type of artificial intelligence (AI)-based EWSS that collates and analyzes data with five vital signs, and it is approved by the Food and Drug Administration (FDA).¹⁵ The benefit of VSI is that the score could be calculated with fewer vital sign components.

Therefore, we conducted this study to investigate the efficacy of the NEWS and VSI as predictive tools for detecting critically ill patients among all first-visit patients in our internal medicine outpatient clinic.

Study Design, Population, and Setting

This cohort study was conducted at a single center from June 1, 2018, to November 30, 2018, at a 350-bed teaching community hospital with a 14-bed ICU and a 12-bed high dependency unit (HDU). The study population consisted of new patients of an internal medicine outpatient clinic, excluding a cardiology clinic. Follow-up visit patients were excluded. Clinical nurses conducted a medical interview and evaluated their physiological measurements, including respiratory rate (RR), peripheral oxygen saturation (SpO₂), body temperature(BT), systolic blood pressure (BP), heart rate (HR), and level of consciousness for every patient. During the study period, the NEWS was prospectively recorded based on these measurements. Patients deficient in any of the vital signs were excluded. RRS-activated outpatients were evaluated by an emergency physician-led medical emergency team (MET) and transferred to the emergency room (ER). Conventionally, our MET-calling criteria is single-parameter criteria (conventional criteria) based on the University of Pittsburgh Medical Center ‘Condition C’ calling criteria (Supplemental Table 1). From June 1, 2020, to August 31, 2020, the RRS was activated if patients met conventional criteria; from September 1, 2020, to November 30, 2020, it was activated if patients met conventional criteria or NEWS criteria with score of ≥ 5.

Data Collection

Data of patients who visited the internal medicine outpatient clinic for the first time were extracted by reviewing the medical interview sheets of outpatients, medical records, and MET records. Age, sex, and physiological measurements including RR, SpO₂, BT, systolic BP, HR, and level of consciousness were graded using the AVPU scale (Alert; Verbal, voice response present; Pain, pain response present; Unresponsive). NEWS, disposition status, and outcomes were collected. The VSI (OBS Medical, Cirencester, UK) of the study population was retrospectively calculated.

Outcomes

Primary outcomes were the prediction accuracies of NEWS and VSI for hospital admission, unplanned HDU or ICU transfer, and unplanned ICU transfer.

Secondary outcomes were cardiac arrest and 24-h, 30-day, and 90-day mortality rates. The NEWS was calculated as reported by the Royal College of Physicians. Aggregated NEWS scores were categorized into three groups: low-risk (NEWS 0–4), intermediate-risk (NEWS 5–6), and high-risk (NEWS ≥ 7), according to NEWS threshold criteria.¹⁰

Statistical analysis

R (The R Foundation for Statistical Computing, Vienna, Austria, version 3.3.3) and EZR (Saitama Medical Center, Jichi Medical University, Saitama, Japan, version 1.3.5) were used for data analysis. EZR is a graphical user interface for R. More precisely, it is a modified version of the R commander designed to add statistical functions frequently used in biostatistics.¹⁵The Mann–Whitney U test was used for statistical comparison of continuous variables between two groups, and the Kruskal–Wallis test was used for statistical comparison of continuous variables between three or more groups. Fisher’s exact test was used to analyze data involving categorical variables. DeLong’s test was used for comparison of the area under the two correlated receiver operating characteristic (ROC) curves. Sensitivity, specificity, positive and negative predictive values (PPV and NPV, respectively) for admission, HDU or ICU admission, and ICU admission were also analyzed. A p-value of < 0.05 was determined to be statistically significant.

Study Population

During the study period, 4542 outpatients visited the internal medicine outpatient clinic for the first time. Patients with missing age, sex, or physiological parameters were excluded. Of these patients, 1.61% (73/4542) were missing data for age, 1.19% (54/4542) for sex, 0.99% (45/4542) for BT, 0.90% (41/4542) for HR, 0.97% (44/4542) for BP, 26.06% (1184/4542) for RR, and 24.06% (1093/4542) for SpO₂. As a result, 27.32% (1241/4542) of the patients were excluded.

Patient characteristics are shown in Table 1. The median patient age was 53 years (interquartile range [IQR], 37–71 years). Of the patients, 1677 (50.8%) were male. The median NEWS score was 1.0 (IQR, 0–1.0), and the median VSI score was 0.9 (IQR, 0.6–1.0). The number of patients in the low-, intermediate-, and high-risk groups based on NEWS scores were 3253 (98.5%), 29 (0.9%), and 19 (0.6%), respectively. The number of patients in high-risk group based on VSI scores of > 3 was 149 (4.5%). There were 108 (3.3%), 16 (0.5%), and 5 (0.2%) patients admitted to a general ward, HDU, or ICU, respectively. The 24-h, 30-day, and 90-day mortality rates were 0% (0/3301), 0.06% (2/3301), and 0.12% (4/3301), respectively. Further, 34 patients (1.0%) had activated RRSs. All patients who had activated RRSs were transferred to the ER and evaluated by an emergency physician-led MET, according to our hospital RRS protocol.

EWS and outcomes

The NEWS, based on admission status (admission vs. discharge) and disposition (discharge vs. ward, HDU, and ICU), is shown in Fig. 1. The median NEWS score of patients who required admission was significantly higher than that of patients who were discharged (0 [IQR, 0–1] vs. 1 [IQR, 1–4], p < 0.01). The median NEWS scores of patients who were admitted to the ward, HDU, and ICU were 1 (IQR, 1–3), 2 (IQR, 1.75–6.25), and 8 (IQR, 7–10), respectively (p < 0.01). The VSI, based on admission status (admission vs. discharge) and disposition (discharge vs. ward, HDU, and ICU), is shown in Fig. 2.

The median VSI score of patients who required admission was significantly higher than that of patients who were discharged (1.5 [IQR, 0.7–2.8] vs. 0.9 [IQR, 0.6–1.4], p < 0.01). The median VSI scores of patients who were admitted to the ward, HDU, and ICU were 1.2 (IQR, 0.65–2.4), 1.9 (IQR, 1.18–4.7), and 3.6 (IQR, 3.3–3.7), respectively (p < 0.01). The NEWS and VSI categories and dispositions are listed in Table 2.

Intermediate- and high-risk NEWS scores were significantly associated with hospital admission, HDU or ICU admission, and ICU admission (all p < 0.01). A VSI score of ≥ 3 was also significantly associated with admission, HDU or ICU admission, and ICU admission (all p < 0.01). Intermediate- or high-risk NEWS scores were seen in 22% (28/129), 48% (10/21), and 100% (5/5) of patients requiring general ward, HDU, and ICU admissions, respectively. High-risk VSI scores were also detected in 22% (28/128), 48% (10/21), and 80% (4/5) of patients who required general ward, HDU, and ICU admissions, respectively (Table 2).

Figure 3 shows the ROC curves of the NEWS and VSI for clinical outcomes. The areas under the curve (AUCs) of NEWS for hospital admission, HDU or ICU admission, and ICU admission were 0.71 (95% CI, 0.66–0.76), 0.88 (95% CI, 0.80–0.97), and 0.998 (95% CI, 0.996–1.0), respectively. The AUCs of VSI for hospital admission, HDU or ICU admission, and ICU admission were 0.66 (95% CI, 0.60–0.71), 0.82 (95% CI, 0.71–0.93), and 0.97 (95% CI, 0.96–0.98), respectively. The AUC of NEWS was significantly superior to that of VSI for hospital admission (p = 0.03) and for ICU admission (p < 0.01). The AUCs of NEWS and VSI for HDU or ICU admission were not significantly different (p = 0.07). The best VSI cut-off values, according to the ROC curves for admission, HDU or ICU admission, and ICU admission, were 1.7, 1.7, and 2.8, respectively. Table 3 shows the validity of the NEWS and VSI for clinical outcomes using the variable cut-off points.

To the best of our knowledge, this study is the first to evaluate the efficacy of EWSSs as prediction tools for detecting critically ill patients among first-visit patients in an internal medicine outpatient clinic. Contrary to our hypothesis that the NEWS and VSI are equally effective in predicting hospitalization in outpatient settings, the NEWS is more effective than the VSI in predicting hospital admission.

Our data showed that both the NEWS and VSI could predict disposition with high accuracy. Both scores showed higher prediction accuracy for unanticipated ICU admission than that for ICU/HDU admission or hospital admission. The AUC of the NEWS was significantly superior to that of the VSI for predicting hospital admission and ICU admission. The NEWS also had both a high PPV and NPV for unanticipated ICU admission in discriminating critically ill patients accurately among all first-visit outpatients.

In a previous study in an inpatient setting, the NEWS had a greater ability to discriminate patients at risk of the combined outcome of cardiac arrest, unanticipated ICU admission, or death within 24 h (area under the ROC curve [AUC], 0.873; 95% CI, 0.866–0.879).¹⁷ The NEWS has also been validated in ER prehospital settings and has been shown to be a useful tool for predicting unplanned ICU transfer, cardiac arrest, and short-term mortality within 48 h.^7–9 Our results suggest that even in an outpatient setting, the NEWS could predict clinical outcomes as accurately as inpatient, ER, and prehospital settings with consistency.

To organize an effective RRS, the balance between the trigger rate and the risk of missing critically ill patients is important.¹⁸ In implementing a track-and-triage system in a clinical setting, PPVs and NPVs should be considered for workload efficacy. A low PPV can lead to excessive staff workload. A low NPV indicates a high risk of missing serious outcomes. In our study, 0.6% of outpatients met the high-risk NEWS threshold of 7. The PPVs for hospital admission, HDU orICU admission, and ICU admission were 85%, 50%, and 26%, respectively. Meanwhile, 1.5% of outpatients met the intermediate-risk NEWS threshold of 5. The PPVs for hospital admission, HDU or ICU admission, and ICU admission were 59%, 21%, and 10%, respectively. The Royal College of Physicians has recommended that medium-risk patients (NEWS, 5–6) should be closely monitored and assessed urgently by a clinician or team with core competencies in the care of acutely ill patients. It has also been recommended that high-risk patients (NEWS, ≥ 7) should be assessed by a team with critical care competencies, including practitioners with advanced airway management skills.¹⁰ Therefore, intermediate- or high-risk NEWS scores might be reasonable MET-calling criteria in outpatient settings, and the patient should be considered for transfer to the ER.

In contrast, the AUC of the VSI for predicting hospital admission and ICU admission was inferior to that of NEWS (p = 0.03, p < 0.01, respectively; Fig. 3). Although the VSI showed a high AUC for ICU admission, the PPVs for hospital admission, HDU or ICU admission, and ICU admission remained at 19%, 7%, and 3%, respectively, for the high-risk VSI threshold of 3.0. From our results, the VSI does not meet the MET-calling criteria in an outpatient setting because of the very low PPV, which may lead to high false-positive cases. The VSI uses five physiological parameters: BT, BP, HR, RR, and SpO₂. In contrast, NEWS uses seven parameters: mental status, oxygen use, and the VSI parameters, and this score requires manual evaluation and calculation of mental status and oxygen use. However, in our study population (n = 3301), 9 patients had altered mental status, and 2 had supplemental oxygen use. The prevalence of these two parameters was quite low, and the influence of their validity on clinical outcomes was very low. In addition, the VSI was developed using a vital sign dataset of inpatients in the monitoring unit; thus, to apply the VSI in an outpatient setting, an outpatient database should be collected, and modification of the VSI is required.

Limitations

Our study has some limitations. First, this was a single-facility retrospective study. Second, 1241 patients were excluded because of insufficient vital sign data, mainly RR or SpO₂. Finally, the study population exclusively comprised first-visit outpatients at an internal medicine clinic without a cardiology clinic; therefore, our results cannot be extrapolated to a general outpatient clinic population.

Our results validated the NEWS and VSI in an outpatient setting. Both scores revealed high efficacy in predicting HDU or ICU admission, especially ICU admission. The NEWS was significantly superior to the VSI in predicting hospital and ICU admissions (p = 0.03, p < 0.01, respectively).

ACKNOWLEDGEMENTS

Contributors: We wish to sincerely thank Ms. Yuiko Hoshina, Ms. Satomi Noguchi, Ms. Tomoko Muto, and Yuko Kudo for their support with the data collection.

Funders: none

Prior presentations: The 50^th annual meeting of Japanese society of intensive care medicine (e-poster presentation) on 2^nd March,2023.

Conflict of interest statement: Nihon Kohden corporation provided a VSI calculation software for the authors.

Ethical approval: This study was approved by the Research Ethics Board of Tokyo bay Urayasu Ichikawa medical center.

Barwise A, Thongprayoon C, Gajic O, Jensen J, Herasevich V, Pickering BW. Delayed rapid response team activation is associated with increased hospital mortality, morbidity, and length of stay in a tertiary care institution. Crit Care Med. 2016;44(1):54-63. doi: 10.1097/CCM.0000000000001346.
DeVita MA, Braithwaite RS, Mahidhara R, et al. Use of medical emergency team responses to reduce hospital cardiopulmonary arrests. Qual Saf Health Care. 2004;13(4):251-254. doi: 10.1136/qhc.13.4.251.
Chan PS, Jain R, Nallmothu BK, Berg RA, Sasson C. Rapid response teams: a systematic review and meta-analysis. Arch Intern Med. 2010;170(1):18-26. doi: 10.1001/archinternmed.2009.424.
Winters BD, Weaver SJ, Pfoh ER, Yang T, Pham JC, Dy SM. Rapid-response systems as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 Pt 2):417-425. doi: 10.7326/0003-4819-158-5-201303051-00009.
DeVita MA, Hillman K, Bellomo R, et al. Textbook of Rapid Response Systems. Cham, Switzerland: Springer; 2017.
Royal College of Physicians. National Early Warning Score (NEWS) 2. https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news-2. Accessed [July 6, 2022].
Abbott TEF, Vaid N, Ip D, et al. A single-centre observational cohort study of admission National Early Warning Score (NEWS). Resuscitation. 2015;92:89-93. doi: 10.1016/j.resuscitation.2015.04.020.
Alam N, Vegting IL, Houben E, et al. Exploring the performance of the National Early Warning Score (NEWS) in a European emergency department. Resuscitation. 2015;90:111-115. doi: 10.1016/j.resuscitation.2015.02.011.
Silcock DJ, Corfield AR, Gowens PA, Rooney KD. Validation of the National Early Warning Score in the prehospital setting. Resuscitation. 2015;89:31-35. doi: 10.1016/j.resuscitation.2014.12.029.
King E, Horvath R, Shulkin DJ. Establishing a rapid response team (RRT) in an academic hospital: one year’s experience. J Hosp Med. 2006;1(5):296-305. doi: 10.1002/jhm.114.
Dechert TA, Sarani B, McMaster M, et al. Medical emergency team response for the non-hospitalized patient. Resuscitation. 2013;84(3):276-279. doi: 10.1016/j.resuscitation.2012.06.022.
Alansari MA, Althenayan EA, Hijazi MH, Maghrabi KA. The rapid response team in outpatient settings identifies patients who need immediate intensive care unit admission: a call for policy maker. Saudi J Anaesth. 2015;9(4):428-432. doi: 10.4103/1658-354X.159469.
Lakshminarayana PH, Darby JM, Simmons RL. Addressing patient safety in rapid response activations for nonhospitalized persons. J Patient Saf. 2017;13(1):14-19. doi: 10.1097/PTS.0000000000000098.
Ehara J, Hiraoka E, Hsu HC, Yamada T, Homma Y, Fujitani S. The effectiveness of a national early warning score as a triage tool for activating a rapid response system in an outpatient setting: a retrospective cohort study. Medicine. 2019;98(52):e18475. doi: 10.1097/MD.0000000000018475.
National Institute for Health and Care Excellence. Visensia for early detection of deteriorating vital signs in adults in hospital. https://www.nice.org.uk/guidance/mib36. Accessed [July 6, 2022].
Kanda Y. Investigation of the freely available easy-to-use software “EZR” for medical statistics. Bone Marrow Transplant. 2013;48(3):452-458. doi: 10.1038/bmt.2012.244.
Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84(4):465-470. doi: 10.1016/j.resuscitation.2012.12.016.
Smith GB, Prytherch DR, Jarvis S, et al. A comparison of the ability of the physiologic components of medical emergency team criteria and the U.K. National Early Warning Score to discriminate patients at risk of a range of adverse clinical outcomes. Crit Care Med. 2016;44(12):2171-2181. doi: 10.1097/CCM.0000000000002000.

Table 1. Patient characteristics (n = 3301)

Age in years, median (IQR)	53 [37-71]
Male gender, n (%)	1677 (50.8)
NEWS, median (IQR)	1.0[0- 1.0]
NEWS category, n (%)
Low risk 0-4	3253 (98.5)
Moderate risk 5-6	29 (0.9)
High risk 7≥	19 (0.6)
VSI, median (IQR)	0.9 [0.6-1.0]
VSI category, n (%)
Low risk 0-3.0	3141 (95.5)
High risk 3.0≥	149(4.5)
NA	11
RRS activation, n (%)	34 (1.0)
Dispositions n (%)
Admission	129 (3.9)
General ward	108 (3.3)
HDU	16 (0.5)
ICU	5 (0.2)
24 hour mortality, n (%)	0
30 day mortality, n (%)	2(0.06)
90 day mortality, n (%)	4(0.12)

IQR, interquartile range; NEWS, National Early Warning Score; VSI, Visensia Score Index; RRS, Rapid Response System; ICU, intensive care unit; HDU, high dependency unit

Table 2. NEWS and VSI category and disposition
NEWS category
	Low risk^＃１　 n = 3253	Intermediate risk^＃２ n = 29	High risk^＃3 n = 19	p value
Hospital admission, n(%) n=129	101(3.1)	12(41.4)	16(84.2)	<0.01
HDU or ICU admission, n(%) n=21	11(0.3)	1(3.4)	9(47.4)	<0.01
ICU admission, n(%) n=5	0	0	5(26.3)	<0.01
VSI category
	Low risk^＃4 n = 3141		High risk^＃5 n = 149	p value
Hospital admission n(%) n=129	100(3.2)		28(18.8)	<0.01
HDU or ICU admission n(%) n=21	11(0.4)		10(6.7)	<0.01
ICU admission n(%) n=5	1(0.03)		4(2.7)	<0.01

NEWS: National Early Warning Score. VSI, Visensia score index;
HDU, high dependency unit; ICU, intensive care unit.

#1: NEWS ≤４points, #2: NEWS 5-6 points, #3: NEWS 7 ≥ points, #4: VSI < 3 points, #5: VSI 3 ≥ points

Table 3. Validity of the NEWS and VSI in the outpatient population

	Hospital admission				HDU or ICU admission				ICU admission
	Sen	Spe	PPV	NPV	Sen	Spe	PPV	NPV	Sen	Spe	PPV	NPV
NEWS Cut off =2	50	84	11	98	81	83	3	100	100	83	1	100
NEWS Cut off =5	22	99	59	97	48	99	21	100	100	99	10	100
NEWS Cut off =7	13	99	85	97	45	100	50	100	100	100	26	100
	Sen	Spe	PPV	NPV	Sen	Spe	PPV	NPV	Sen	Spe	PPV	NPV
VSI Cut off =2.0	39	87	11	97	57	87	3	100	100	87	1	100
VSI Cut off =2.5	28	93	14	97	52	93	4	100	100	92	2	100
VSI Cut off =3.0	22	96	19	97	48	96	7	100	80	96	3	100

NEWS: National Early Warning Score. VSI, Visensia score index;
HDU, high dependency unit; ICU, intensive care unit.

Sen, sensitivity; Spe, specificity; PPV, positive predictive value; NPV, negative predictive value.

Supplementaltable1.docx

Efficacy of early warning score systems as prediction tools for detecting critically ill patients in an outpatient setting

Status:

Version 1

Abstract

Figures

INTRODUCTION

METHODS

Study Design, Population, and Setting

Data Collection

Outcomes

Statistical analysis

RESULTS

Study Population

EWS and outcomes

DISCUSSION

Limitations

Conclusion

Declarations

References

Tables

Supplementary Files

Status:

Version 1