Derivation and Validation of a Clinical Deterioration Early Warning System (Cdews) Score in Hospital Inpatients

doi:10.21203/rs.3.rs-5282831/v1

Download PDF

Research Article

Derivation and Validation of a Clinical Deterioration Early Warning System (Cdews) Score in Hospital Inpatients

https://doi.org/10.21203/rs.3.rs-5282831/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Electronic health records allow access to data on potential predictors of clinical deterioration in hospital inpatients beyond vital signs, e.g., lab test results and comorbidities. Using such data, we aimed to develop and validate a model to predict clinical deterioration.

A retrospective cohort study was conducted with information on vital signs, comorbidities, and lab test results for consecutive over-17-year-olds hospitalized in Galdakao-Usansolo University Hospital. Deterioration was defined as death or unplanned admission to an intermediate/intensive care unit. A multivariate generalized linear mixed model was constructed using a derivation dataset, adjusting for the random effect of the patient. We calculated risk scores and categories and the area under the receiver operating characteristic curve (AUC) for this cohort, and also for an external validation cohort, to test their validity with external data. The model was calibrated in the derivation dataset considering 10 decile-based groups. The score was calibrated in both datasets.

Overall, 6,372 hospitalizations of 9,084 patients (70.5%) were included in the derivation dataset and 7,812 of 10,531 patients (74.2%) in the validation dataset. In these sets, 5% and 6% of patients respectively reached the composite endpoint (pvalue = 0.02). Older patients, and those with polypharmacy and/or neoplasms were more likely to deteriorate. Deterioration was associated with higher blood pressure and C-reactive protein and lower oxygen saturation, glycemia, and potassium levels. The final model´s AUC was 0.83 (0.81-0.85).

We provide a reliable easy-to-implement score to predict death or unexpected ICU admission; further research is needed to assess potential clinical benefits.

ClinicalTrials.gov ID: NCT06499376Registered at 2024-07-12 (retrospectively registered)

Clinical deterioration

early warning scoring systems

Patients admitted to hospitals often face unforeseen complications such as sudden cardiac arrest or intensive care unit (ICU) admission, sometimes leading to death due to delayed detection of critical vital sign irregularities and insufficiently prompt responses¹. The primary objective of early warning systems (EWS) in a hospital ward is to proactively identify patients at risk of ICU admission and/or developing the aforementioned type of complication due to sepsis, respiratory failure, or other unexpected events. Their goal is to trigger prompt intervention to prevent adverse outcomes. Both EWS and rapid response systems (RRS) are designed to facilitate the early identification of physiological instability and swift implementation of response protocols to address deteriorating conditions among patients in acute hospital settings².

The National Early Warning Score (NEWS) and Modified Early Warning Score (MEWS) are scoring systems used to identify and respond to deteriorating patients in hospital settings³. They assign scores to physiological parameters and sum these scores; the higher the total score, the greater the risk of clinical deterioration. Nowadays, many hospitals and healthcare facilities use these scores, especially in emergency departments (EDs). A recent systematic review concludes however that there is no evidence that EWS reduce mortality²; it was not possible to perform a meta-analysis due to differences in the interventions and settings across the studies included, but some showed fewer unplanned ICU admissions and adverse events. These findings, added to the benefits perceived by clinical practitioners, suggest the need to update these systems, to ensure that they are suitable both for all types of patients and for current healthcare data storage methods.

Electronic health records (EHRs), available nowadays in most healthcare settings in high-income countries, allow us to easily consult lab test data as soon as they are available. Research on the use of lab results to enhance the predictive power of the NEWS⁶ led us to believe that a comprehensive model that included data from various sources, including comorbidities, and lab test results, as well as vital signs, should provide even more accurate data related to the prognosis of in-hospital patients.

Winslow et al studied the impact of a machine learning-based EWS on hospital mortality⁷. Demographic, mental status, vital signs, and laboratory inputs on over 60,000 patients were used to create a score. Their findings confirm that using a combination of data is a potentially good way to predict outcomes more accurately. While these systems might be fundamental in the future, the reality is that many healthcare systems are not ready for the costs or complexity of machine learning-based systems, implying a role for simpler tools applicable in diverse settings.

In this context, we believe there is a need for properly validated, simple, and fast prediction scores, that can be used in all types of patients and supported by hospital information systems without notable additional costs.

Data collection:

This is a retrospective cohort study of consecutive patients hospitalized in Galdakao-Usansolo University Hospital. This hospital is a public secondary center, with a catchment population of 300,000 people in the province of Bizkaia, the Basque Country (Spain).

The outcome variable was defined as death or unplanned admission to an intensive or intermediate care unit (ICU). To distinguish planned from unplanned admissions, we excluded patients who underwent surgery straight after a planned admission to the hospital and were immediately admitted for postsurgical care. Further, as some ancillary tests need sedation provided in the ICU, admissions for less than 12 hours that did not end in death were also excluded. All other admissions were considered unplanned.

To design the Clinical Deterioration Early Warning Score (CDEWS), we included every patient over 17 years old admitted to a ward, except those whose length of stay was shorter than 24 hours. Readmissions within 48 hours were considered to be the same episode.

Data were obtained from the Osakidetza-Basque Public Health Service information system. After anonymization and removal of protected health information, the data were released in a text-delimited format for research purposes. We retrieved patient-level data for the initial analyses in our study.

Data were collected on demographic variables (age, sex), as well as comorbidities and baseline treatments. Comorbidities were assessed based on International Classification of Diseases, Tenth Revision (ICD-10) codes that were active on the patients’ EHR on arrival to the ED. Baseline treatments were defined as the medications that patients were taking at the time of admission, and hospital treatments as those administered during hospitalization until 24 hours before deterioration. Both were recorded based on the Anatomical, Therapeutic and Chemical system/Defined Daily Dose (ATC/DDD) index in the EHR. Patients were considered to have no comorbidities or treatments if none were documented in the EHR. They were classified as polymedicated if they used more than five medications per day⁸. Supplementary material Table 1 provides a complete list of comorbidities and ICD-10 codes.

Regarding history of hospitalization, as well as treatment, we collected information related to vital signs: temperature, blood pressure, respiratory and heart rate, oxygen saturation as measured by pulse oximetry, and fraction of inspired oxygen. We recorded the worst values from each day during the hospital stay: for the entire stay in the case of patients who did not deteriorate and for the 72 to 24 hours before deterioration in the case of those who did.

Similarly, for patients who did and did not deteriorate, we also gathered laboratory test results from throughout their hospitalization up to 72 hours before deterioration. We extracted key values from a range of common routine hematology, blood biochemistry, and blood coagulation tests (i.e., hemoglobin concentration, white cell count, and levels of urea, albumin, creatinine, sodium, and potassium) and arterial blood gas analysis.

Figure 1 shows the timeline of the study data collection. All the data we used were structured. We defined deterioration as death or length of stay more than 12 hours in the ICU.

Datasets:

1. Development dataset (derivation and testing): We downloaded patient-level data for August 1 to December 31, 2019. Data were pre-processed to address quality issues, such as allowing the elimination of repeated entries for some episodes and the exclusion of features with more than 25% of missing data from the list of potential predictors.

2. Validation dataset (external validation): Furthermore, a prospective validation dataset of patients, independent from the other datasets (hereon referred to as the validation dataset), was created with patients admitted from July 1 to December 31, 2021. This dataset was used for the external validation and was pre-processed in the same way as the development dataset. Further, the demographic and clinical data recorded for these patients were comparable with those of the patients in the development cohort.

Ethical considerations

The Ethics Committee of the Basque Country (PI2019030) approved the study, including a waiver of informed consent. Research was carried out in accordance with the relevant guidelines and regulations.

Statistical Analysis

The units of analysis were episodes. Descriptive statistics were calculated including frequency tables for categorical variables and means and standard deviations (SDs) for continuous variables. A category was created for missing values in laboratory data. Patient characteristics were compared between the two subsamples (derivation vs. external validation datasets) using chi-square or Fisher’s exact tests for categorical variables, and Student’s t-test or nonparametric Wilcoxon tests for continuous variables as appropriate.

Univariate analysis was performed in the derivation sample to identify variables associated with deterioration, using chi-square or Fisher’s exact tests and Student’s t or Wilcoxon tests for categorical and continuous variables, respectively. All risk factors for deterioration that obtained p-values < 0.20 were entered as possible explanatory variables in a multivariate model. The final multivariate generalized linear mixed model was adjusted for the random effect of the patient, for the variables that were statistically significant at the 5% level. The predictive power of the model was evaluated using the area under the receiver operating characteristic curve (AUC) comparing predicted and observed deterioration.

To develop our predictive risk score (the CDEWS), each risk factor was assigned a weight based on each β coefficient of the multilevel model. Considering an optimal classification and taking into account the distribution of the outcome variable (deterioration), four risk categories were created: mild, moderate, severe, and very severe. To determine cut-off points for each category, optimal cut-offs on the continuous risk scores were determined with the catpredi function from the R package CatPredi using the genetic algorithm [21]. The score and the risk categories were used separately as independent variables, to explore their ability to predict deterioration, using the AUC, in both the derivation and validation datasets.

To test the validity of the CDEWS and risk categories with external data, they were calculated and the AUC was obtained for the external validation dataset as for the derivation cohort. The calibration of the model in the derivation dataset was carried out considering 10 groups based on the deciles. Similarly, the score was calibrated in the derivation dataset and the external validation dataset.

All statistical analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Data collection:

For the analysis, 6,372 hospitalizations of 9,084 patients (70.5%) were included in the development dataset, while 7,812 of 10,531 patients (74.2%) were included in the validation dataset. In these sets, 5% and 6% of patients respectively reached the composite endpoint (p value = 0.02) (Figure 2).

The group of patients who deteriorated did not differ significantly from those who did not deteriorate in sex composition, but they were older with more comorbidities (except for respiratory comorbidities). They were more likely to have been admitted previously and to have long-term prescriptions, including of diuretics. Regarding vital signs, there were significant between-group differences in oxygen saturation, heart rate, and blood pressure: patients who deteriorated had lower oxygen saturation levels, higher heart rates, and lower blood pressure. Further, they had higher levels of glucose, creatinine, bilirubin, leucocytes, C-reactive protein (CRP), and procalcitonin than those who did not deteriorate (Table 1).

There were clinically important differences between the derivation and external validation cohorts in long-term prescriptions; specifically, in the external validation cohort, a higher percentage of patients used code A drugs, antimicrobial agents, immunomodulating agents, and nervous and respiratory system drugs. Further, heart rate and blood pressure were lower in the validation cohort. Supplementary Table 2 lists all the variables considered in the univariate analysis.

Table 2 shows the CDEWS model: patients who were older, polymedicated, and with neoplasms were more likely to deteriorate. Clinical deterioration was associated with higher CRP, lower blood pressure and lower oxygen saturation, glycemia, and potassium levels. The final model´s AUC was 0.83 (0.81-0.85).

Seeking to develop a measure of the probability of deterioration in hospitalized patients, a prediction score was developed. The prediction score showed good discriminatory power, as measured by the AUC (0.81). Applying this prediction score to the external validation cohort yielded an AUC of 0.76. The calibration of the models and the score was evaluated using calibration plots (Figure 3).

Figure 4 shows the risk groups created and their corresponding risk of deterioration and Table 3 shows the prognostic performance statistics for risk groups in the external validation cohort.

Main findings and strengths

In the current study, we have developed and validated a score to predict deterioration based on a robust predictive model, which includes vital signs, but also other predictors related to age, comorbidities, number of medications, and lab test results. This score has shown excellent prognostic performance.

The strength of this research is the wide range of data used, and this has allowed us to obtain reliable results in a broad range of patients. Furthermore, the predictors used in the CDEWS are easily obtained from EHRs, allowing for real-time results. The diversity of the information used (demographic, clinical, etc.) suggests its applicability to all demographic groups.

The timeline we considered in this study is notably different from those focused on in previously proposed tools. Most of these other proposals, including the well-known MEWS⁹, aim to predict deterioration within 24 hours, seeking to prompt immediate intensification of care and avoid further deterioration. We retrieved earlier data, from 72 to 24 hours before the event, to set the threshold for triggering an alarm without altering usual care, allowing healthcare providers’ actions to be modified with enough time to obtain more information regarding vital signs or laboratory test results, i.e., implementing early monitoring. Given this, we suggest that CDEWS could be used together with other scores, that would be applied once the warming system’s alarm has been triggered.

Further, the variables found to be relevant in our study differ from those already used in other scales. In contrast to NEWS/MEWS^9,10, temperature and heart rate were not significant in our model. This could be explained by our patients being relatively old, and older age being associated with physiological changes, such as baseline heart rate variability¹¹ and a weaker response to infection¹², suggesting that CDEWS could be an appropriate score regardless of the age group. Considering that scores’ performance might be influenced by age, Shamout et al. looked for an age-specific model¹, and found that, rather than adding age to prognosis models, different scales performed better depending on the age range. We did not find differences in the risk of clinical deterioration in the age ranges between 45 and 85 years, perhaps due to lower statistical power in some age ranges, which could be a limitation of this work as well as the lack of data on frailty. We believe that further studies should include an age-frailty interaction term as a predictor and explore stratification of models according to age groups.

Among vital signs, only mean arterial pressure and oxygen saturation were retained in the model. As with the NEWS score, it could be argued that the use of oxygen saturation in the model makes it inappropriate for patients with chronic obstructive pulmonary disease ¹³. In our study, we opted to group comorbidities by systems, to make it simple to use. In that regard, 22% of the population was previously diagnosed with a chronic respiratory disease. Moreover, as many as 30% of the entire population were on medications related to the respiratory system, which makes us think that chronic respiratory disease could be underdiagnosed. Neither of those items had a significant influence in the prediction model, however, suggesting that the final model did work well for patients with chronic respiratory conditions. In any case, these questions need to be addressed in future prospective research.

Notably, in spite of considering all comorbidities for which data were available in our modeling, only neoplasms diagnosed in the year before admission was significant in the final prediction model, regardless of the stage of the disease. Cancer is one of the main causes of in-hospital death in Spain and also acts as a predictor of death or fatal conditions such as heart failure^14,15.

Polypharmacy is strongly associated with frailty and several commonly prescribed drugs are strongly associated with increased mortality⁸. These factors explain its inclusion in the score, although none of the medication classes had a significant impact by themselves.

Redfern et al. used blood test results to enhance the predictive power of the NEWS ⁶. We chose to include not only the parameters already studied, such as potassium level, but also those related to acute disease and identified a positive impact of glucose and CRP.

Limitations and future research

As expected in retrospective research, some information was missing. Specifically, there were missing data on some clinical parameters, such as respiratory rate and mental status, due to incomplete recording under routine clinical practice conditions on hospital wards. To avoid this issue influencing the other results as well as to develop a tool suitable for use in routine clinical practice, we did not include these parameters in the model. Further, a substantial proportion of patients did not have laboratory samples collected daily; we opted to interpret the clinical decision as a sign of stability, and “no laboratory tests” was considered a value in itself. This assumption could have introduced bias, although it strengthened the statistical analysis.

The external validation of the model was performed in a different set of patients from the same hospital. On the other hand, the samples were obtained 2 years apart (during which time there may have been changes in clinical practice routines) and separated by the pandemic period, with the impact that this had both on patients’ chronic disease status¹⁶ and healthcare professional turnover. These factors suggest that the scenarios from which the datasets were obtained differed and in turn that it may be feasible to extrapolate our findings to other settings.

In a recent review, Henry et al pointed out the need for clinical deterioration scores validated in prospective studies, as well as the risk of increasing workload due to future use of prediction tools ¹⁷. These issues remain to be addressed for the score we propose, the CDEWS, underlining the need for new prospective studies in both the original hospital and other healthcare centers. From our point of view, the CDEWS should be integrated into care pathways as part of a multidisciplinary approach, to favor the success of the “ICU without walls” concept¹⁸. Furthermore, it could be tested as a tool to predict ICU readmission, as -to our knowledge- there are no effective tools for this purpose to date¹⁹.

Finally, although it can be envisaged that the simplicity of the CDEWS would make it easy to install it on any EHR device, further research is warranted to explore the fastest and most practical way to apply this tool in clinical settings. It is generally recognized that clinicians often use their own devices to obtain valuable data regarding the prognosis of their patients²⁰. We believe that this kind of information should be automatically available for healthcare workers through the electronic devices used in routine clinical practice, and hence, one of the issues to be addressed is how best to implement this type of warning system for use in real-world settings.

In conclusion, we provide a reliable easy-to-implement score to predict death or unexpected ICU admission. Further research is needed to assess potential clinical benefits from its use.

Funding

The research project (reference number 2018111094) received funding from the Department of Health of the Basque Government. The editing expenses have been covered by the Research Committee of Galdakao-Usansolo University Hospital.

The authors declare they have no conflicts of interest.

Acknowledgments: We are grateful to the Research Commission of Galdakao-Usansolo Hospital for the financial support provided for the edition of this manuscript, and to the former PI of this project, Dr. Pedro María Olaechea, for launching this work and his ongoing encouragement.

CDEWS working group: Sira Iturrizaga Correcher, Sergio Castaño Ávila, José Ferri Rosalen, Inmaculada Arostegui Madariaga.

Shamout, F. et al. Early warning score adjusted for age to predict the composite outcome of mortality, cardiac arrest or unplanned intensive care unit admission using observational vital-sign data: a multicentre development and validation. BMJ Open 9, e033301 (2019).
McGaughey, J., Fergusson, D. A., Bogaert, P. V. & Rose, L. Early warning systems and rapid response systems for the prevention of patient deterioration on acute adult hospital wards. Cochrane Database Syst. Rev. 2021, CD005529 (2021).
Smith, G. B. et al. The National Early Warning Score 2 (NEWS2). Clin. Med. 19, 260–260 (2019).
Rodríguez-Artalejo, F., Guallar-Castillón, P. & Otero, C. M. Health-Related Quality of Life as a Predictor of Hospital Readmission and Death Among Patients With Heart Failure. ACC Curr. J. Rev. 14, 28 (2005).
Puig-Campmany, M. & Ris-Romeu, J. Frail older patients in the emergency department: main challenges. Emerg.: Rev. Soc. Espanola Med. Emerg. 34, 415–417 (2022).
Redfern, O. C. et al. Predicting in-hospital mortality and unanticipated admissions to the intensive care unit using routinely collected blood tests and vital signs: Development and validation of a multivariable model. Resuscitation 133, 75–81 (2018).
Winslow, C. J. et al. The Impact of a Machine Learning Early Warning Score on Hospital Mortality: A Multicenter Clinical Intervention Trial. Crit Care Med Publish Ahead of Print, (2022).
Pereira, F. et al. Risk of 30-day hospital readmission associated with medical conditions and drug regimens of polymedicated, older inpatients discharged home: a registry-based cohort study. BMJ Open 11, e052755 (2021).
Subbe, C. P., Kruger, M., Rutherford, P. & Gemmel, L. Validation of a modified Early Warning Score in medical admissions. QJM 94, 521–526 (2001).
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E. & Featherstone, P. I. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 84, 465–470 (2013).
Chester, J. G. & Rudolph, J. L. Vital Signs in Older Patients: Age-Related Changes. J. Am. Méd. Dir. Assoc. 12, 337–343 (2011).
Gavazzi, G. & Krause, K.-H. Ageing and infection. Lancet Infect. Dis. 2, 659–666 (2002).
Hodgson, L. E. et al. A validation of the National Early Warning Score to predict outcome in patients with COPD exacerbation. Thorax 72, 23 (2017).
Méndez-Bailón, M. et al. Cancer Impacts Prognosis on Mortality in Patients with Acute Heart Failure: Analysis of the EPICTER Study. J. Clin. Med. 11, 571 (2022).
Cano-Escalera, G. et al. Mortality Risks after Two Years in Frail and Pre-Frail Older Adults Admitted to Hospital. J. Clin. Med. 12, 3103 (2023).
Olmastroni, E., Galimberti, F., Tragni, E., Catapano, A. L. & Casula, M. Impact of COVID-19 Pandemic on Adherence to Chronic Therapies: A Systematic Review. Int. J. Environ. Res. Public Heal. 20, 3825 (2023).
Henry, K. E. & Giannini, H. M. Early Warning Systems for Critical Illness Outside the Intensive Care Unit. Crit. Care Clin. 40, 561–581 (2024).
Glover, G., Metaxa, V. & Ostermann, M. Intensive Care Unit Without Walls. Crit. Care Clin. 40, 549–560 (2024).
Long, J. et al. The risk assessment tool for intensive care unit readmission: A systematic review and meta-analysis. Intensiv. Crit. Care Nurs. 76, 103378 (2023).
Fijačko, N. et al. A Review of Mortality Risk Prediction Models in Smartphone Applications. J. Méd. Syst. 45, 107 (2021).
Barrio I, Arostegui I, Rodríguez-Álvarez MX, et al. A new approach to categorising continuous variables in prediction models: proposal and validation. Stat Methods Med Res 2017; 26: 2586–2602

Tables 1 to 3 are available in the Supplementary Files section

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Derivation and Validation of a Clinical Deterioration Early Warning System (Cdews) Score in Hospital Inpatients

Status:

Version 1

Abstract

Figures

BACKGROUND

METHODS

RESULTS

DISCUSSION

Conclusions

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1