The effects of triage applying artificial intelligence on triage in the emergency department: A systematic review of prospective studies

doi:10.21203/rs.3.rs-3288343/v1

Download PDF

Research Article

The effects of triage applying artificial intelligence on triage in the emergency department: A systematic review of prospective studies

https://doi.org/10.21203/rs.3.rs-3288343/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

This study aimed to identify the effects of a prospective study applying artificial intelligence-based triage in the clinical field.

Methods

We conducted a systematic review of prospective studies. The Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) checklist was used to guide the systematic review and reporting. Three researchers independently extracted the data, assessed the study quality, and presented the findings in a descriptive summary. Inconsistencies between the researchers were resolved after discussion. We manually searched for relevant articles through databases, including CINAHL, Cochrane, Embase, PubMed, ProQuest, and two South Korean search engines (KISS and RISS) from March 9 to April 18, 2023.

Results

Of 1,633 articles, eight met the inclusion criteria for this review. Most studies applied machine learning to triage, and only one study was based on fuzzy logic. Except for one study, all used a 5-level triage classification system, and some developed target-level prediction models. Although the model performance exceeded 70%, the triage prediction accuracy varied from 33.9 to 99.9%. Other outcomes included time reduction, overtriage and undertriage checks, triage risk factors, and outcomes related to patient care and prognosis.

Conclusions

Triage nurses in the emergency department can use artificial intelligence as a supportive means for patient classification. Ultimately, we hope that it will be a resource that can reduce undertriage and positively affect patient health. Verification of the optimal artificial intelligence algorithm by conducting rigorous interdisciplinary research will be a powerful tool to support triage nurses' decision-making in overcrowded emergency departments. Thus, direct nursing activities will increase and become an important factor in improving the quality of nursing care.

Trial registration:

We have registered our review in PROSPERO (registration number: CRD***********).

Artificial Intelligence

Triage

Emergency Service

Hospital

Nurses

The emergency department has implemented a 24-hour stay limit for emergency patients since 2017 [1], based on the law concerning emergency medical care. Therefore, it is necessary to provide faster and more accurate medical services to patients requiring emergency treatment in restricted environments. Despite the regulations imposed by law, the problem of overcrowding in emergency departments persists [2]. This has resulted in a high proportion of patients visiting the emergency department who are unable to receive treatment due to the inefficient distribution of limited medical resources. It has been associated with negative disease outcomes, low satisfaction with emergency departments, and high mortality rates among patients admitted to and discharged from the emergency department [3]. Furthermore, the increasing elderly population, who generally have a high level of emergency severity, and the rising proportion of severe emergency patients, such as those with acute myocardial infarction, acute stroke, and severe trauma, including cardiac arrest [4, 5], require emergency department medical staff to provide a higher level of medical services and consume more time and medical resources for diagnostic tests than before [5–7].

The delay in the length of stay (LOS) in the emergency department perpetuates a vicious cycle intertwined with overcrowding and waiting time delays for new patients. A delay in the LOS increases the mortality rate of critically ill patients within the hospital or after admission [8, 9], increases the occurrence rate of cardiac arrest [10], and decreases patient satisfaction [11]. In other words, overcrowding in the emergency department and delays in LOS are negative factors not only for patients but also for healthcare professionals, leading to noncompliance with clinical guidelines, increased stress, and exposure to violence throughout emergency care [12]. Consequently, it is essential to prioritize patients who require care through emergency patient triage and securely manage lower-priority patients, thereby controlling the flow of patients within the emergency department [13]. This should be prioritized as a remedy for these issues.

The concept of "Triage," which refers to the triage of emergency patients, has historically been interpreted as a severity-based triage. However, it has recently been defined as a concept that encompasses both severity, which signifies the need for emergency procedures, surgeries, or hospitalization and the likelihood of death or permanent disability, as well as urgency, which indicates the need for prompt medical attention, regardless of the patient's prognosis [13]. Therefore, the purpose of classifying emergency patients in the emergency department based on severity and urgency is to prioritize patient treatment and identify those who cannot wait for medical care [14].

To determine the priority of emergency patients, triage tools have been developed and used worldwide to assess the severity and urgency of patients because relying solely on the experience and knowledge of emergency healthcare professionals can lead to inconsistencies. However, despite the development of triage tools, if emergency patients are not appropriately classified and undertriage as non-emergency patients, there is a risk that their condition will deteriorate while waiting for treatment because they are prioritized lower. However, if non-emergency patients are overtriage as emergency patients, it may result in the misallocation of emergency resources that could have been provided to other patients in need of immediate care [15]. However, accurate and prompt triage can lead to positive outcomes for patients by reducing the LOS in the emergency department [16] and allowing for quick treatment and interventions [17] through the efficient utilization of limited medical resources.

Emergency patient triage is typically performed by trained healthcare professionals who evaluate patients, make judgments regarding priorities, and often record information manually in an electronic medical record (EMR) system. In most healthcare settings [18, 19], this responsibility falls upon emergency department nurses [20]. However, the manual input method for patient triage by nurses can be hindered by factors such as the pressure to deal with many patients, the chaotic environment within the emergency department, and the presence of waiting ambulances, which can disrupt the decision-making process [21]. It has been observed that nurses’ capacity for triage, including their training, knowledge, clinical experience, and communication skills, can influence its effectiveness [22, 23]. Moreover, accurate assessment and triage are crucial for ensuring patient safety and saving lives. However, the manual input method is susceptible to subjective judgments by nurses, which can decrease the accuracy of triage and raise concerns regarding patient safety [24]. As a result, there is an emphasis on the need for efficient triage systems that can support emergency department nurses in their triage roles, helping and facilitating timely and informed decision-making [21, 25].

Artificial intelligence (AI) aims to create systems with human-like intelligence capable of learning and reasoning [26]. Machine learning (ML), a type of AI, comprises a series of algorithms that enable computers to continuously learn from data and make predictions or classify certain data. It is categorized into supervised learning, unsupervised learning, and reinforcement learning and includes various algorithms such as clustering, decision trees, deep learning, logistic regression, naive Bayes, natural language processing, neural networks, and random forests [27]. In the field of nursing, machine learning has shown the potential to support nursing plans in specific areas, such as predicting vital signs, falls, pressure ulcer occurrence, risk factors, and monitoring critically ill patients, in addition to everyday nursing activities [28, 29]. Other types of AI include fuzzy logic and computer vision. Fuzzy logic is an inference method that obtains clear output values in situations that are imprecise or ambiguous, using terms like "very," "slightly," "somewhat," or "usually" instead of binary logic [30]. It has been applied in nursing to monitor tasks, such as intravenous transfusion and vital signs [31, 32]. However, computer vision analyzes images, videos, and other visual data to make decisions, such as determining the need for diabetic retinopathy tests [33]. The application of AI in the nursing field can serve as an auxiliary tool to support clinical decision-making, enabling nurses to provide more efficient and personalized nursing plans [34].

Advancements in such technologies have shown that AI-based predictive models for emergency patient triage not only reduce the time required for manual input but also minimize the potential for errors resulting from subjective judgments by nurses [35, 36]. In a study that utilized ML techniques and the U.S.-based Emergency Severity Index (ESI) with a dataset of 130,000 cases, the model accurately classified critical- and admission-level patients with an accuracy ranging from 71.3–81.6% [35]. Furthermore, AI-based Clinical Decision Support Systems (CDSS) for emergency patient triage have been reported to improve patient prioritization, predict ward and ICU admissions, and reduce the LOS in the emergency department [25]. In domestic research, machine learning was employed to predict adult emergency patient triage, resulting in higher reliability (77.2–83.0%) between nurses experienced in triage and AI [37]. Similarly, in a study using deep learning algorithms in a pediatric emergency department, the AI model predicted mortality rates and classified patients for ICU and ward admissions more effectively than traditional triage tools [38]. AI-based triage enables swift triage of variables and accurate detection of urgency levels, thereby reducing the risk of over- or under-triaging compared to conventional manual methods and improving the prediction of clinical outcomes [39].

However, most previous studies have emphasized establishing evidence and employing big data to forecast emergency patient triage. Researchers have mostly used retrospective research techniques to verify prediction models or improve algorithms using previously acquired data. To ensure that the developed prediction models can accurately classify patients, they must be validated in actual clinical situations. Therefore, studies are required to determine whether newly developed models are useful and efficient in actual clinical settings, especially when considering the rapid advancement of AI in the healthcare field.

The purpose of this study was to evaluate the efficacy of AI-based emergency patient triage in real-world clinical settings, by carefully reviewing prospective research that makes use of AI-based triage in actual clinical situations, to improve triage accuracy and explore methods to assist emergency department nurses with triage tasks.

The specific purposes of the study were as follows.

1) Identify intervention contents of AI-based emergency patient triage systems.

2) Investigate the intervention effect of AI-based emergency patient triage systems.

2.1. Design

This systematic literature review aimed to select and synthesize relevant domestic and international studies on the efficacy of AI-based emergency patient triage systems. The objective of this study is to conduct a comprehensive literature review and provide an overview of the topic. The protocol for the systematic review was registered in The International Prospective Register of Systematic Reviews (PROSPERO) prior to conducting this study (registration number: CRD***********).

2.2. Inclusion and/or Exclusion Criteria

2.2.1. Inclusion Criteria

According to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines, the literature selection criteria for this study were defined using the Participants, Intervention, Comparison, Outcome, Study Design (PICO-SD) format [40, 41].

• Patient/Population/Problem: Emergency department
• Intervention: AI-based emergency patient triage system
• Comparison: No limit
• Outcome: No limit
• Study design: All prospective experimental studies.
• There were no limits to the study period.

2.2.2. Exclusion Criteria

• Studies conducted in settings other than the emergency department, such as pre-arrival, intensive care units, and hospital wards.
• Retrospective study design, academic conference papers, and study protocols.
• Studies without the application of AI in triage.
• Studies that did not apply the emergency patient triage system, such as the coronavirus disease-19 (COVID-19) severe patient classification.
• Studies that were not accessible in the full text.
• Studies published in languages other than Korean or English.

2.3. Study Search and Selection

2.3.1. Study Search

From March 9 to April 18, 2023, three researchers conducted a literature search and selection for this study. The search period was not restricted by the primary research query and articles published in academic journals until February 2023 were included in the search. The analysis did not include conference abstracts or research protocols, because they were not considered part of the study's quality assessment. The database search was based on the Core, Standard, Ideal (COSI) model presented by the National Library of Medicine (NLM), five foreign databases (CINAHL, Cochrane, Embase, PubMed, and ProQuest), and two domestic databases (KISS and RISS). Seven databases were selected for this study. Based on the selection and exclusion criteria, three researchers independently searched seven databases for relevant studies. To conduct an extensive literature search, the study participants and research interventions of the PICO-SD served as the search criteria. The search was conducted using MEDLINE's default search terms AND’ and ‘OR Boolean operators 'AND' and 'OR, ’ without using controlled vocabulary. Truncation is used to broaden a search by capturing multiple forms and alternative keyword endings. In selecting the search fields, only the title and abstract were selected to exclude irrelevant information such as author, table of contents, date, and department. Three researchers initially devised the search method, which was then reviewed for further refinement by a medical library information specialist affiliated with one of the researchers. The following search terms were used in international databases: [emergency department' OR emergency room OR ' OR ' OR ' OR” OR ‘OR emergency medical system OR emergency unit OR emergency care'] OR ['triage' OR 'classif*' OR 'CTAS' OR 'ESI' OR 'KTAS'] OR ('artificial intelligence' OR 'AI' OR'machine learning' OR 'deep learning' OR 'neural network' OR'reinforcement learning'). In the domestic search databases, the following Korean search terms were used: [‘응급실’] AND [‘분류’] AND [‘인공지능’].

2.3.2. Study Selection

Literature selection was conducted in accordance with the PRISMA Reporting Items for Systematic Reviews [40]. Each piece of literature was downloaded using Endnote© Version X9 with a file extension and uploaded to the literature review software Covidence (https://covidence.org) for further analysis. Co-vidence is a well-known instrument for efficient collaboration among researchers and for screening article titles, abstracts, and full texts [42]. Through a two-step procedure, eight articles were selected for the systematic literature review. In the initial phase, seven domestic and international databases were searched, yielding 1,633 articles. After eliminating 494 duplicates, the titles and abstracts of 1,139 publications were evaluated in accordance with the inclusion and exclusion criteria. A total of 1,109 articles were excluded because they did not meet the selection criteria, and 30 relevant articles were selected based on the research objectives. Eighteen articles with retrospective study designs, four unrelated to AI-based triage interventions, and articles published in languages other than English or with inaccessible full texts or duplicates were excluded, leaving five articles for inclusion. Second, a manual search was conducted to identify additional pertinent literature and examine the reference lists of the retrieved articles. Based on this procedure, 12 articles were selected. Five articles with retrospective study designs and four unrelated to AI-based triage interventions were excluded after evaluating their full texts, resulting in the inclusion of only three articles. Finally, eight articles were selected for quality evaluation in the systematic review (Fig. 1). Three researchers independently evaluated and selected relevant literature. After collectively reviewing the complete texts, any disagreements were resolved through discussion and consensus.

2.4. Quality Appraisal

The final 8 articles were prospective cohort study designs. An appraisal of the quality of the studies was conducted using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) protocol, which is the reporting guideline for observational studies, 4th edition [43], an appraisal of the study's quality was conducted. The STROBE inventory consists of six domains with a total of 22 items: title and abstract, introduction, methods, results, discussions, and other information. Each item gets evaluated with "yes," "no," or "not applicable." In this study, the three researchers assessed the quality of the 8 selected articles independently and, in cases of disagreement, reached a consensus through discussion. Each article's evaluation was converted into a Completeness of Reporting (COR) score. The COR (%) was calculated using the formula (yes / [yes + no]) × 100. Planelles et al. [44] categorized COR scores as "low" for scores between 0% and 49%, "moderate" for scores between 50% and 74%, and "high" for scores of 75% or higher. In addition, the STROBE results for all the selected articles were summarized to assess the extent of reporting for each specific item in terms of frequency and percentage.

2.5. Data Abstraction

For the systematic review, we recorded the general characteristics of the literature (first author and publication year, country, study design, clinical setting, characteristics of the study population and sample size, triage scale, applied AI programs, comparison groups, and key findings), the characteristics of AI-based programs for triage (type of triage level, delivery methods of AI, types and techniques used for AI, use of retrospective data, model performance, and classification prediction accuracy), and the effectiveness of AI-based triage (model performance, classification prediction accuracy, and other outcomes) using Microsoft Excel 2019. To ensure the veracity of the data, three researchers independently analyzed the eight final studies. In the event of disagreement, the articles were re-evaluated and a secondary review was conducted for the three researchers to reach a consensus on the final decision.

3.1. General Characteristics

From October 2017 to January 2023, we analyzed eight articles that applied AI-based triage. Table 1 presents the main characteristics of the analyzed literature. Except for one cross-sectional study (12.5%), all the studies used prospective designs, with seven articles (87.5%) adopting cohort designs [A2, A3, A4, A5, A6, A7, and A8]. Two studies were performed in Greece (25%) [A5, A6], and each study was conducted in Turkey, South Korea, Germany, Iran, Taiwan, and China (12.5% each).

Table 1

Summary of the Studies (Location: at the end of the document text file.)
No.	First Author (year)	Country	Design	Setting	Participants/ Sample size	Triage scale	AI intervention	Program Comparison	Main outcomes
A1	Aydin (2020)	Turkish	Prospective, cross- sectional	ED of tertiary hospital	ED patients StageⅠ(n = 1999) StageⅢ(n = 100) ≥ 18 years	ATS	Artificial intelligence model (DT)	• Manual ATS -Two senior residents -Paramedic and physicians	• Accuracy of classification (Cohen's kappa coefficient) - Two independent observers: 0.997 - Manual ATS and Decision trees: 0.999 - Paramedic and physicians: 0.541
A2	Cho (2022)	Republic of Korea	Prospective cohort	Level 1 ED of tertiary hospital	ED patients (n = 1063)	KTAS	A real-time medical record input assistance system with voice artificial intelligence (RMIS-AI)	• Manual KTAS -ED nurses	• Time for perform the triage task - RMIS-AI: 204sec (IQR 155, 277), - Manual record: 231sec (IQR 180, 313) • The record completion rate -1st Chief concern: 81.84%, highest value in RMIS-AI record • The accuracy of reproducing records by RMIS-AI -SBP, DBP, SPO2, Chief concern variables ≧ 50% -Categorical variables less accuracy than continuous variables
A3	Cotte (2022)	Germany	Prospective cohort	ED of university hospital	ED patients (n = 378) ≥ 18 years German- speaking Walk-in MTS level 3, ༔,༕	MTS	Symptom assessment apps (SAAs) is Ada	• Manual MTS -ED nurses	• Urgency assessment results -App’s urgency assessments matched with MTS 33.9% -Degree of App overtriage 57.1% -Degree of App undertriage 8.9% -Considered as a potential AHS 5.3% -Advice considered safe 94.7% • Agreement between MTS and the app’s advice level -Cohen's kappa coefficient: 0.033 (95% CI–0.023 to 0.089)
A4	Farahmand (2017)	Iran	Prospective cohort	ED of complex hospital	ED patients (n = 215) ≥ 18 years Acute abdominal pain	ESI	Web-based interface (View layer)	• Manual ESI -Emergency medicine physician	• First generation models (Levels 1 & 5 omitted) -Level 2: All systems (AR, CL, LR, DT, NB, NN) showed fair level of prediction with Neural Network being the highest (AUC-ROC: 0.769) -Level 3: All systems (AR, CL, LR, DT, NB, NN) showed fair level of prediction -Level 4: Decision tree was the only system with fair prediction (AUC-ROC: 0.713) • Ensemble models of ESI level 4 -Ensemble models with the Naïve Bayes algorithm highest overall accuracy (AUC-ROC: 0.839)
A5	Karlafti (2023)	Greece	Prospective cohort	ED of tertiary university hospital	ED patients (n = 322) ≥ 16 years COVID-19 patients excluded	ESI	Automatic patient screening classifier (ANN)	• Manual ESI -Measurer not confirmed	• ANN was trained for an average of 21 epochs considering all runs • Performance of the trained model -overall accuracy: 84.6% -F1 score (more appropriate performance metric in unbalanced datasets): 72.2%

Table 1

Summary of the Studies (Continue)
No.	First Author (year)	Country	Design	Setting	Participants/ Sample size	Triage scale	AI intervention	Program Comparison	Main outcomes
A6	Kipourgos (2022)	Greece	Prospective cohort	ED of tertiary university hospital	ED patients (n = 616)	ESI	Intelligent-Triage (i-TRIAGE)	• Manual ESI -Expert triage nurse	• Efficiency of the system (WEKA): ESI level accuracy -i-TRIAGE_rt(resuscitation team system) 95% -i-TRIAGE_derm(dermatological system) 72% • Rules for exporting output classes (Fuzzy Clips) -Accuracy 0.99 -Precision 0.93 -Sensitivity 0.99 -Specialization 0.99
A7	Leung (2021)	Taiwan	Prospective cohort	ED of tertiary academic medical center	ED patients (n = 146) ≥ 20 years	TTAS	TabNet with deep learning architecture	• Retrospective data	• Performance of the trained model -AUC-ROC 0.836 -Mean accuracy 0.805 • TabNet is way better than the conventional fully connected layer for handling structural data.
A8	Liu (2021)	China	Prospective cohort	ED of tertiary urban teaching hospital	ED patients (n = 17072)	ETS	Machine learning system (MLS) protocol	• Retrospective data	• MLS protocol medel performance -AUC: 0.875 ± 0.006 (CI:95%) -Performance with more than 3% increase • Life-threatening miss-triage rate -Control group 1.2% (110 out of 8839), -Intervention group 0.9% (72 out of 7936) • Called a supervising ED physician for aid in a triage decision increased 24.3% in intervention group • Associated variables with a higher mis-triage risk higher or lower HR, extreme DBP or SBP, lower SPO2, older age, arrival at night • Shock index: higher sensitivity in older patients • Pulse pressure: higher sensitivity in younger patients
Abbreviation: AI = Artificial inelligence; ANN = Artificial neural network; AR = Association rules; ATS = Australasian triage scale; AUC = Area under the curve; AUC-ROC = Area under the receiver operating characteristic curve; CI = Confidence interval; CL = Clustering; DBP = Diastolic blood pressure; DT = Decision Tree; ED = Emergency department; ESI = Emergency severity index; ETS = Emergency Triage Scale; ICU = Intensive care unit; IQR = Inter quartile range; LR = Logistic regression; MTS = Manchester triage scale; N/A = Not applicable; NB = Naïve Bayes; NN = Neural network; ROC = Receiver operating characteristic curve; SBP = Systolic blood pressure; SPO2 = = Oxygen saturation; WEKA = Waikato Environment for Knowledge Analysis

All studies examined patients who visited university hospitals or tertiary care emergency departments, with sample sizes ranging from 146 to 17,072 individuals. Five of the eight studies [A1, A3, A4, A5, and A7] imposed an age restriction of 16 years, whereas the remaining three studies [A2, A6, and A8] did not. Regarding the selection criteria for study participants, two studies (25.0%) did not include all triage levels [A3, A4], excluding levels 1 and 2 [A3] or levels 1 and 5 [A4], owing to the small sample size. In addition, one study (12.5%) [A4] focused on specific conditions such as acute abdominal pain, and another study [A5] excluded all COVID-19-related patients.

Seven (87.5%) of the eight articles included in this study adopted a 5-level triage scale. The ESI developed in the United States was used in three articles (37.5% of the total) [A4, A5, A6]. Other scales included the MTS, which was developed in the United Kingdom [A3]; the ATS, which was developed in Australia [A1]; the KTAS, which was developed in South Korea [A2]; and the TTAS, which was developed in Taiwan [A7], each used in one study (12.5% each). Another study (12.5%) [A8] used a China-developed 4-level Emergency Triage Scale (ETS).

3.2. Quality Appraisal

Table 2 displays the outcomes of the quality assessment of the selected studies, which included one prospective cross-sectional study and seven prospective cohort studies, using the STROBE checklist. 2 articles (25%) [A1, A6] were rated "high" with a COR score of 77%, and the remaining 6 articles (75%) [A2, A3, A4, A5, A7, A8] were rated "high" with COR scores spanning from 86–100%. Table 3 displays the outcomes of the itemized analysis based on the STROBE guidelines. Thirteen of the 22 analyzed items (background and rationale, study design, setting, participants in methods, variables, data sources and measurements, quantitative variables, descriptive data, outcome data, main results, key results, interpretation, and generalizability) were reported in full in each of the eight studies (100%). The remaining two items, bias and limitations, were described well in seven studies (87.5%), and five items (title and abstract, objectives, participants in results, other analyses, and funding) were described well in six studies (75%), totaling 20 items (90.9%) with good descriptions. Only four studies [A2, A3, A4, and A5] (50.0%) explained how the study size was determined. This was the most inadequately reported aspect in the examined studies. In addition, only 5 studies (62.5%) adequately described the statistical analysis methods in the research methodology section [A2, A3, A4, A7, and A8], indicating an insufficient level of reporting.

Table 2

Critical Appraisal with STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) (Location: positioned after "3.2. Quality Appraisal")
No.	First Author (year)	STROBE Item Number																						COR/Quality
No.	First Author (year)	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	COR/Quality
A1	Aydin (2020)	√	√	√	√	√	√	√	√	√		√			√	√	√	√	√		√	√		17/22(77%) High
A2	Cho (2022)	√	√		√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	21/22(95%) High
A3	Cotte (2022)	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√	22/22(100%) High
A4	Farahmand (2017)	√	√		√	√	√	√	√	√	√	√	√	√	√	√	√		√	√	√	√	√	20/22(91%) High
A5	Karlafti (2023)		√	√	√	√	√	√	√	√	√	√		√	√	√	√		√	√	√	√	√	19/22(86%) High
A6	Kipourgos (2022)		√	√	√	√	√	√	√			√		√	√	√	√	√	√	√	√	√		17/22(77%) High
A7	Leung (2021)	√	√	√	√	√	√	√	√	√		√	√		√	√	√	√	√	√	√	√	√	20/22(91%) High
A8	Liu (2021)	√	√	√	√	√	√	√	√	√		√	√	√	√	√	√	√	√	√	√	√	√	21/22(95%) High
Check symbols indicate the presence of these items in the chosen studies. Blank spaces indicate the absence of the item. The evidence column indicates the number of STROBE items present in the article relative to the total number of STROBE items. The quality of studies was measured according to the Completeness of Reporting (COR) score: “low” (COR: 0–49%), “moderate” (COR: 50–74%) and “high” if _75% of items were met.

Table 3

Critical Appraisal by Detail items from STROBE (N = 8)
STROBE Item No.	Scale Item		N	%
1	Title and Abstract		6	75.0%
2	Introduction	Background/Rationale	8	100%
3		Objectives	6	75.0%
4	Methods	Study Design	8	100%
5		Setting	8	100%
6		Participants	8	100%
7		Variables	8	100%
8		Data Sources/Measurement	8	100%
9		Bias	7	87.5%
10		Study size	4	50.0%
11		Quantitative variables	8	100%
12		Statistical methods	5	62.5%
13	Results	Participants	6	75.0%
14		Descriptive data	8	100%
15		Outcome data	8	100%
16		Main results	8	100%
17		Other analyses	6	75.0%
18	Discussion	Key results	8	100%
19		Limitations	7	87.5%
20		Interpretation	8	100%
21		Generalizability	8	100%
22	Other Information	Funding	6	75.0%
(Location: positioned after "3.2. Quality Appraisal")

3.3. Characteristics of AI-Based Triage

Table 4 presents the characteristics of AI-based triage literature. Among these articles, 87.5% (seven articles) [A1, A2, A4, A5, A6, A7, A8] used cloud-based network delivery, whereas 12.5% (one article) [A3] used a standalone executable application.

Table 4

Summary of the AI Intervention features (Location: at the end of the document text file.)
No.	First Author (year)	Types of Triage levels in the study	AI delivery	AI classification	AI technology	Utilize retrospective data	Evaluation for Classification Model
No.	First Author (year)	Types of Triage levels in the study	AI delivery	AI classification	AI technology	Utilize retrospective data	Model Performance	Prediction Accuracy of classification
A1	Aydin (2020)	ATS 5-level	Cloud	Machine learning	Decision trees	Yes	N/A	• Manual ATS & Decision trees 99.9% (kappa)
A2	Cho (2022)	KTAS 5-level	Cloud	Machine learning	Natural Language Processing (NLP)	Yes	N/A	N/A
A3	Cotte (2022)	MTS: 3,4,5-level Ada: 8- level	App	Machine learning	Deep learning (Complex Bayesian networks)	No	N/A	• App’s urgency assessments that matched with MTS 33.9% (kappa)
A4	Farahmand (2017)	ESI 2,3,4-level (Triage Levels 1 and 5 were omitted due to low number of cases)	Cloud	Machine learning	Mixed-model approach (AR, CL, LR, DT, NB, NN)	Yes	Level 2,3,4 order (AUC) • Association rules 0.737, 0.704, 0.459 • Clustering 0.790, 0.739, .751 • Decision tree 0.712, 0.712, 0.770 • Logistic regression 0.714, 0.763, 0.642 • Navie bayes 0.635, 0.708, 0.839 • Neural network 0.782, 0.749, 0.495	Accuracy (kappa) • Association rules 75.00% • Clustering 75.00% • Decision tree 71.88% • Logistic regression 73.44% • Navie bayes 78.13% • Neural network 73.44%
A5	Karlafti (2023)	ESI 5-level	Cloud	Machine learning	Deep learning (Feed-Forward Neural Network)	No	• F1 score: 72.2% Level 1–5 order (kappa value) • Precision: 0.33, 0.44, 0.94, 0.80, 0.85 • Recall: 1.0, 0.67, 0.86, 0.86, 0.83	• ESI level overall accuracy: 84.6%
A6	Kipourgos (2022)	ESI 5-level	Cloud	Fuzzy logic	Fuzzy Clips Model	No	• Precision 93% • Sensitivity 99% • Specialization 99%	• Fuzzy accuracy 99% • WEKA(ML) accuracy subgroup72 ~ 95%
A7	Leung (2021)	TTAS 5-level	Cloud	Machine learning	Mixed-model (TabNet with deep learning architecture)	Yes	• Mean AUC-ROC 0.836 • Sensitivity 0.803 • Specificity 0.807	• Mean accuracy 0.805
A8	Liu (2021)	ETS 4-level	Cloud	Machine learning	SHAP14: game theoretic approach	Yes	• AUC 0.875 ± 0.006 (95%CI)	• Life-threatening mis-triage’ rate -Control group 1.2% (110 out of 8839), -Intervention group 0.9% (72 out of 7936)
Abbreviation: App = Application; AR = Association Rules; AUC = Area under the curve; AUC-ROC = Area under the receiver operating characteristic curve; CL = Clustering; LR = Log`istic Regression; DT = Decision Tree; NB = Naïve Bayes; NN = Neural Network; SHAP14 = SHapley Additive exPlanations

For triage, seven articles (87.5%) [A1, A2, A3, A4, A5, A7, A8] were based on machine learning, whereas one article (12.5%) [A6] used fuzzy logic. Two (25%) [A4, A7] out of the seven machine-learning-based articles used hybrid models, and two (25%) [A3, A5] applied deep learning. In addition, each study used decision trees [A1], natural language processing [A2], and a game-based method known as SHAP14 [A8] (12.5% each). An article based on fuzzy logic (12.5%) [A6] utilized a fuzzy clip model and compared its results with those of the Waikato Environment for Knowledge Analysis (WEKA), a type of machine learning.

Comparative studies between AI and manual input methods comprised six of the eight articles (75%) [A1, A2, A3, A4, A5, A6], whereas comparative studies between AI and retrospective data comprised two articles (25%) [A7, A8].

3.4. Effects of AI-Based Triage

3.4.1. Model Performance

Among the selected studies, five out of eight articles (62.5%) [A4, A5, A6, A7, A8] examined the performance of the model, and all of them demonstrated high performance, with an accuracy above 70% (Tables 4 and 5). The model with the best performance was the fuzzy clip model that applied fuzzy logic, with 93% accuracy, 99% sensitivity, and 99% specificity [A6]. Using a game-based approach, the SHAP14 model demonstrated a high area under the curve (AUC) of 0.875 [A8]. A hybrid model that introduced a new framework by combining TabNet and deep learning achieved an area under the receiver operating characteristic curve (AUC-ROC) of 0.836 [A7]. In the literature that used a hybrid model based on ESI, the accuracy of the Naive Bayes algorithm used in the ensemble model was 78.13%, which was higher than the accuracy of the related rules, clustering, decision trees, logistic regression, and neural networks [A4]. In addition, the feed-forward neural network's F1 score in deep learning was 72.2%. The analysis of the F1 score for each ESI level revealed that levels 1 and 2 were performance-degrading factors [A5].

Table 5

Effect of AI-based Triage (Location: at the end of the document text file.)
Outcomes		Study No.
Outcomes		A1	A2	A3	A4	A5	A6	A7	A8
Model Performance	AUC				0.495- 0.839			0.836	0.875
	F1 score (%)					72.20
	Precision (%)					33–94	93
	Recall (%)					67–100
	Sensitivity (%)						99	80.3
	Specialization (%)						99	80.7
Prediction Accuracy	Kappa (%)	99.90		33.90	71.88- 78.13
	Matched data/ Total data (%)					84.60	99	80.5
	Mis-triage rate (%)								0.9
Other outcomes	Time		↓ (27sec)
	Record completion rate		↑
	Accuracy of reproducing records (ICC)		↑ (50% in some)
	Overtriage/ Undertriage (%)			57.1% / 8.9%
	Discharged							Clinics by/ Triage
	Admitted							Clinics by/ Acute change
	Global							Clinics by
	Mis-triage risk factors								Arrival mode, Age, Sex
Abbreviation: AUC = Area under the curve; AUC-ROC = Area under the receiver operating characteristic curve; ICC = Intraclass correlation coefficient; sec = seconds;

3.4.2. Triage Prediction

Seven of the eight selected articles (87.5%) [A1, A3, A4, A5, A6, A7, A8] examined classification predictions and made triage predictions ranging from 33.9–99.9%. The model with the most accurate triage predictions was a machine learning model utilizing a decision tree, with a Cohen's kappa coefficient of 99.9% concordance with the manual input [A1]. The accuracy of the fuzzy clip model based on fuzzy logic was 99.9% [A6]. The accuracy of the study using the feed-forward neural network algorithm was 84.6% [A5], whereas the accuracy of the Bayesian network algorithm using articles was only 33.9% [A3]. A hybrid model combining TabNet and deep learning achieved an accuracy of 80.5% in triage prediction [A7]. In the ensemble model, the naïve Bayes algorithm was used to analyze the classification prediction of each ESI level. The lowest prediction accuracy was observed at ESI Level 2, with a score of 63.5%. The highest prediction performance was at Level 4, with a score of 83.9%.

3.4.3. Other Results

Four of the eight articles (50.0%) [A2, A3, A7, A8] of the included literature provided explanations of the model performance, classification process, and treatment- and prognosis-related factors in triage.

First, 3 studies (37.5%) [A2, A3, A8] presented additional classification-related findings. One study [A2] examined the average time required for triage and found that AI was 27 seconds faster than manual input; however, the completion rate of the records was only 81.84%, indicating incomplete documentation. Another study [A3] compared the classification of urgency and reported an overtriage rate of 57.1% and an undertriage rate of 8.9%, and 94.7% of cases were accurately classified without exposing patients to potential risks. Another study [A8] identified risk factors that could result in significant classification errors, such as higher or lower heart rate, extreme diastolic or systolic blood pressure, lower oxygen saturation, older age, and arrival at night. Elderly individuals exhibited high sensitivity as measured by the shock index, whereas younger individuals exhibited high sensitivity as measured by pulse pressure.

Another study (12.5%) [A7] reported patient treatment and prognosis-related findings. The classification of urgency was higher when patients arrived by ambulance, acute changes were associated with hospitalization factors, and low triage levels were associated with discharge factors [A7].

Our study was conducted to systematically review the characteristics of prospective studies that applied AI to emergency triage systems and integrate their effects.

Six types of triage were applied in the studies included in this review; among them, ESI was the most common, with three [A4, A5, A6]. ESI is a classification scale divided into five levels, from level 1 requiring immediate resuscitation to level 5 of least urgent, developed in the United States in 1999 and used in an increasing number of countries [14]. Except for one study that classified the ETS into four levels [A8], most studies used a 5-level triage. This is due to the publication of previous studies showing that the 5-level triage has higher reliability and validity than the 3-level triage used in the past [45, 46], and it is thought that this is because the triage developed and used in each country follow the 5-level triage.

In our study, most of the 5-level triages were used; however, in two studies, a predictive model was trained by setting the target classification stage, which had the advantage of being able to accurately and independently classify patients in the target classification stage [A3, A4]. Among them, patients with levels 1 and 2, which were considered severe and life-threatening in one study, were excluded because AI-based interventions were judged to be unsafe and unfeasible [A3]. This highlights the limitations of the currently developed AI-based triages. As a result supporting this, among the eight literatures included in this study, the results of two studies that confirmed the model performance or classification prediction for each classification level included in the study can be presented. In a study by Karlafti et al. [A5], Levels 1 and 2 were factors that lowered the model performance, and in a study by Farahmand et al. [A4], Level 2 had the lowest level of prediction of emergency patient classification. When triaging emergency patients, first-impression risk assessment, in which the overall patient condition is grasped within the first 3–5 seconds, is very important [47]. The currently developed AI-based triage identifies and classifies patients using only data; however, nurses can evaluate the risk of first impressions through face-to-face observation of patients. Patients with levels 1 and 2, which are considered to have severe and/or life-threatening conditions, can be a limitation of AI-based triage, and cannot be observed because there is also a lot of information that can be obtained through the observation of the patient's appearance.

According to the South Korea 2021 Emergency Medical Statistical Yearbook, only 7.1% of patients visiting the emergency department had a high severity of KTAS 1 and 2, but 92.9% of patients had KTAS 3–5, it is becoming the main cause of emergency department overcrowding [6]. In particular, in the current triage system, classification accuracy is reduced owing to middle-level classification ambiguity, such as level 3 [48], and some patients classified as non-emergency patients face problems of undertriage and overtriage [49, 50]. In addition, previous studies have shown that a triage system using an automated algorithm predicts hospitalization for ESI level 3–5 patients better than nurses do, suggesting the need for AI-based triage to increase the accuracy of classification by targeting level 3–5 patients [51]. The use of AI-based triage targeting a specific level is expected to increase and accelerate the accuracy of decision-making by triage nurses, thereby reducing the LOS in the emergency department and improving the flow of the emergency department.

The term "AI delivery method" generally refers to an approach for spreading and providing services to users. This delivery method is set by developers to implement AI effectively rather than simply spreading algorithmic models when designing the best possible medical delivery system for a given problem in the medical field [52]. The AI delivery methods used in our study can be divided into standalone applications and cloud-based networks. Stand-alone applications have the advantage of better data privacy than cloud-based networks, as all data remain on the local device without an Internet connection and respond faster owing to the lack of network latency. However, maintenance and updating must be performed for each device, so scalability is poor and there is a risk of inconsistency. Therefore, only one study was used in this systematic review. Cloud-based networks are scalable, easy to maintain and update, and do not require much storage space on the device used; therefore, they were used in seven out of the eight studies. However, a stable Internet connection is essential, and when unstable, delays may occur. Because of the nature of cloud-based services, privacy-related problems may occur compared to standalone applications [53, 54]. Therefore, the delivery method suitable for effectively implementing AI-based programs in the future may vary depending on the available resources and user requirements.

In our meta-analysis, seven studies applied machine learning, and one study applied fuzzy logic. Machine learning has the advantage of being easy to implement because it is designed as a series of algorithms that computers continuously learn [27], but in this systematic review, the performance of the model varied from 60–99%. On the other hand, one study predicted classification using fuzzy logic, but the fuzzy clip model accurately classified 615 out of 616 cases by 99%, compared to 72–95% for WEKA, a type of machine learning. [A6]. A previous study that applied fuzzy logic to an ESI classification system using retrospective data also showed 99% accuracy. Compared with machine learning methods, this facilitates the processing of ambiguous variables, thereby increasing the accuracy and predictive power of classification. Unlike machine learning, which requires repetitive learning, classification worked quickly and could help support emergency department triage nurses’ decision-making [55]. However, it is dangerous to judge that fuzzy logic is superior only because fuzzy logic has a higher accuracy than machine learning in the results of this study. Notably, costs and benefits may vary depending on the sample used, available resources, and knowledge [56]. Based on the results of this study, it is expected that the two AI methods can be used in a complementary manner to efficiently solve complex problems.

Two studies combined various machine learning models in this systematic review. When machine learning was first introduced, research was conducted to verify the performance of single models. Recently, various algorithm models have been mixed and repeatedly trained, and better-performing algorithms have been constructed and applied in clinical environments [57]. For example, the trauma hybrid-suite entry algorithm (THETA), which was developed to classify patients with severe trauma, combines six algorithms–Bayesian ridge regression, linear regression, multilayer perceptron, clustering, support vector machine, and XGBoost–into a model. It showed 2–3 times higher performance than existing algorithms, making predictions more robust [58]. In this study, the ensemble model that combined the prediction results of the first-generation model showed an approximately 3% better performance than the prediction result of the first-generation model [A4]. However, one study failed to confirm the model performance and prediction accuracy of the classification [A2]. It collects categorical information such as chief complaints and medical history using natural language processing based on voice recognition. This is because the Korean language, which is no simpler than the continuous category, has limitations expressed in various words, and is difficult to process in natural languages [59]. There may have been limitations in the performance and classification prediction that could be evaluated quantitatively.

In this study, the classification accuracy of the same type of machine learning model was different, and the classification accuracy was different when the machine learning model type was different, even for the same sample type. The two studies using deep learning had a low classification predictive value of 33.9% [A3] and a high value of 84.6% [A5], indicating a large difference. This is because the types of deep learning algorithms, such as Bayesian networks and feedforward neural networks, are different. In the case of the Bayesian network model, 33.9% of the classifications completely matched the gold standard classification level, and 57.1% of the overtriages showed that 91% were classified conservatively. In addition, the classification accuracy of the study using the decision tree was very high at 99.9% [A1]; however, the classification accuracy of the first-generation study using only the decision tree in the mixed model was approximately 70%. Even in the same decision tree model, the classification accuracy differed, and even in the same sample, the accuracy differed depending on the type of machine learning model used [A4]. Therefore, as in all intervention studies, even with the same program, if the sample is not the same, the results will be different. Even with the same sample, different machine-learning methods can show different levels of accuracy [60].

Thus, in the results of this study, there was no AI model with perfect performance or prediction accuracy. Therefore, for technical supplementation, data that can be continuously monitored in the future should be continuously introduced to optimize the algorithm, and follow-up studies should be conducted to increase high-level performance and classification prediction. In addition, because triage is a serious issue linked to patient safety, it is important to develop a new model for accurate prediction and find the optimal model before applying it to actual patients by continuously comparing it with previous studies.

Most of the studies in this systematic review focused on classification accuracy and prediction, but some studies dealt with interesting results, such as a reduction in classification time, emergency classification comparison, risk factors for classification errors, treatment, and prognosis. Cho et al. [A2] created a real-time medical record input assistance system by applying natural language processing to convert raw voice data generated in the clinical field into text using a speech-to-text algorithm. The researcher compared the time required to classify emergency patients using the AI system with the time required to classify emergency patients using only the existing handwriting input method, which was applied by the nurse during triage through a Bluetooth microphone. Consequently, the classification time using the AI system was 27 s shorter than the manual classification time. Although the record completion rate was 81.84% and the record reproduction accuracy was only 50% in the study by Cho et al. [A2], it is believed that if the AI model's completeness increases, it will contribute to the possibility of clinical use and the reduction of classification time as a classification aid for nurses. A study by Cotte et al. [A3] confirmed the effectiveness of AI-based triage by focusing on undertriage and overtriage of classification and the resulting potential threat situation. In addition, by confirming the retrospective evaluation of the medical staff's manual classification, not only the nurse's manual classification and comparison but also the retrospective evaluation of the classification results, it was possible to supplement the limitations of subjective judgment when setting the medical staff's manual classification as the gold standard. In addition, by checking the variables to be considered in AI-based triage [A8] and those to be considered when evaluating predictive effects [A7], efforts to supplement the limitations of programs developed thus far should be continuously made.

The quality appraisal of the eight studies included in this systematic review revealed that the statistical methods and study sizes were somewhat insufficiently described. Among the statistical analysis methods in three studies, missing value treatment, confounding factor control, and sensitivity analysis were not mentioned. Although there was a record of the number of samples in all studies, four studies did not fully describe the basis for calculating the study size. Depending on the treatment method, missing values and confounding factors threaten internal validity, which can cause errors or biases and affect the research results [61, 62]. In the future, when designing a study to verify the effect of AI-based triage, an analysis method that includes or controls for missing values and confounding factors should be considered and recorded as an important part of observational studies to present correct results. In addition, when planning a study, the basis for sampling, based on statistical grounds and similar previous studies, should be specified in detail. STROBE was first drafted in 2004, then the 4th edition was revised in 2007; it is currently being used as a reporting guideline for observational studies [43]. However, unlike other quality appraisal tools, no subsequent revisions were made. Because some of the detailed items in the literature are not suitable for evaluation or present somewhat strict standards, the guidelines need to be revised and supplemented through adjustment and discussion for proper application.

Through this study, it is meaningful to compare the effects of various AI models in prospective studies, providing basic data that can support the work of nurses for triage in emergency departments. AI cannot replace real nurses because nurses' interactions with patients may contain important information that is difficult to enter into the system [51]. However, through this study, it is considered that identifying the degree of prediction of various models according to the type of model can be a supportive means of clinical decision-making for triage in emergency departments where emergency situations occur simultaneously. In addition, more rigorous interdisciplinary studies should be conducted to examine their effectiveness in improving patient health and other outcomes before they can be applied in real-world clinical settings. For interdisciplinary research, it is necessary to provide an environment in which nurses can develop, use, and evaluate AI models. Support and intervention at the public level, such as local governments, should precede the integration of AI-related subjects into the nursing curriculum and provide nursing students with opportunities to learn new knowledge and skills. Thus, if the optimal algorithm is verified and the technology is standardized, it will become a powerful tool to support triage nurses' decision-making in overcrowded emergency rooms. Furthermore, accurate and rapid triage can reduce undertriage and overtriage, and improve emergency room flow. Ultimately, we hope that this will become a resource that can positively affect the treatment outcomes of patients.

This study had several limitations. First, the predictive accuracy of the AI model was measured based on the judgment levels of nurses and doctors as the gold standard. When evaluating the performance of an AI model, it is important to compare it with a gold standard. Using the judgments of nurses and doctors as the gold standard for classification raises concerns that bias among raters may occur because the subjective opinion of the rater affects the classification result [22–24]. In this study, only one supplemented the vulnerability of the gold standard by adding the results of specialists, and all others were compared based on the judgment results of nurses and doctors, which may vary depending on the practitioner. Although the subjects of the classification decision for comparison were variously trained nurses, emergency department nurses, triage nurses, professional triage nurses, and doctors with specialized training and extensive experience, inter-recipient reliability was measured in only one study. To eliminate the subjectivity of implementers and raise the gold standard, a comparison through reliability measurement between implementers should be made in future studies, and it is thought that a certain level of qualification standards, such as basic education and experience required as a classifier, should be applied. In addition, in follow-up studies, it is necessary to secure the reliability of the gold standard by using clearly defined empirical criteria such as 'length of stay in emergency department,’ ‘inpatient rate in intensive care unit,’ and 'in-hospital mortality,’ which are not open to interpretation or subjectivity [63]. Second, the selected studies were all conducted using a prospective observational design, and the results cannot be generalized to all emergency department environments because of the limited number of analyzed studies and the environments of various countries in which the studies were conducted. Therefore, high-quality experimental studies applied to clinical practice should be conducted in the future through the verification of various models, identification of factors that can increase classification accuracy, and comparative analysis with prior literature.

Through this study, it was found that the effect of triage applied with AI in the emergency department was confirmed to play a supporting role in the classification of emergency patients. Therefore, in an emergency department with high variability, where various emergencies occur, it can be effectively used for the distribution of human resources and appropriate work arrangements when there is a lack of triage experience or limited manpower. This can be effectively applied in emergency situations by providing an improved environment for patients with a high level of moderate need and a high rate of direct nursing care, while also improving problems that may arise due to overcrowding in emergency departments. Furthermore, it is expected that the cumulative research results can be applied in practice after confirming the results using an integrated statistical analysis method.

AI, artificial intelligence

AUC, area under the curve

AUC-ROC, area under the receiver operating characteristic curve

CDSS, Clinical Decision Support Systems

COR, Completeness of Reporting

COVID-19, coronavirus disease-19

EMR, electronic medical record

ESI, Emergency Severity Index

ETS, Emergency Triage Scale

LOS, length of stay

ML, machine learning

PICO-SD, Participants, Intervention, Comparison, Outcome, Study Design

PRISMA, Preferred Reporting Items for Systematic Review and Meta-Analysis

STROBE, Strengthening the Reporting of Observational Studies in Epidemiology

THETA, trauma hybrid-suite entry algorithm

WEKA, Waikato Environment for Knowledge Analysis

Ethics approval and consent to participate

This study was conducted after receiving a review exemption from the Ewha womans university institutional review board as a systematic review study of previous studies (IRB No. ****-**-***-***). Consent from participants was not required, as all data gathered or analyzed during this study has been incorporated into published journals.

Consent for publication

Not applicable.

Availability of data and materials

The data supporting the study findings are available from the corresponding author upon reasonable request.

Competing interests

The authors declare no competing interests.

Funding

Not applicable.

Authors' contributions

B.G.H., B.D.I., and Y.N.Y. envisioned the systematic review and drafted the manuscript. B.G.H., B.D.I., and Y.N.Y. are involved in data search and assessing articles for eligibility and the risk of bias. B.G.H., B.D.I., and Y.N.Y. read and approved the final version of the manuscript. B.G.H., B.D.I., and Y.N.Y. wrote the manuscript together (B.D.I. was mainly in charge of the background and discussion and B.G.H. was mainly involved in the method and discussion). All authors reviewed and revised the manuscript.

Acknowledgements

Not applicable

Korea Ministry of Government Legislation. Emergency Medical Services act. Article 33 – 2. 2016. https://www.law.go.kr/LSW/lsSc.do?dt=20201211&subMenuId=15&-menuId=1&query=%EC%9D%91%EA%B8%89%EC%9D%98%EB%A3%8C%E-C%97%90%EA%B4%80%ED%95%9C%EB%B2%95%EB%A5%A0#liBgcolor0. Accessed 10 April 2023.
Lindner G, Woitok BK. Emergency department overcrowding: analysis and strategies to manage an international phenomenon. Wien Klin Wochenschr. 2021;133(5–6):229–33.
Carter EJ, Pouch SM, Larson EL. The relationship between emergency department crowding and patient outcomes: a systematic review. J Nurs Scholarsh. 2014;46(2):106–15.
National Emergency Medical Center. Severe emergency disease emergency room visit status report (NMC-2022-0045-10). 2022. https://www.e-gen.or.kr/nemc/statistics_annual_report.do?brdclscd=03. Accessed 15 April 2023.
Kahn JH, Magauran BG Jr, Olshaker JS, Shankar KN. Current trends in geriatric emergency medicine. Emerg Med Clin North Am. 2016;34(3):435–52.
National Emergency Medical Center. Emergency medical statistical annual report (11-1352000-001248-10). 2022. https://www.e-gen.or.kr/nemc/statistics_annual_report.do. Accessed 30 April 2023.
Burgess L, Ray-Barruel G, Kynoch K. Association between emergency department length of stay and patient outcomes: A systematic review. Res Nurs Health. 2022;45(1):59–93.
Sabaz MS, Asar S, Cukurova Z, Sabaz N, Doğan H, Sertcakacilar G. Effect of delayed admission to intensive care units from the emergency department on the mortality of critically ill patients. Iran Red Crescent Med J. 2020;22(6). 10.5812/ircmj.102425.
Zhang Z, Bokhari F, Guo Y, Goyal H. Prolonged length of stay in the emergency department and increased risk of hospital mortality in patients with sepsis requiring ICU admission. Emerg Med J. 2019;36(2):82–7.
Kim JS, Seo DW, Kim YJ, Jeong J, Kang H, Han KS, et al. Prolonged length of stay in the emergency department and increased risk of in-hospital cardiac arrest: a nationwide population-based study in South Korea, 2016–2017. J Clin Med. 2020;9(7):2284.
Walsh M, Knott JC. Satisfaction with the emergency department environment decreases with length of stay. Emerg Med J. 2010;27(11):821–8.
Morley C, Unwin M, Peterson GM, Stankovich J, Kinsman L. Emergency department crowding: A systematic review of causes, consequences and solutions. PLoS ONE. 2018;13(8):e0203316.
Park JB, Je SH, Oh JH, Kim O, Park YS, Ko J. Provider manual of Korean Triage and Acuity Scale. 2nd ed. Seoul: Korean Society of Emergency Medicine; 2017. pp. 2–3.
Gilboy N, Tanabe P, Travers D, Rosenau AM, Schaumburg IL, Emergency Nurses Association. ; 2020. Emergency Severity Index (ESI): A triage tool for emergency department care. 4th version. Available from: https://www.ena.org/docs/default-source/education-document-library/esi-implementation-handbook-2020.pdf?sfvrsn=fdc327df_2.
Silva JAD, Emi AS, Leão ER, Lopes MCBT, Okuno MFP, Batista REA. Emergency Severity Index: accuracy in risk classification. Einstein (São Paulo). 2017;15(4):421–7.
Ausserhofer D, Zaboli A, Pfeifer N, Solazzo P, Magnarelli G, Marsoner T, et al. Errors in nurse-led triage: an observational study. Int J Nurs Stud. 2021;113:103788.
Oredsson S, Jonsson H, Rognes J, Lind L, Göransson KE, Ehrenberg A, et al. A systematic review of triage-related interventions to improve patient flow in emergency departments. Scand J Trauma Resusc Emerg Med. 2011;19:43.
Park JB, Choi HJ, Kang BS, Kim CS, Kang HG, Lim TH. A nationwide survey of Korean emergency department triage systems and scales: A first step towards reform of the emergency medical service system. J Korean Soc Emerg Med. 2014;25(5):499–508.
Roquette BP, Nagano H, Marujo EC, Maiorano AC. Prediction of admission in pediatric emergency department with deep neural networks and triage textual data. Neural Netw. 2020;126:170–7.
Dulandas R, Brysiewicz P. A description of the self-perceived educational needs of emergency nurses in Durban, KwaZulu-Natal, South Africa. Afr J Emerg Med. 2018;8(3):84–8.
Reay G, Smith-MacDonald L, Then KL, Hall M, Rankin JA. Triage emergency nurse decision-making: incidental findings from a focus group study. Int Emerg Nurs. 2020;48:100791.
Moon SH. Factors affecting triage competency for nurses working in Korean emergency departments. Crisis Emerg Manag. 2021;17(3):125–35.
Hwang S, Shin S. Factors affecting triage competence among emergency room nurses: A cross-sectional study. J Clin Nurs. 2023;32(13–14):3589–98.
Lee JH, Park YS, Park IC, Lee HS, Kim JH, Park JM, et al. Over-triage occurs when considering the patient’s pain in Korean Triage and Acuity Scale (KTAS). PLoS ONE. 2019;14(5):e0216519.
Fernandes M, Vieira SM, Leite F, Palos C, Finkelstein S, Sousa JMC. Clinical decision support systems for triage in the emergency department using intelligent systems: a review. Artif Intell Med. 2020;102:101762.
Clancy TR. Artificial intelligence and nursing: the future is now. J Nurs Adm. 2020;50(3):125–7.
Shafaf N, Malek H. Applications of machine learning approaches in emergency medicine; A review article. Arch Acad Emerg Med. 2019;7(1):34.
Pepito JA, Locsin R. Can nurses remain relevant in a technologically advanced future? Int J Nurs Sci. 2019;6(1):106–10.
Ng ZQP, Ling LYJ, Chew HSJ, Lau Y. The role of artificial intelligence in enhancing clinical nursing care: A scoping review. J Nurs Manag. 2022;30(8):3654–74.
Zadeh LA. Fuzzy sets. Inf Control. 1965;8(3):338–53.
Qiao Y, Wu X. A design of intelligent transfusion monitoring system based on the DSP and fuzzy control algorithm. Fourth International Conference on Intelligent Control and Information Processing (ICICIP). 2013.
Stevens N, Giannareas AR, Kern V, Viesca A, Fortino-Mullen M, King A et al. Smart alarms: multivariate medical alarm integration for post CABG surgery patients. In: Proceedings of the 2nd ACM SIGHIT international health informatics symposium; 2012. p. 533 – 42.
Esteva A, Chou K, Yeung S, Naik N, Madani A, Mottaghi A, et al. Deep learning-enabled medical computer vision. NPJ Digit Med. 2021;4(1):5.
McGrow K. Artificial intelligence: essentials for nursing. Nursing. 2019;49(9):46–9.
Raita Y, Goto T, Faridi MK, Brown DFM, Camargo CA, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019;23(1):64.
Yu JY, Jeong GY, Jeong OS, Chang DK, Cha WC. Machine learning and initial nursing assessment-based triage system for emergency department. Healthc Inf Res. 2020;26(1):13–9.
Choi SW, Ko T, Hong KJ, Kim KH. Machine learning-based prediction of Korean Triage and Acuity Scale level in emergency department patients. Healthc Inf Res. 2019;25(4):305–12.
Kwon JM, Jeon KH, Lee M, Kim KH, Park J, Oh BH. Deep learning algorithm to predict need for critical care in pediatric emergency departments. Pediatr Emerg Care. 2021;37(12):e988–94.
Sánchez-Salmerón R, Gómez-Urquiza JL, Albendín-García L, Correa-Rodríguez M, Martos-Cabrera MB, Velando-Soriano A, et al. Machine learning methods applied to triage in emergency services: A systematic review. Int Emerg Nurs. 2022;60:101109.
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
Saaiq M, Ashraf B. Modifying Pico Question into Picos model for more robust and reproducible presentation of the methodology employed in A scientific study. World J Plast Surg. 2017;6(3):390–2.
Babineau J. Product review: Covidence (systematic review software). J Can Health Libr Assoc. 2014;35(2):68–71.
Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344–9.
Díaz Planelles I, Navarro-Tapia E, García-Algar Ó, Andreu-Fernández V. Prevalence of potentially inappropriate prescriptions according to the new STOPP/START criteria in nursing homes: A systematic review. Healthc (Basel). 2023;11(3):422.
Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch Ärztebl Int. 2010;107(50):892–8.
Travers DA, Waller AE, Bowling JM, Flowers D, Tintinalli J. Five-level triage system more effective than three-level in tertiary emergency department. J Emerg Nurs. 2002;28(5):395–400.
Korean Triage and Acuity Scale Committee. Korean triage and acuity scale manual version 1.0. 2nd ed. Seoul: Korean Society of Emergency Medicine KTAS Committee; 2021.
Kang DY, Cho KJ, Kwon O, Kwon JM, Jeon KH, Park H, et al. Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services. Scand J Trauma Resusc Emerg Med. 2020;28(1):17.
Lee JT, Hsieh CC, Lin CH, Lin YJ, Kao CY. Prediction of hospitalization using artificial intelligence for urgent patients in the emergency department. Sci Rep. 2021;11(1):19472.
Liu Q, Yang L, Peng Q. Artificial intelligence technology-based medical information processing and emergency first aid nursing management. Comp Math Methods Med. 2022;2022:8677118.
Davis S, Ju C, Marchandise P, Diagne M, Grant L. The effect of human supervision on an electronic implementation of the Canadian Triage Acuity Scale (CTAS). J Emerg Med. 2022;63(4):498–506.
Li RC, Asch SM, Shah NH. Developing a delivery science for artificial intelligence in healthcare. npj Digit Med. 2020;3:107.
Bass L, Clements P, Kazman R. Software architecture in practice. Boston, MA: Addison–Wesley; 2003.
Thomas E, Zaigham M, Ricardo P. Cloud computing concepts, technology & Architecture. Boston: Pearson; 2013.
Dehghani Soufi M, Samad-Soltani T, Shams Vahdati S, Rezaei-Hachesu P. Decision support system for triage management: A hybrid approach using rule-based reasoning and fuzzy logic. Int J Med Inform. 2018;114:35–44.
Ivanova M, Petkova P, Petkov N. Machine learning and Fuzzy logic in electronics: applying intelligence in practice. Electronics. 2021;10(22):2878.
Inokuchi R, Iwagami M, Sun Y, Sakamoto A, Tamiya N. Machine learning models predicting undertriage in telephone triage. Ann Med. 2022;54(1):2990–7.
Senda A, Endo A, Kinoshita T, Otomo Y. Development of practical triage methods for critical trauma patients: machine-learning algorithm for evaluating hybrid operation theatre entry of trauma patients (THETA). Eur J Trauma Emerg Surg. 2022;48(6):4755–60.
Shin D, Kam HJ, Jeon MS, Kim HY. Automatic classification of thyroid findings using static and contextualized Ensemble Natural Language processing Systems: Development Study [development study]. JMIR Med Inform. 2021;9(9):e30223.
Alanazi HO, Abdullah AH, Qureshi KN. A critical review for developing accurate and dynamic predictive models using machine learning methods in medicine and health care. J Med Syst. 2017;41(4):69.
van Smeden M, Penning de Vries BBL, Nab L, Groenwold RHH. Approaches to addressing missing values, measurement error, and confounding in epidemiologic studies. J Clin Epidemiol. 2021;131:89–100.
Wang X, Kattan MW. Cohort studies: design, analysis, and reporting. Chest. 2020;158(1S):72–8.
Miles J, Turner J, Jacques R, Williams J, Mason S. Using machine-learning risk prediction models to triage the acuity of undifferentiated patients entering the emergency care system: A systematic review. Diagn Progn Res. 2020;4:16.

A1: Aydin AG, Eray O, Sayrac AV, Oskay A, Ulusar UD. The reliability of an artificial intelligence tool, Decision Trees’in emergency medicine triage. Research Square. 2020;2020; doi: 10.21203/rs.3.rs-127447/v1.

A2: Cho A, Min IK, Hong S, Chung HS, Lee HS, Kim JH. Effect of applying a real-time medical record input assistance system with voice artificial intelligence on triage task performance in the emergency department: prospective interventional study. JMIR Med Inform. 2022;10(8):e39892.

A3: Cotte F, Mueller T, Gilbert S, Blümke B, Multmeier J, Hirsch MC et al. Safety of triage self-assessment using a symptom assessment app for walk-in patients in the emergency care setting: observational prospective cross-sectional study. JMIR mHealth uHealth. 2022;10(3):e32340.

A4: Farahmand S, Shabestari O, Pakrah M, Hossein-Nejad H, Arbab M, Bagheri-Hariri S. Artificial intelligence-based triage for patients with acute abdominal pain in emergency department; a diagnostic accuracy study. Adv J Emerg Med. 2017;1(1):e5.

A5: Karlafti E, Anagnostis A, Simou T, Kollatou AS, Paramythiotis D, Kaiafa G et al. Support systems of clinical decisions in the triage of the emergency department using artificial intelligence: the efficiency to support triage. Acta Med Lituanica. 2023;30(1):19-26.

A6: Kipourgos G, Tzenalis A, Diamantidou V, Koutsojannis C, Hatzilygeroudis I. An artificial intelligence based application for triage nurses in emergency department, using the emergency severity index protocol. Int J Caring Sci. 2022;15(3):1764-72.

A7: Leung KC, Lin YT, Hong DY, Tsai CL, Huang CH, Fu LC. A novel interpretable deep-learning-based system for triage prediction in the emergency department: A prospective study. 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Melbourne. 2021:2979-2985. doi: 10.1109/SMC52423.2021.9658729.

A8: Liu Y, Gao J, Liu J, Walline JH, Liu X, Zhang T et al. Development and validation of a practical machine-learning triage algorithm for the detection of patients in need of critical care in the emergency department. Sci Rep. 2021;11(1):24044.

No competing interests reported.

AITriageSRTable.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

The effects of triage applying artificial intelligence on triage in the emergency department: A systematic review of prospective studies

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Trial registration:

Figures

1. Background

2. Methods

2.1. Design

2.2. Inclusion and/or Exclusion Criteria

2.2.1. Inclusion Criteria

2.2.2. Exclusion Criteria

2.3. Study Search and Selection

2.3.1. Study Search

2.3.2. Study Selection

2.4. Quality Appraisal

2.5. Data Abstraction

3. Results

3.1. General Characteristics

3.2. Quality Appraisal

3.3. Characteristics of AI-Based Triage

3.4. Effects of AI-Based Triage

3.4.1. Model Performance

3.4.2. Triage Prediction

3.4.3. Other Results

4. Discussion

5. Conclusions

Abbreviations

Declarations

References

Selected 8 articles

Additional Declarations

Supplementary Files

Status:

Version 1