The majority of NEs (62.32%) occurred in six main departments: General Surgery, 19 (18.81%); Gynecology, 17 (16.83%); Orthopedics, 16 (15.84%); Cardiac and Cardiothoracic 15 (14.85%); Ophthalmology 8 (7.92%); and Urology, 7 (6.93%) (Table 1). Therefore, our analysis focused on the occurrence of NEs in these six departments.
Table 1
Characteristics of the data set according to surgical specialty
Observations n=9234
|
Never Events
n=101
|
Phase
Specialty
|
*Pre-procedure
(n=1,539)
(missing data on 760 cases)
|
Sign in
(n=1,504)
|
Time out
(n=1,498)
|
First count
(n=1,518)
|
Second count
(n=1,501)
|
Third count
(n=1,498)
|
|
Urology
|
72
|
156
|
148
|
124
|
118
|
124
|
7 (6.93%)
|
Orthopedics
|
185
|
331
|
324
|
341
|
302
|
326
|
16 (15.84%)
|
ENT
|
64
|
105
|
105
|
99
|
102
|
93
|
3 (2.97%)
|
Gynecology
|
63
|
143
|
139
|
149
|
153
|
153
|
17 (16.83%)
|
General surgery
|
313
|
537
|
558
|
576
|
623
|
604
|
19 (18.81%)
|
Plastic surgery
|
22
|
39
|
37
|
40
|
36
|
42
|
2 (1.98%)
|
Vascular surgery
|
18
|
45
|
42
|
45
|
42
|
43
|
5 (4.95%)
|
Neurosurgery
|
7
|
25
|
19
|
22
|
19
|
19
|
5 (4.95%)
|
Dermatology
|
7
|
16
|
26
|
21
|
22
|
24
|
2 (1.98%)
|
Ophthalmology
|
12
|
41
|
34
|
33
|
19
|
18
|
8 (7.92%)
|
Maxillofacial
|
3
|
12
|
10
|
8
|
10
|
11
|
2 (1.98%)
|
Cardiac and Cardiothoracic
|
13
|
54
|
56
|
60
|
55
|
41
|
15 (14.85%)
|
Table 2
Characteristics of patients and surgery in the dataset
Characteristic
|
Observations
|
Never Events
|
Average age
|
50.8 years (SD 20.4)
|
46
|
Gender
|
Male (n=388 (49.8%)), Female (n=391 (50.2%))
|
Male (n=46 (45.5%))
Female n=55 (54.5%)
|
Length of surgery
|
Up to 1 hour: 2124 (23%)
1–2 hours: 4340 (47%)
3–4 hours: 2031 (22%)
Over 4 hours: 739 (8%)
|
Length of surgery:
Up to 1 hour: 54 (53.5%)
1–2 hours: 13 (12.9%)
3–4 hours: 17 (16.8%)
Over 4 hours: 17 (16.8%)
|
In order to evaluate our models, we adopted the Area Under the Curve (AUC) measure which is especially suited for imbalanced data, as in our case in this study, since it does not have any bias toward models that perform well on the minority of majority classes in the expense of the other. [26] Our three RF models demonstrated good performance, exhibiting an Area Under the Curve (AUC) between 0.81 and 0.85. Generally, AUC scores between 0.8 to 0.9 are considered excellent. [27]. AUC is interpreted as the probability that our model will rank a randomly chosen positive instance higher than a randomly chosen negative one. [28] As such, our models can be considered relatively strong and accurate despite their limitations.
Feature Importance
Figure 1 presents the top contributing features to the occurrence of NEs (of both types combined) in the six departments along with the associated probability change.
The top 14 contributing features varied significantly across departments, and there was no single feature set which was consistently more informative across all operations in predicting NEs. For example, feature [C], Discrepancy in second count, varied significantly across departments (160% to 1,950%). Feature [B], Surgery is paused because of discrepancy in third count, appeared in four of the six departments, and the associated probability change varied dramatically as well, between 269% and 1,540%. There were 10 features that consistently decreased the chance of an NE, including [F]; Surgeon scans the cavity/fascia before closure during the second count, which affected five out of six departments, which was consistent in its probability change between 65–100%. Features [I], [J], [ K], [L], [M], and [N] decreased the chances of NEs between 2–100% in three departments. Three features, [A], Discrepancy in absorbing materials, [E], Surgery time > 4 hours, and [G], Surgery time < 1 hour appeared just once across departments, with a medium impact on NE occurrence.
Analysis of the results per department shows a variation among contributing features. For example, in Ophthalmology, the probability was consistently -100% in five features, while in General Surgery, two features that increased the probability of an error varied between 1,168–1,283%: features [B] Surgery is paused because of discrepancy in third count; and [C] Discrepancy in second count. In Orthopedics, those same two features, [B] and [C], increased the probability of error (1,540–1,950%). Three features decreased the probability of error: [F] Surgeon scans the cavity/fascia before closure; [H] Second count is performed before closure of fascia/cavity; and (I) Procedure type is compared to the one written in patient's file, -65 to -87%.
Effects of Feature Combinations
In the following analysis (Figure 2), we examine the effects of paired features, i.e., features that occur together in the data. It is important to note that, when considering feature combinations, their occurrence is expected to be very low especially in the NEs class. As such, the estimated effects are likely to be very high, yet their confidence is significantly low.
Interestingly, in General Surgery, there were 14 feature combinations that caused a probability change of 13,600% (Figure 2A). In comparison, the single feature analysis (Figure 1) revealed a probability change of 1,287% and 1,168%, surprisingly by two features that were not part of the 14 feature combinations identified here.
In Figure 2A (Gynecology), the effect of every feature combination is associated with a probability change of 1,000–2,000%. In the single feature analysis (Table 2), the effect of two of the features separately was <900%, and the rest lagged behind with <150%. In Urology (Figure 2B), results show there were dozens of pairs with an effect of 1,900–2,500%, while the effect of a single feature had <1150% effect on error. In General Surgery (Figure 2E), the accumulated effect of two features together showed a dozen pairs with an effect of 1,900–4,200%, while the effect of a single feature had an <1,950% indication on error, and the rest even lower percentages.
Features Affecting Types A and B
Turning to Models 2 and 3, there is an overlap in three of the top five contributing features to Types A and B errors (Figures 3 and 4): 1) the presence of two nurses during the surgery predicts a greater occurrence of Type A (66%) and Type B (88%); 2) an operation < 1 hour had a greater occurrence of Type A (122%), and Type B (87%); and 3) when the operation lasted between one to two hours, both Types A and B were less frequent, decreasing by 60% and 74%, respectively. The surgical department that was most affected regarding the occurrence of Type A NEs was Ophthalmology, with a prevalence of 504%, while General Surgery was associated with a decrease of 63% in Type A (Figure 3). For Type B, the two remaining features were staff driven; the feature “more than three physicians” was associated with an increased prevalence of Type B (151%), while “two physicians” was associated with a decreased prevalence of 52% with Type B (Figure 4).