Overall in this study we found that the scoring systems and pathways assessed performed poorly, with low to moderate sensitivity and overall poor specificity. The positive predictive values were suboptimal in identifying bacteraemic patients presenting to the ED.
This study demonstrated that the modified Shapiro criterion was the most sensitive method of detecting true bacteraemia. However, our data suggests that this method is not sensitive enough as a stand-alone tool to predict bacteraemia, with almost 8% of patients whose bacteraemia would be missed using this scoring system alone. Our calculated sensitivity is lower than previous data, however has similar specificity. [12, 17, 18] Larger studies are therefore required to demonstrated reproducibility in these findings. Practical application of Shapiro in real-world settings should also be considered. Four of the nine minor criteria are obtained from haematological or biochemical blood analysis, meaning that the calculation of the score is not available at initial point of care. Waiting for these results may lead to delays in taking blood cultures and administering antibiotics in septic Bacteraemic patients or result in blood cultures being taken after antibiotic administration, significantly reducing their yield. Prospective assessment of the score in real-time, with outcomes including delay to antibiotics and taking blood cultures should be examined.
The CEC SEPSIS KILLS pathway demonstrated the most balanced sensitivity and specificity in detecting bacteraemia with moderate accuracy. In initial development and application of the pathway, there was an emphasis on between the flags (BTF) vital sign criteria and serum lactate level only in the assessment of a patient with possible sepsis. However in recent years there has been the addition of other criteria including clinician clinical concern of sepsis. As the pathway now incorporates both vital signs and clinical assessment by a senior physician in determining the presence or absence of sepsis, it raises the possibility of inter-hospital variability of the tool due to variability in clinical experience. Another single centre NSW based study demonstrated low sensitivity (45.1%) and moderate specificity (78.9%) when clinical impression was not incorporated.[19] However other studies have shown an overestimation in bacteraemia prediction when based on clinical judgement alone.[20] Our study did not analyse whether the individual vital sign criteria was responsible for the activation of the SEPSIS KILLS pathway, or individual senior clinician suspicion, therefore we feel that a multisite assessment into this may be warranted to determine the performance of the tool with vital sign criteria alone. If the vital sign criteria alone were associated with reasonable sensitivity and specificity, we could feel more confident in this tool being used by clinicians of all skill levels.
While CEC SEPSIS KILLS criteria performed reasonably overall, the associated SEPSIS KILLS FirstNet tool was rarely applied in either groups, with an activation rate of 14% in our patients with bacteraemia. We speculate that poor uptake by clinical staff was potentially due to inadequate of knowledge of the existence of the tool, the lack of clinical criteria provided in the CEC package in the recommendation for its application, inexperience and insufficient education of staff activating the tool and under-recognition of sepsis due to heterogeneity of presenting symptoms and its evolving pathophysiology. It is also clear that sepsis is an evolving process, so a single point of assessment at a variable point in the pathophysiological process is unlikely to be clinically reliable. We found that Bacteraemic patients were correctly identified more often than Non Bacteraemic patients, which overall is of questionable significance given the overall poor rates of use of the tool. Identification of patients at risk of sepsis with the icon also did not lead to earlier administration of antibiotics. We recommend further evaluation of the uptake of the SEPSIS KILLS in other NSW EDs and include assessment of the rates of its use and clinical impact, including whether time to antibiotic therapy and fluid resuscitation were optimised. Targeted education programs could then be developed to inform frontline ED staff about the recognition of sepsis using these evaluations. The small numbers in our study suggest that antibiotic time did not improve with activation of the SEPSIS KILLS, and identification and activation of the pathway was not associated with improved mortality.
The SIRS criteria have underpinned the principles of detection of bacteraemia in early sepsis recognition pathways that include CEC SEPSIS KILLS. Previous evaluations have demonstrated the tool to be sensitive but poorly specific in the detection of septic patients including those with bacteraemia.[21] However in our population SIRS has demonstrated moderate sensitivity and poor specificity, performing slightly better than in other contemporary studies in this setting.[9, 10] This has become an increasingly recognised phenomenon, which again highlights the diversity of underlying pathophysiology underlying patients with bacteraemia, and can range from gross biochemical and clinical disturbance to minor changes in physiology depending on the time of presentation, underlying comorbidities and pathogen characteristics. SIRS also historically underperformed against more recently developed scoring systems, including qSOFA which has led to its removal from the current SEPSIS-3 definition.[22] Lastly, we found that the criteria could not identify patients at risk of death. This study therefore confirms its lack of value as a clinical prediction tool in sepsis and bacteraemia.
qSOFA was designed as a tool designed to assess severity of illness in sepsis, and initial comparative literature indicated its superiority over its predecessor SIRS in the prediction of mortality and ICU admission[23]. It has however been evaluated as a clinical prediction tool for sepsis and bacteraemia with poor results due to modest sensitivity and specificity[21]. This study confirmed poor sensitivity in clinical diagnosis of sepsis, which we concede is probably driven by its design as an illness severity score rather than a clinical decision making score. The tool was able to predict mortality when it was activated, however it identified few patients in our cohort with bacteraemia and therefore overall was not useful in predicting mortality in this group overall. Similar conclusions have been met, highlighting the inability of qSOFA in identifying mortality risk or likelihood of requiring ICU admission in patients who had severe sepsis, suggesting the need for re-evaluation of whether this tool is clinically useful.[14].
Limitations
In the design of our study we only controlled our population for age, not other comorbidities which could have significant impact on morbidity and mortality outcomes. Despite patients being randomly selected from a cohort of blood cultures over 2 years, there remains the possibility of selection bias. In addition, our patient population is limited to patients from two moderate sized hospitals, which raises the possibility of reproducibility at other centres. Therefore further studies with a more diverse population are encouraged.
For the benefit of analysis, we used the assumption that patients in the Non Bacteraemic group never had an episode of bacteraemia. However, we understand that bacteraemia can be episodic, and that taking a blood culture at a single point in time may lead to a false negative result. There are also technical microbiological issues which may yield a truly bacteraemic patient negative, including low volumes of blood inoculated and the presence of organisms that are unable to be cultured.
In the calculation of scores, we used vital signs and clinical information collected within a four-hour period of triage. However, the scoring systems and clinical pathways assessed are designed to be used in real time, therefore this methodology may have led to higher sensitivity scores for the tools then if used as designed. We felt that this was more representative of a true clinical scenario, where there is constant clinical reassessment, especially in a process such as sepsis and bacteraemia where the underlying pathophysiology evolves throughout time.
Lastly this study has not incorporated all current scoring systems in use, such as the New Early Warning score, which has been demonstrated to accurately predict sepsis and mortality when used in conjunction with qSOFA in certain populations. Further studies analysing the sensitivity and specificity of scoring systems when used in combination may demonstrate an improved performance.