A non-inferiority trial comparing the accuracy of classifying urinary gram-stain findings between an artificial intelligence smartphone-based application and microbiology specialists

doi:10.21203/rs.3.rs-4284383/v1

Background

Gram staining results interpreted by microbiology specialists (MS) were compared to a developed computer-aided diagnosis system (CAD) using artificial intelligence for interpretation.

Methods

Using a non-inferiority study, CAD-predicted Gram stain results, generated from images of an iPhone camera in two hospitals, were compared to those of MS, between 1 April and 31 December 2022. The prediction accuracies, classified as Class 1 based on the bacterial morphology, were the primary endpoint. The datasets were created according to the separate hospitals; 10 MS interpreted 153 images from each hospital. CAD predicted 306 images overall.

Results

The accuracies (95% confidence intervals) of MS and CAD predictions were 83.0% (81.6–84.3) and 87.9% (83.7–91.3), respectively; with a difference of –4.93% (–8.43 to –0.62), indicating non-inferiority of CAD.

Conclusions

Non-inferiority of CAD to MS predictions was demonstrated; therefore, CAD urine Gram staining can be used to select empirical antibiotics in facilities without MS.

Artificial intelligence

Computer-aided diagnosis

Gram staining

Non-inferiority trial

Urinary tract infection

Prompt administration of antimicrobial agents is ideal after diagnosis because bacterial infection outcomes can be improved by appropriate empirical antimicrobial agents at an early stage [1,2]. However, traditional culture tests for bacteria require time to determine the causative organism. Multiplex nucleic acid amplification tests and matrix-assisted laser desorption/ionization time-of-flight mass spectrometry can shorten the turnaround time, contributing to antimicrobial stewardship with appropriate support [3]. Although new point-of-care tests for urinary tract infections (UTI) that directly use urine specimens are being developed [4,5], such new testing methods for initial treatment are unavailable now. Moreover, that novel testing devices are expensive to install and maintain is a barrier against expanding new testing methods, even with ground-breaking ones. Although the choice of empirical antibiotics is based on treatment guidelines, patient risk, and symptoms, point-of-care Gram staining (PCGS) is also a useful test for empirical antimicrobial selection.

Urine Gram staining contributes to UTI diagnosis [6,7] and empirical antimicrobial agent selection, reducing broad-spectrum antimicrobial agent usage and antimicrobial agent cost compared to guideline-based selection [8]. Urine specimens are relatively easy to collect, and specimen quality evaluation is not required; however, proficiency is required in reading Gram stain findings for accurate interpretation [9,10]. Facilities where clinical technologists can perform Gram staining of urine specimens after hours are rare, with a reported rate of 38% in Japan [11]. The accuracy of Gram stain decipherment between microbiology specialists (MS) and non-specialists showed significantly lower accuracy among non-specialists [12].

To put into practical use a tool to assist in the selection of an appropriate initial antimicrobial agent for urinary tract infection, with readings equivalent to MS in a situation such as Japan where 24-hour Gram stain decipherment by MS is not possible, we developed an inexpensive computer-aided diagnosis system (CAD) to speculate on bacterial classification (Table 1) in urine specimens by artificial intelligence (AI) and conducted a non-inferiority study to prove that CAD can decipher Gram stain findings to the same degree of accuracy as MS.

Study design

A retrospective observational study using residual clinical specimens from two tertiary care hospitals (the National Center for Global Health and Medicine [NCGM] and Kobe University Hospital [KUH]) was conducted. The primary analysis was a non-inferiority test of CAD versus 20 MS in the two hospitals. MS included physicians and clinical laboratory technicians who were infectious disease specialists certified by the Japanese Association of Infectious Diseases, and technologists in microbiology certified by the College of Laboratory Medicine in Japan or by Medical Technologists in Clinical Microbiology. Appendix S1 shows the algorithm for developing the CAD model.

Because residual clinical specimens were used retrospectively, the requirement of individual informed consent was waived by providing an opt-out opportunity. The research methods were collectively reviewed and approved by the Institutional Review Board of the NCGM (NCGM-S-004480-02).

Inclusion criteria and collected information

Gram-stained slides of urine specimens sampled between 1 April and 30 June 2022 at NCGM and 1 April and 31 December 2022 at KUH were used. Gram staining was performed at both sites using Bartholomew and Mittwer (B&M) methods. Clinical information including the sampling date, specimen type, Gram staining findings, and species identification results was collected.

After the routine examination, samples with both Gram staining findings and confirmed species identification results were included. Samples were excluded if a single organism was observed by microscopy, but more than one organism was identified by bacterial culture (e.g., Gram-negative rod [GNR] was observed by microscopy, but Staphylococcus aureus and Escherichia coli were isolated), or identified organisms differed (e.g., Gram-positive rod [GPR] was observed by microscopy but Escherichia coli was isolated). Note that the images used as training data are based on specimens collected between February 1, 2020, and March 31, 2023, except for April 1 through December 31, 2022, and did not overlap with these experimental images.

Table 2 shows the number of images captured according to the number of samples by class. For classifications with no more than three samples, spiked samples, which were generated using the ATCC standard strains (Supplementary Appendix S2), were added to make it at least three samples. In addition, 10 images were captured per sample from 15 to 50 samples randomly selected among samples with undetected bacteria during microscopy (labeled as ‘None’), and samples with two or more bacteria with different morphology (e.g., not applicable to cases where Gram-positive cocci [GPC] clusters and chains were mixed but applicable where GPC and Gram-negative cocci [GNC] were mixed), labeled as ‘Polymicrobial’. From these samples, three were randomly selected in each Class 2 category, and three image files per sample were included in the non-inferiority study (Appendix). The devices used for imaging and decipherment are presented in Appendix S3.

Outcome

The primary outcome, accuracy, was determined as the correct answer when the correct data matched the predicted data. As secondary outcomes, the macro-average sensitivity, recall, precision, and F1 score (harmonic mean of recall and precision) for Classes 1 and 2, respectively, were calculated. The agreement between correct and incorrect answers for MS and CAD was also evaluated as a secondary outcome.

Sample size calculation

The pilot study results showed accuracies of MS and CDA of 96.4% and 95.7%, respectively, in discriminating bacterial morphology (excluding ‘None’ and ‘Polymicrobial’ categories). Power was calculated for 306 images in the dataset, 306 predictions by CAD, and 3060 predictions by MS, analyzed based on a 5% non-inferiority margin and one-sided alpha error of 2.5%. The power was greater than 80% (approximately 83%).

Statistical analysis

Accuracy was expressed as point estimates with 95% confidence intervals. Non-inferiority was analyzed using Farrington and Manning analysis at a significance level of 5%. The agreement between correct and incorrect answers per image for each CAD and MS was evaluated using Cohen’s kappa coefficient and Gwet’s “first-order agreement coefficient” (AC1) statistics. Inter-rater reliability was evaluated using Fleiss’s kappa coefficient and Gwet’s AC1 statistics. R version 4.3.0 (R Foundation for Statistics. Computing, Vienna, Austria) was used for statistical analysis, and PASS 2023 (NCSS, LLC, Utah, USA) was used for sample size calculation.

Of the 1,388 and 1,869 eligible samples, 253 and 152 clinical samples were included from NCGM and KUH, respectively (Supplementary Table S4). Six spiked samples (three each of GNC and other GNR glucose non-fermenting bacteria) and 11 spiked samples (three GNC, three other GNR glucose non-fermenting bacteria, two Enterococcus faecium, one other GPC, and two Klebsiella oxytoca) were included in NCGM and KUH, respectively. The MS group included 9 physicians and 11 laboratory technicians.

Class 1

The overall accuracy for the 3,060 predictions by MS was 83.0% (81.6 to 84.3), and individual results ranged from 75.1% to 89.5%. The Fleiss’s kappa coefficient and AC1 statistic were 0.79 and 0.85 in NCGM and 0.68 and 0.77 in KUH, with relatively high inter-rater agreement in both sites. The confusion matrix for the MS is shown in Fig. 1a. Predictions were relatively accurate for GNR, yeast, and ‘None’ categories but a little less accurate when classifying polymicrobial, GNC, GPC, and GPR. The macro-average sensitivity was 79.1%. The results for other secondary outcomes are shown in Table 3. The overall accuracy for the 306 predictions by CAD was 87.9% (83.7 to 91.3), with a macro-average sensitivity of 79.1%. The confusion matrix of CAD is shown in Fig. 1b. Predictions were relatively accurate for GNR, GNC, GPC, yeast, and ‘None’ categories but a little less accurate when classifying polymicrobial and GPR. The accuracy difference was −4.93% (95% confidence interval [CI]: −8.43 to −0.62) (p=1.0), indicating non-inferiority of deciphering by CAD compared to MS, based on the Farrington and Manning analysis. In 2,323 predictions, both MS and CAD were correct but were incorrect in 256 predictions. Among the remaining images, 216 and 256 were correct only for MS and CAD, respectively. The agreement between correct and incorrect answers for MS and CAD was not very high, at a Cohen’s kappa coefficient of 0.42. However, the AC1 statistic showed a high agreement rate of 0.84.

Class 2

The accuracy for MS was 34.3% (32.6 to 36.0), ranging from 24.2% to 46.4% for individuals. The Fleiss’s kappa coefficient and AC1 statistic of 0.40 and 0.44 for NCGM and 0.31 and 0.37 for KUH indicated a low inter-rater agreement. The confusion matrix for the MS is shown in Fig. 2a. The predictions were often incorrect when classifying GPC (U-02–U-06) and GNR (U-08–U-14). In particular, the precision and recall of GNR tended to be low, ranging from 9.2% to 28.4% and 3.3% to 31.1%, respectively (Supplementary Table S5). The accuracy of CAD was 46.1% (40.4 to 51.8). The predictions were often incorrect when classifying GNRs, similar to MS, whereas these were relatively correct for GPCs, especially in the GPC cluster. The precision and recall of GNRs tended to be higher than those of MS, ranging from 14.3% to 40.6% and 5.6% to 72.2%, respectively (Supplementary Table S5). In particular, the F1 scores for glucose non-fermenting bacteria were 38.4% for MS and 60.3% for CAD (Supplementary Table S6).

The accuracy difference was –11.8% (95% CI: –17.6 to –6.04) (p=1.0), indicating non-inferiority of deciphering by CAD compared to MS, based on the Farrington and Manning analysis. However, the 95% CI demonstrated the superiority of CAD.

Both predictions by MS and CAD were correct with 749 images but incorrect with 1,350 predictions. Among the remaining images, 300 and 661 were correct only for MS and CAD, respectively. Compared to Class 1, the agreement on correct answers was lower for MS and CAD, with a Cohen’s kappa coefficient of 0.36 and an AC1 statistic of 0.69.

Although there have been attempts to use image AI analysis based on deep learning methods to assess Gram staining findings with blood cultures [13] and bacterial vaginosis [14], this is the first report to compare and validate Gram staining findings between CAD and MS using clinical urine specimens. In Class 1 differentiation, CAD performed better than MS with GPR and GNC but less with polymicrobial, based on F1 scores. The accuracy of the Gram stain readings by MS was 83%. The Gram staining error rate in discriminating bacterial morphologic features, verified in multiple institutions, was approximately 5% [15], and microbiologists in Japan are also surveyed annually to check the quality of their Gram stain interpretation by Japanese Association of Medical Technologists [Undisclosed data], in which the errors were below 5% most years. Nonetheless, MS performance in the present study was poor compared with that of this standard error rate. However, accuracy in discriminating the five bacterial classifications (GPC, GPR, GNR, GNC, and yeast) by 11 MS in urine specimens was 96.4%, the same as this error rate, in a pilot study [12]. Although the pilot study findings have been validated except those of polymicrobial and ‘None’ classifications, the prediction for polymicrobial by MS in this study was poor as well as that by CAD. In addition, the deciphering of Gram stain findings, normally performed using microscopy, was performed in this study using a single image file, for fairness, when assessing using CAD because of the no significant difference in accuracy between microscopy and single image assessment results in the pilot study [12]. However, this may have affected the prediction of polymicrobial classification using the single image rather than microscopy since the impact of the single-image judgment was verified in the pilot study. The accuracy of prediction for GPR was also poor with MS. Although a single specimen was used in the pilot study, the accuracy of predictions by both imaging and microscopy was low (data not shown). Corynebacterium spp. in urine specimens were rare compared to those in GNR and GPC. In this study, because samples were not selected to reflect their epidemiology, it is possible that this study’s higher GPR percentage than the proportion in the real world affected the overall accuracy. As reported, Corynebacterium spp. may form coccoids, depending on the culture condition [16,17], and such changes may occur more likely with urine. Although the reason for this is unclear, the use of a single-image judgment and the high rate of GPR, unlike actual epidemiology, may have contributed to the high error rate of classification of MS.

The impact of accurate Gram staining prediction on antimicrobial use remains unclear, but the results of Gram staining are known to influence empirical antimicrobial selection for common infectious diseases, such as community-acquired pneumonia [18]. A randomized controlled trial on ventilator-associated pneumonia showed that broad-spectrum antimicrobial use was reduced compared to empirical antimicrobial selection by guideline compliance, with no change in clinical efficacy [19]. The possibility that a narrower spectrum of antibiotics can be selected using PCGS has also been shown when antimicrobials were selected using either PCGS or guidelines in patients with infectious diseases, with UTI accounting for approximately half of cases [8]. In pediatric patients with UTI, PCGS showed high diagnostic performance and concordance with bacteria detected in culture, suggesting the possibility of a more efficient selection of susceptible agents [20]. In adult patients with UTI, Taniguchi et al. noted that PCGS may contribute to narrow-spectrum antimicrobial selection because antimicrobial agents selected based on PCGS results and information, such as antibiograms, are susceptible to many detected uropathogens [21]. Although the predictive value of antibiograms for appropriate empirical antimicrobials may not be high [22], antibiograms are one of the few pieces of information available for empirical treatment selection, and their use may contribute to reducing broad-spectrum antimicrobial usage.

As with the eligible specimens in Appendix Table S4, GNR is the main pathogen of urinary tract infection [23], and the F1 value for GNR of CAD is more than 95% and higher than that of MS. Therefore, the prerequisites for estimating the causative organism of urinary tract infection are satisfied. Differentiating bacterial species by Gram staining showed differentiation between S. aureus and other staphylococci in blood culture [24-26] and whether enterococci were penicillin-resistant [27]. Furthermore, the process of differentiating non-fermenting bacteria using the length and thickness ratio for GNR has been reported [28]. Although these studies showed relatively good performance, achieving sufficient accuracy when differentiating among various species remains challenging, as in this study, rather than within certain species. However, the Class 2 performance by MS was not high either. Although the differential performance of the GPC cluster was relatively good, the precision and recall of the main pathogens, such as GNRs, tended to be lower for MS than for CAD (Supplementary Table S5 and S6). The F1 score of non-fermenting bacteria was <40% in MS and >60% in CAD. Although the results of decipherment by both CAD and MS in this study did not look good, it is true that PCGS decipherment by medical professionals, including MS, in actual practice has reduced the use of broad-spectrum antimicrobial agents compared with guideline-based antimicrobial use [8]. Therefore, future clinical evaluation is needed, though, it is possible that the decipherment of PCGS CAD, which was not inferior to that of MS, can be used effectively in PCGS for choosing empiric antimicrobial therapy.

A limitation of this study is that its accuracy has not been verified at other centers, and generalizability remains an issue. Gram staining of urine specimens from facilities where training data were not collected was performed satisfactorily using a similar method and imaging device (data not shown). Although the included samples did not reflect the UTI epidemiological data, it is possible to generalize on bacterial species. This is because we estimated the top detected species in a previous single-center study [23] and national surveillance in Japan [29]. However, it is unclear whether CAD can be generalized to Gram staining methods other than the B&M methods and imaging data captured by smartphone devices other than the iPhone. Further studies may resolve this by increasing variations in the training data.

Predicting Gram staining findings in urine specimens by CAD is non-inferior to that by MS following B&M methods for staining samples, and images were captured using an iPhone camera. Although CAD could facilitate infectious disease treatment using PCGS for urine when MS is unavailable to interpret Gram staining findings, future studies are needed to determine its generalizability.

AC1 - Agreement Coefficient 1

AI - Artificial Intelligence

ATCC - American Type Culture Collection

B&M - Bartholomew and Mittwer

CAD - Computer-Aided Diagnosis

CI - Confidence Interval

F1 Score - F1 Score (harmonic mean of recall and precision)

GNC - Gram-Negative Cocci

GNR - Gram-Negative Rod

GPC - Gram-Positive Cocci

GPR - Gram-Positive Rod

KUH - Kobe University Hospital

MS - Microbiology Specialists

NCGM - National Center for Global Health and Medicine

PCGS - Point-of-Care Gram Staining

UTI - Urinary Tract Infection

Ethics approval and consent to participate

The research methods were collectively reviewed and approved by the Institutional Review Board of the NCGM (NCGM-S-004480-02).

Consent for publication

Because residual clinical specimens were used retrospectively, the requirement of individual informed consent was waived by providing an opt-out opportunity.

Availability of data and materials

All data associated with this study are presented in the main text or appendix and are available from the authors upon reasonable request. Individual-level, de-identified participant data are not available for publication.

Competing interests

K.Y. received research grants from Fujirebio Inc., Mizuho Medy Co. Ltd., VisGene Co. Ltd., Canon Medical Systems Corp., Sanyo Chemical Industries Ltd., CarbGeM Inc., Sysmex Corp., and Toyobo Co. Ltd. outside the submitted work and has patent rights to this CAD, but no income has been generated from it. I.M. and S.M. are employees of CarbGeM Inc.

Funding

This work was supported by AMED (grant number JP23hk0102076h0003).

Authors' contributions

Conceptualization: KY, GO, KFE, YU, and NO

Data curation: KY, AM, M. Kurokawa, GO, KFE, KO, M. Kusuki, SM, and IM

Formal analysis: KY, SM, and YU

Funding acquisition: KY, GO, and IM

Investigation: KY, GO, KFE, SM, and IM

Methodology: KY, AM, GO, KFE, and IM

Project administration: KY, GO, IM, and NO

Resources: AM, M. Kurokawa, KO, M. Kusuki, SM, and IM

Software: SM and IM

Supervision: NO

Visualization: KY and SM

Writing – original draft: KY

Writing – review & editing: GO, IM, KFE, AM, SM, HN, M. Kurokawa, KO, M. Kusuki, YU, and NO

Acknowledgements

The authors express their deepest gratitude to Masami Yoshino for managing the imaging data used in this study. We thank Kayoko Hayakawa, Taketomo Maruki, Yutaro Akiyama, Yuki Moriyama, Ayano Motohashi, Kazuhisa Mezaki, Saeko Kinpara, Taiji Koyama, Kenta Iijima, Saori Kobayashi, Sayaka Matsumoto, Masako Nishida, and Nami Ishida for their help in deciphering the Gram staining. We also thank the staff, Yoshihiko Hirayama, Mayumi Kawakami, Motoko Ishida, Noriko Iwamoto, Gen Yamada, Satoshi Ide, Ayako Okuhama, Akiho Sugita, Nozomu Tsurumaki, Sei Ueda, Wataru Ochi, Reo Iguma, Kaito Mimura, Ryo Sakuma, Mayu Hiraiwa, Shunsuke Matsuoka, Akiko Hamana, Noriko Tomita, and Moto Kimura, at NCGM and KUH for their cooperation. We thank Editage (www.editage.jp) for the English language editing.

Im Y, Kang D, Ko RE, Lee YJ, Lim SY, Park S, et al. Time-to-antibiotics and clinical outcomes in patients with sepsis and septic shock: a prospective nationwide multicenter cohort study. Crit Care. 2022;26(1):19. doi:10.1186/s13054-021-03883-0.
Ohnuma T, Chihara S, Costin B, Treggiari MM, Bartz RR, Raghunathan K et al. Association of appropriate empirical antimicrobial therapy with in-hospital mortality in patients with bloodstream infections in the US. JAMA Netw Open. 2023;6(1):e2249353. doi:10.1001/jamanetworkopen.2022.49353.
Timbrook TT, Morton JB, McConeghy KW, Caffrey AR, Mylonakis E, LaPlante KL. The effect of molecular rapid diagnostic testing on clinical outcomes in bloodstream infections: A systematic review and meta-analysis. Clin Infect Dis. 2017;64(1):15-23. doi:10.1093/cid/ciw649.
Baltekin Ö, Boucharin A, Tano E, Andersson DI, Elf J. Antibiotic susceptibility testing in less than 30 min using direct single-cell imaging. Proc Natl Acad Sci U S A. 2017;114(34):9170-5. doi:10.1073/pnas.1708558114.
Arienzo A, Murgia L, Cellitti V, Ferrante V, Stalio O, Losito F, et al. A new point-of-care test for the rapid antimicrobial susceptibility assessment of uropathogens. PLOS ONE. 2023;18(7):e0284746. doi:10.1371/journal.pone.0284746.
Wiwanitkit V, Udomsantisuk N, Boonchalermvichian C. Diagnostic value and cost utility analysis for urine Gram stain and urine microscopic examination as screening tests for urinary tract infection. Urol Res. 2005;33(3):220-2. doi:10.1007/s00240-004-0457-z.
Williams GJ, Macaskill P, Chan SF, Turner RM, Hodson E, Craig JC. Absolute and relative accuracy of rapid urine tests for urinary tract infection in children: a meta-analysis. Lancet Infect Dis. 2010;10(4):240-50. doi:10.1016/S1473-3099(10)70031-1.
Taniguchi T, Tsuha S, Shiiki S, Narita M. Gram-stain-based antimicrobial selection reduces cost and overuse compared with Japanese guidelines. BMC Infect Dis. 2015;15:458. doi:10.1186/s12879-015-1203-6.
Guarner J, Street C, Matlock M, Cole L, Brierre F. Improving Gram stain proficiency in hospital and satellite laboratories that do not have microbiology. Clin Chem Lab Med. 2017;55(3):458-61. doi:10.1515/cclm-2016-0556.
Munson E, Block T, Basile J, Hryciuk JE, Schell RF. Mechanisms to assess Gram stain interpretation proficiency of technologists at satellite laboratories. J Clin Microbiol. 2007;45(11):3754-8. doi:10.1128/JCM.01632-07.
Shime N, Yanagihara K, Watanabe M, Morita M, Sasaki M, Shinagawa M, et al. A multicenter survey for microbiological testing in emergency department. Journal of Japan Society for Clinical Microbiology. 2019;29:28-31. [Japanese].
Ikegaki S, Ohji G, Yamamoto K, Ohnuma K, Ebisawa K, Kusuki M, et al. [O-149: A study of differences in prediction accuracy in urine Gram stain decipherment by decipherment method and rater]. The 97th Annual Meeting of the Japanese Association for Infectious Diseases. Yokohama; 2023/4/28. [Japanese, no abstract].
Smith KP, Kang AD, Kirby JE. Automated interpretation of blood culture Gram stains by use of a deep convolutional neural network. J Clin Microbiol. 2018;56(3):e01521-17. doi:10.1128/JCM.01521-17.
Wang Z, Zhang L, Zhao M, Wang Y, Bai H, Wang Y, et al. Deep neural networks offer morphologic classification and diagnosis of bacterial vaginosis. J Clin Microbiol. 2021;59(2):e02236-20. doi:10.1128/JCM.02236-20.
Samuel LP, Balada-Llasat JM, Harrington A, Cavagnolo R. Multicenter assessment of Gram stain error rates. J Clin Microbiol. 2016;54(6):1442-7. doi:10.1128/JCM.03066-15.
Ichikawa L, Ishigaki S, Matsumura M, Asahara M. Case study of morphological changes of Corynebacterium species to gram-positive cocci. Jpn J Technol. 2018;67:299-306. doi:10.14932/jamt.17-143.
Grubb TC. Coccus forms of Corynebacterium diphtheriae. J Infect Dis. 1935;56(1):64-77. doi:10.1093/infdis/56.1.64.
Pendergrast J, Marrie T. Reasons for choice of antibiotic for the empirical treatment of CAP by Canadian infectious disease physicians. Can J Infect Dis. 1999;10(5):337-45. doi:10.1155/1999/928438.
Yoshimura J, Yamakawa K, Ohta Y, Nakamura K, Hashimoto H, Kawada M, et al. Effect of Gram stain-guided initial antibiotic therapy on clinical response in patients with ventilator-associated pneumonia: the GRACE-VAP randomized clinical trial. JAMA Netw Open. 2022;5(4):e226136. doi:10.1001/jamanetworkopen.2022.6136.
Yodoshi T, Matsushima M, Taniguchi T, Kinjo S. Utility of point-of-care Gram stain by physicians for urinary tract infection in children ≤36 months. Med (Baltim). 2019;98(14):e15101. doi:10.1097/MD.0000000000015101.
Taniguchi T, Tsuha S, Shiiki S, Narita M. Point-of-care urine Gram stain led to narrower-spectrum antimicrobial selection for febrile urinary tract infection in adolescents and adults. BMC Infect Dis. 2022;22(1):198. doi:10.1186/s12879-022-07194-9.
Hasegawa S, Livorsi DJ, Perencevich EN, Church JN, Goto M. Diagnostic accuracy of hospital antibiograms in predicting the risk of antimicrobial resistance in Enterobacteriaceae isolates: A nationwide multicenter evaluation at the Veterans Health Administration. Clin Infect Dis. 2023;77(11):1492-500. doi:10.1093/cid/ciad467.
Shigemura K, Tanaka K, Okada H, Nakano Y, Kinoshita S, Gotoh A et al. Pathogen occurrence and antimicrobial susceptibility of urinary tract infection cases during a 20-year period (1983-2002) at a single institution in Japan. Jpn J Infect Dis. 2005;58(5):303-8. doi:10.7883/yoken.JJID.2005.303.
Murdoch DR, Greenlees RL. Rapid identification of Staphylococcus aureus from BacT/ALERT blood culture bottles by direct Gram stain characteristics. J Clin Pathol. 2004;57(2):199-201. doi:10.1136/jcp.2003.10538.
Hadano Y, Isoda M, Ishibashi K, Kakuma T. Validation of blood culture Gram staining for the detection of Staphylococcus aureus by the ’oozing sign’ surrounding clustered gram-positive cocci: a prospective observational study. BMC Infect Dis. 2018;18(1):490. doi:10.1186/s12879-018-3412-2.
Kondo S, Yamada T, Misawa S, Nakamura A, Oguri T. [Morphological identification of Staphylococcus spp. from blood culture bottles based on direct Gram staining]. Kansenshogaku Zasshi. 2008;82(6):656-7. doi:10.11150/kansenshogakuzasshi1970.82.656. [Japanese].
Hayashi T, Yoshida M. The Peanuts Sign: a Novel Clue to the Rapid Discrimination of Penicillin-susceptible Enterococci Based on Gram Staining Findings. Kansenshogaku Zasshi. 2019;93(3):306-11. doi:10.11150/kansenshogakuzasshi.93.306. [Japanese].
Bartlett RC, Mazens-Sullivan MF, Lerer TJ. Differentiation of Enterobacteriaceae, Pseudomonas aeruginosa, and Bacteroides and Haemophilus species in Gram-stained direct smears. Diagn Microbiol Infect Dis. 1991;14(3):195-201. doi:10.1016/0732-8893(91)90032-b.
Ministry of Health, Labour and Welfare, Japan. Japan nosocomial infections surveillance (JANIS) [cited Nov 21 2023]. Available from: https://janis.mhlw.go.jp/english/index.asp.

Table 1. Gram staining classification

Code	Class 1	Class 2
U-01	Yeast	Candida spp.
U-02	GPC	GPC cluster
U-03	GPC	Enterococcus faecalis
U-04	GPC	Enterococcus faecium
U-05	GPC	Streptococcus agalactiae
U-06	GPC	Other GPC
U-07	GPR	Corynebacterium spp.
U-08	GNR	Enterobacter cloacae
U-09	GNR	Escherichia coli
U-10	GNR	Klebsiella oxytoca
U-11	GNR	Klebsiella pneumoniae
U-12	GNR	Other GNR Enterobacteriaceae
U-13	GNR	Pseudomonas aeruginosa
U-14	GNR	Other GNR glucose non-fermenting bacteria
U-15	GNC	GNC
Poly	Polymicrobial	Polymicrobial
None	None	None
GPC, Gram-positive cocci; GPR, Gram-positive rod; GNC, Gram-negative cocci; GNR, Gram-negative rod.

Table 2. Number of target specimens extracted, and images taken

Actual number of samples from each institute	Number of samples included	Number of images per sample
0–4	3^a	12
5–9	5	7
10–14	10	4
15–19	15	3
≥20	20	2
^aWhen the available specimens were less than three, to incorporate three specimens, the specimens were supplemented with spiked specimens.

Table 3. Precision, recall, and F1 score (harmonic mean) for Class 1

	CAD			Microbiology specialist
	Precision	Recall	F1	Precision	Recall	F1
None	62.1%	100.0%	76.6%	55.8%	96.7%	70.7%
Polymicrobial	44.4%	44.4%	44.4%	62.5%	50.0%	55.6%
GPC	93.2%	91.1%	92.1%	87.3%	85.6%	86.4%
GPR	75.0%	66.7%	70.6%	75.7%	45.0%	56.4%
GNR	95.3%	96.0%	95.7%	89.9%	95.8%	92.7%
GNC	100.0%	61.1%	75.9%	60.6%	35.0%	44.4%
Yeast	100.0%	94.4%	97.1%	91.7%	85.6%	88.5%
CAD, computer-aided diagnosis system; GPC, Gram-positive cocci; GPR, Gram-positive rod; GNC, Gram-negative cocci; GNR, Gram-negative rod; F1, harmonic mean of precision and recall.

Competing interest reported. K.Y. received research grants from Fujirebio Inc., Mizuho Medy Co. Ltd., VisGene Co. Ltd., Canon Medical Systems Corp., Sanyo Chemical Industries Ltd., CarbGeM Inc., Sysmex Corp., and Toyobo Co. Ltd. outside the submitted work and has patent rights to this CAD, but no income has been generated from it. I.M. and S.M. are employees of CarbGeM Inc.

A non-inferiority trial comparing the accuracy of classifying urinary gram-stain findings between an artificial intelligence smartphone-based application and microbiology specialists

Status:

Version 1

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1