Successful creation of clinical AI without data scientists or software developers: radiologist-trained AI model for identifying suboptimal chest-radiographs

doi:10.21203/rs.3.rs-1570309/v1

Download PDF

Article

Successful creation of clinical AI without data scientists or software developers: radiologist-trained AI model for identifying suboptimal chest-radiographs

https://doi.org/10.21203/rs.3.rs-1570309/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Objectives: Suboptimal chest radiographs(CXR) can limit interpretation of critical findings. Radiologist-trained AI models were evaluated for differentiating suboptimal(sCXR) and optimal(oCXR) chest radiographs.

Methods: Our IRB-approved study included 3278 CXRs from adult patients (mean age 55 ± 20 years) identified from a retrospective search of CXR in radiology reports from 5 sites. A chest radiologist reviewed all CXRs for the cause of suboptimality. The de-identified CXRs were uploaded into an AI server application for training and testing 5 AI models. The training set consisted of 2202 CXRs(n= 807 oCXR; n= 1395 sCXR) while 1076 CXRs(n=729 sCXR; n=347 oCXR) were used for testing. Data were analyzed with the Area under the curve(AUC) for the model's ability to classify oCXR and sCXR correctly.

Results: For the two-class classification into sCXR or oCXR from all sites, for CXR with missing anatomy, AI had sensitivity, specificity, accuracy, and AUC of 78%, 95%, 91%, 0.87(95% CI 0.82-0.92), respectively. AI identified obscured thoracic anatomy with 91% sensitivity, 97% specificity, 95% accuracy, and 0.94 AUC (95% CI 0.90-0.97). Inadequate exposure with 90% sensitivity, 93% specificity, 92% accuracy, and AUC of 0.91 (95% CI 0.88-0.95). The presence of low lung volume was identified with 96% sensitivity, 92% specificity, 93% accuracy, and 0.94 AUC(95% CI 0.92-0.96). The sensitivity, specificity, accuracy, and AUC of AI in identifying patient rotation were 92%, 96%, 95%, and 0.94 (95% CI 0.91-0.98), respectively.

Conclusion: The radiologist-trained AI models can accurately classify optimal and suboptimal CXRs. Such AI models at the front end of radiographic equipment can enable radiographers to repeat sCXRs when necessary.

Artificial Intelligence

Computer-assisted Image processing

Quality improvement

Radiography

Thoracic Radiography

Chest radiographs (CXR) represent the most important and frequent imaging test accounting for almost 40% of all imaging examinations due to ease of access, feasibility, low cost, and moderate sensitivity in diagnosing pulmonary, mediastinal, and cardiac pathology ^1,2. Yet, there is tremendous inter-radiologist variability in the interpretation of CXRs. Higher quality images could result in more consistent and reliable interpretations from projection radiography. However, suboptimal quality CXR are extremely common and multifactorial, including those related to radiographers, radiographic equipment, as well as patient morphology and condition ^3,4.

Suboptimal CXR from improper positioning and inadequate coverage of the entire region of interest could be related to the radiographer's lack of attention to details or the patient's complex anatomy or critical condition. Such causes can lead to obscuration, non-inclusion, or variations in appearance of lungs or mediastinal abnormalities. Variations in exposure related to radiographic equipment, improper exposure factors, and patient body habitus can lead to over-or under-exposed CXR and result in suboptimal evaluation of key findings. Low inspiratory effort on CXR ranks high among the cause of suboptimal CXR, specifically in portable settings at the patient's bedside. The resulting low lung volumes can obscure a large portion of the lungs and make it difficult to assess cardio-mediastinal structures. Overlying devices and foreign bodies as well as overlapping from anatomic structures such as arms and chin, also limit evaluability of CXR. Besides these, radiographic artifacts due to equipment or patient movement contribute to suboptimal CXR as well ⁵.

Although suboptimality is more frequent in portable, anteroposterior projection CXR than in ambulatory, posteroanterior CXR ⁶, suboptimal CXR from either format can lead to incomplete evaluation as well as false positive and false negative findings. Besides these effects, suboptimal CXR can result in reacquisition of CXR, workflow issues, additional radiation exposure, and delays in diagnosis and patient care. Hence, quality control (QC) is crucial in diagnostic radiography. While technical aspects of QC are straightforward with established acceptable operational limits on electrical, mechanical, and radiation outputs, the imaging QC, which includes optimum acquisition parameters and good imaging practices, are subjective and labor-intensive ⁷. With growing applications of artificial intelligence (AI) in triage, segmentation, and detection of abnormalities on imaging tests, including CXR ^2,8,9, we hypothesized that an AI model could also help in accurate, efficient, and rapid identification of suboptimal CXR. The purpose of our study was to train and evaluate the performance of a radiologist-trained AI model for differentiating suboptimal and optimal chest radiographs.

Our institutional review board, Mass General Brigham IRB, approved retrospective study was performed with a waiver of informed consent. The study was compliant with the Health Insurance Portability and Accountability Act (HIPAA) and carried out in accordance with relevant guidelines and regulations. Our institution received a research grant from Cognex Inc. **** received unrelated research grants from Siemens Healthineers, Coreline Inc., and Riverain Tech. Other co-authors have no pertinent financial disclosures.

Study Design and Data Definitions

The described study methodology is in accordance with the CLAIM structure for describing AI algorithms ¹⁰. Our study included adult patients (age > 18 years) who had CXR at five independent community and quaternary hospitals (Brigham and Women's Hospital, Massachusetts General Hospital, Cooley Dickinson Hospital, North Shore Medical Centre, Newton Wellesley Hospital) affiliated with a common healthcare system. We used a radiology report search engine (Nuance mPower) to search for CXR from 5 different hospitals with keywords "suboptimal," "overlying chin," "technically inadequate," "limited," and "incomplete evaluation" for radiological exam, and "Chest X-ray" for both portable and posteroanterior projection radiographs. The presence of frontal CXR (portable or posteroanterior) and any of the keywords represented the inclusion criteria. Exclusion criteria were age < 18 years and lack of frontal CXR. For optimal CXR, absence of the above-mentioned terms (negative search) was used with the same exam type to identify consecutive CXR from 5 sites. The search was limited to CXR performed between January 2015 to June 2021.

After the data collection step, we blinded the identity of individual hospitals to protect patient and hospital privacy. Site-wise numbers of CXR were as follows: Site A (n = 781), Site B (n = 987), Site C (n = 654), Site D (n = 495), and Site E (n = 361). For each CXR, we extracted the radiology report and the corresponding CXR. From the report text, we recorded any mention of suboptimality of CXR and specific reasons such as inadequate exposure, patient rotation, overlapping anatomy, and missing anatomic coverage.

Data Partitioning

Both optimal and suboptimal CXR were included regardless of patient sex, race, radiographic equipment vendor, and projection direction [either postero-anterior (PA) or antero-posterior (AP)]. Lateral and oblique CXR were excluded from the dataset. We also excluded any special views such as decubitus and CXR acquired with expiratory breath-hold. All CXR were exported after de-identifying the Protected Health Information (PHI). To assess the robustness of the models, we also performed stratified testing based on patients' age and sex as well as radiographic projections (anteroposterior [AP] vs. posteroanterior [PA]).

Standard of Reference

In addition to the radiology report on suboptimality of CXRs, an experienced thoracic radiologist (MKK with 16-year experience) classified each CXR into optimal or suboptimal, and specific causes of suboptimality. Thus, the standard of reference (SOR) for optimal or suboptimal CXR was based on interpretation of two radiologists. The classification was based on the American College of Radiology (ACR)-Society of Pediatric Radiology (SPR)-Society of Thoracic Radiology (STR) guidelines for diagnostic radiographic images¹¹ (Fig. 1). The same radiologist also sub-categorized each suboptimal CXR into five subcategories: 1. anatomy not included when any portion of lungs (apical, basal, or lateral) were excluded from the field of view; 2. improper exposure (under-exposure when the spine was not visualized through the heart shadow or over-exposure when there was a lack of contrast in the lung); 3. substantial rotation in the presence of significant asymmetric distance between the medial end of clavicle and the thoracic spinous process (minimal rotation was deemed acceptable as it does not require repeat acquisition and does not have a major effect on interpretation); 4. low lung volume (visibility limited to less than 6 anterior ribs or 9 posterior ribs above the right hemidiaphragm); 5. overlying anatomy- when the chin or part of the skull obscured parts of the lungs.

AI Model

We used COGNEX Vision Pro Deep Learning software from COGNEX Corporation (Natick, MA, United States) which combines a comprehensive machine vision tool library with advanced deep learning tools. The software enables users such as physicians without any technical/ programming knowledge to train AI models for specific clinical application. The various deep learning architectures within the software include green (for classification of objects or images- available in two modes: focused and high detail), red (for segmentation and annotation of specific regions of interest), and blue (for localization and identification of landmarks) tools.

We used the green classification tool to train and test five AI models, one for each suboptimal category. Details of the software and its classification tool have been previously described ¹². The tool is used to classify image features from the entire image. It learns from datasets of images labeled with different classes (such as optimal versus suboptimal) and then can be used to classify unseen images. The tool is based on convolutional neural network architectures like ResNet or DenseNet. However, unlike how neural network architectures are traditionally used, the current tool has embedded image preprocessing steps and does not require the user to perform additional preprocessing steps before model training. Of the two subcategories of "high-detail" and "focused" in the green tool. High detail mode utilizes a neural network architecture that trains on the entire image, whereas focused mode works selectively on only parts of the image that it deems useful for the prediction decision. As a result, the high detail mode takes longer to train but can provide improved performance especially in cases where important details are spread throughout the image. We employed the high-detail subcategory for our models with an epoch count of 100 (default). Once the green tool is selected with the subcategory (and default 100 epoch count), the model training starts when the "brain" icon is clicked. The platform also generates separate heat maps for each CXR evaluated with the five AI models.

Training

Two study physician coinvestigators (GD, MKK) without any prior knowledge or training in data science or computer programming trained different labelAI models for each of the five subcategories of suboptimal CXR described in the preceding section. The de-identified DICOM images were uploaded on the software within the firewall of the hospital. Each image was labeled with optimal and suboptimal subcategories based on the SOR. First, we evaluated the overall performance of AI models on data from all the 5 sites with 70%-30% distribution of CXR in the training and testing sets, respectively. Default threshold for each model was set at 50%. For each model, we performed cross validation with repeated 5-fold cross validation method where the test and training data were randomized automatically within the software user interface. Model outputs were separately recorded.

To test generalizability of AI performance across different sites, we then trained separate AI models using training dataset from three sites (A, B, C) and testing datasets from the remaining two sites (Sites D, E).

Statistical Evaluation

The data were analyzed using SPSS version 24 and Microsoft EXCEL (Microsoft Inc.). For analysis, we classified the AI output as true positive (both the SOR and AI model agreed with suboptimal subcategory of CXR), true negative (both the SOR and AI agreed on optimal CXR), false positive (SOR labeled optimal CXR was called suboptimal by the AI), false negative (SOR classified CXR as suboptimal, and AI labeled CXR as optimal). The performance of all AI models for classification of optimal and suboptimal CXR was estimated by calculating sensitivity, specificity, accuracy, and AUC- area under the receiver operating characteristic curve with 95% confidence intervals (CI). Average sensitivity and specificity were calculated from the 5-fold cross validation.

For each AI model, we generated learning curves (also called loss graphs) from the graphic user interface of the Deep Learning Studio employed in our study. These graphs summarize epochs along the x-axis and value of the model loss function along the y-axis.

A total of 3278 CXRs of adult patients from 5 hospitals were included in the study (mean age 55 ± 20 years; 1535 men, 1743 women). The SOR classified 1154 as optimal and the remaining 2124 as suboptimal. Most suboptimal CXR had a single cause for suboptimality (1594/2124 CXR, 75%). In one-quarter of the suboptimal CXR (530/2124), there were multiple reasons for suboptimality. Among the multiple cause suboptimal CXR, 399 CXR (399/2124, 18.8%) radiographs had two causes, 114 CXR had three reasons (114/2124, 5.4%), 16 CXR had four reasons (16/2124, 0.01%) and 1 CXR had five reasons for suboptimality.

The most common causes of suboptimal CXR were low lung volume followed by non-inclusion of entire lungs in the field of view. There were significant differences in distribution of suboptimal and optimal CXR at the five sites included in our study (p < 0.0001).

The distribution of suboptimal CXR by different causes are summarized in Table 1. Sensitivities, specificities, accuracies and AUCs of the five AI models in same site (Sites A, B, C) training and test CXR are summarized in Table 2. Corresponding statistics for assessing generalizability of the AI models with Sites A, B, C as training data, and Sites D, E as testing data are stated in Table 3. Figure 2 illustrate the corresponding heat maps for the two AI models for detecting non-inclusion of entire lungs in the field of view and obscuring anatomy overlying the chest. Tables 4–6 summarize performance of AI models in different age groups (< 40 years vs. 40–60 years vs. > 60 years), sex (men vs women) and CXR projection (PA vs. AP) as represented in the supplementary material.

Table 1

Summary of demographics and optimal and suboptimal CXR at all five sites included in the study.
Category	Sub-categories	Training CXR	Test CXR	Total
Women	-	1072	463	1535
Men	-	1219	524	1743
Suboptimal	Anatomy not included	242	105	347
	Exposure	308	132	440
	Low lung volume	536	230	766
	Overlying anatomy	240	104	344
	Rotation	158	69	227
Optimal	-	807	347	1154

Table 2

Sensitivity, specificity, and AUC (95% CI) of five AI models for training and testing CXR from three sites (A, B, C).
Sub-categories	Training CXR	Test CXR	Total CXR	Sensitivity	Specificity	Accuracy	AUC	95% CI (AUC)
Anatomy not included	171	74	245	82%	96%	93%	0.89	0.84–0.95
Exposure	248	107	355	94%	95%	95%	0.95	0.92–0.98
Low lung volume	341	147	488	91%	97%	95%	0.94	0.91–0.97
Overlying anatomy	178	75	253	88%	98%	96%	0.93	0.89–0.96
Rotation	122	53	175	83%	99%	96%	0.91	0.85–0.97
Optimal	634	272	906	-	-	-	-	-

Table 3

Summary of results for assessing generalizability of the five AI models with training CXR from three sites (A, B, C) and test CXR from the remaining two sites (D, E).
Sub-categories	Training CXR	Test CXR	Total CXR	Sensitivity	Specificity	Accuracy	AUC	95% CI (AUC)
Anatomy not included	171	102	273	75%	93%	88%	0.84	0.78–0.89
Exposure	248	85	333	92%	96%	95%	0.94	0.90–0.98
Low lung volume	341	278	619	80%	90%	85%	0.85	0.81–0.88
Overlying anatomy	178	91	269	90%	94%	93%	0.92	0.88–0.96
Rotation	122	52	174	81%	99.6%	96%	0.90	0.84–0.97
Optimal	634	248	882	-	-	-	-	-

The overall performance of different AI models for classifying optimal and suboptimal CXR from all the 5 sites were as follows: excluded anatomy/lungs (78% sensitivity, 95% specificity, 91% accuracy, 0.87 AUC [95% CI 0.82–0.92]); inappropriate exposure (90% sensitivity, 93% specificity, 92% accuracy, 0.91 AUC [95% CI 0.88–0.95]); low lung volume (96% sensitivity, 92% specificity, 93% accuracy, 0.94 AUC [95% CI 0.92–0.96]); obscured anatomy (91% sensitivity, 97% specificity, 95% accuracy, 0.94 AUC [95% CI 0.90–0.97]); and patient rotation (92% sensitivity, 96% specificity, 95% accuracy, 0.94 AUC [95% CI 0.91–0.98]). Loss graphs (Fig. 3) illustrate stable performance of all five AI models trained in our study.

Our study demonstrates that the assessed AI tool-building platform allows physicians without any prior training in data sciences or computer programming to train AI models with acceptable performance. All five AI models to identify different causes of suboptimal CXR had high sensitivity, specificity, and accuracy for identifying suboptimality from non-included anatomy, improper exposure, substantial patient rotation, low lung volumes, and overlying anatomy obscuring visibility of lungs or mediastinum. Performance of all 5 AI models was consistent across different patient age groups, sex, and radiographic projections. Using CXR data from our internal multi-institutional consortium, we were able to train five AI models from end-to-end. Such process includes data identification, report curating, standard of reference data labeling to model training and testing on a subset of 3278 CXR without help from any data scientists or engineers over a span of less than 8 weeks. At the same time, we were able to restrict our training datasets to three sites and establish "local" generalizability on CXR from the remaining two sites.

To the best of our knowledge, there is one peer-reviewed report on the use of convolutional neural network for quality control on chest radiographs from Nousiainen et al. The authors evaluated three causes of suboptimal CXR (non-inclusion of entire lungs, patient rotation, and low lung volumes) based on the European guidelines for radiographic quality assessment as opposed to the ACR-SPR-STR guidelines ⁷. For these overlapping three suboptimality causes, our AI models (respective AUCs of 0.87, 0.94, 0.94) were comparable or better than the corresponding AUCs of 0.88, 0.70 and 0.79 for lung exclusion, patient rotation, and low lung volume reported in the prior study ⁴. Although such differences in performance might be related to the use of differences in the deployed neural networks, training and test datasets, complexity of CXR, or labeling of CXR as suboptimal and optimal. While Nousiainen et al. included presence of any rotational discrepancy, we opted for presence of substantial rotation due to its higher clinical impact on diagnostic interpretation and lower need for rejecting or repeating the CXR due to minor degree of patient rotation ⁷.

The AI models that we trained has several clinical implications. First, implementation of such algorithms at the point of care, acquisition devices can identify and prompt repeat acquisition of suboptimal CXR targeted towards correction of specific cause (such as correct inspiratory effort, patient rotation or inclusion of previously excluded lung apices). Repeat acquisition involves additional radiation dose to the patients but avoids delay in diagnostic interpretation and patient care which can be key in critical or unstable patients. Several radiography vendors are also working on such AI applications to improve quality and decrease reject rates and repeat acquisition ¹³.

The incidence of suboptimal CXR images reported in the literature varies considerably, and some studies report that to be as high as 50–96% ^3,4. With over 5000 CXRs per week at our healthcare sites, this implies that a manual auditing process in daily workflow is impractical and will undercount the true incidence of suboptimal CXRs. Beyond the point of care devices, the AI models that we trained can help automate and expedite auditing of CXR for quality assurance and improvement purposes which is currently a manual process involving a technologist reviewing all radiographs which required repeat acquisition. An automated process using the trained AI models can help track such information in near time and provide targeted, large-scale feedback to the technologists and the department on specific suboptimal causes. Constant feedback can help initiate mitigating educational activities and help bring down the frequency of suboptimal CXR as well as the repeat rates. A decrease in repeat radiography rates will translate into both radiation dose and cost savings. Indeed, Poggenborg et al reported that feedbacks on image quality resulted in substantial improvement in CXR ³.

There are a few commercially available AI algorithms such as Annalise.ai and Qure.ai that can identify some causes of suboptimal CXR. However, these are not approved for clinical use by the United States Food and Drug Administration (FDA) at the time of preparation of our manuscript. From the diagnostic interpretation perspective, once implemented in the routine clinical workflow. AI models such as the one we trained have the potential to trigger and insert automated and editable report text pertaining to the presence and cause of suboptimal CXR. This can both alert the radiologist and save their time on describing or dictating such limitations.

There are limitations in our study. First, we used enriched, retrospective datasets of optimal and suboptimal CXR rather than continuous, prospective CXR. Given the sheer volume of CXR, a prospective trial will be impractical and require evaluation of thousands of additional CXR to obtain symmetric representation of all suboptimality causes. Second, we did not perform a pre-hoc or post-hoc analysis to determine the sample size. However, high performance of all AI models is reassuring that our study was adequately powered. Third, we did not stratify the suboptimality causes into those that could be avoided or mitigated with repeat CXR versus those that cannot be corrected. Examples of the latter suboptimality would be patients with complex anatomy (scoliosis and/or kyphosis resulting in rotation or oblique projection) or extremely sick or ventilated patients (with low lung volumes). Fourth, we did not assess the impact of our AI models in real-world chest radiography which was not possible since the models are not part of our clinical workflow or radiography equipment yet. Fifth, although we included CXR from five healthcare sites in our data, all sites were affiliated with one consortium and belonged to the Northeast United States. This limits generalizability of our models in different geographic practices or population demographics. Finally, we did not include uncertainty analysis in our study for CXR with different extent or uncertainty levels of suboptimality ¹⁴.

In summary, the assessed deep learning platform can enable physicians to train and test successful AI models without any background in data science or computer programming. The trained AI models can identify and classify different causes of suboptimal CXR. Implementation of such AI models to identify suboptimal CXR can help provide feedback on suboptimal exams resulting in continuous quality control thereby improving the quality of care.

CXR- Chest Xray, sCXR- Suboptimal Chest Xray, oCXR- Optimal Chest Xray, AI- Artificial Intelligence, SOR- Standard of Reference.

Author contribution:

G.D.- literature review, data collection, study analysis, statistical analysis, manuscript writing

B.B.- data management, manuscript editing

R.V.G.- literature review, data collection

P.K.- data collection, study analysis

S.E.- data collection, statistical analysis

D.R.- data collection, manuscript editing

F.A.- data collection, interpretation

N.N.- data management

S.R.D.- Study design, manuscript editing

M.K.K.- Study conceptualization, study design, manuscript writing and critical revisions

K.J.D.- data management, study design, manuscript editing.

Data availability statement

All data generated or analyzed during the study are included in the published paper. Due to the policies of our institution, we are unable to share the images.

Murphy, K. How data will improve healthcare without adding staff or beds., <https://www.wipo.int/edocs/pubdocs/en/wipo_pub_gii_2019-chapter8.pdf. > (2019).
Zotin, A., Hamad, Y., Simonov, K. & Kurako, M. Lung boundary detection for chest X-ray images classification based on GLCM and probabilistic neural networks. Procedia Computer Science 159, 1439-1448 (2019).
Poggenborg, J., Yaroshenko, A., Wieberneit, N., Harder, T. & Gossmann, A. Impact of AI-based Real Time Image Quality Feedback for Chest Radiographs in the Clinical Routine. medRxiv (2021).
Whaley, J. S.et al. Investigation of the variability in the assessment of digital chest X-ray image quality. Journal of digital imaging 26, 217-226 (2013).
Reiner, B. I. Automating quality assurance for digital radiography. Journal of the American College of Radiology 6, 486-490 (2009).
Rubinowitz, A. N., Siegel, M. D. & Tocino, I. Thoracic imaging in the ICU. Critical care clinics 23, 539-573 (2007).
Nousiainen, K., Mäkelä, T., Piilonen, A. & Peltonen, J. I. Automating chest radiograph imaging quality control. Physica Medica 83, 138-145 (2021).
Kwon, T.et al. Diagnostic performance of artificial intelligence model for pneumonia from chest radiography. Plos one 16, e0249399 (2021).
Yoo, H.et al. AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset. European Radiology 31, 9664-9674 (2021).
Mongan, J., Moy, L. & Kahn Jr, C. E. Vol. 2 e200029 (Radiological Society of North America, 2020).
ACR-SPR-STR practice parameter for the performance of chest radiography<https://www.acr.org/-/media/ACR/Files/Practice-Parameters/ChestRad.pdf. > (2017).
Sarkar, A., Vandenhirtz, J., Nagy, J., Bacsa, D. & Riley, M. Identification of images of COVID-19 from chest X-rays using deep learning: comparing COGNEX VisionPro deep learning 1.0™ software with open source convolutional neural networks. SN Computer Science 2, 1-16 (2021).
Sun, L. Applying AI in Radiology to Optimize workflow., <https://www.carestream.com/blog/2021/09/13/applying-ai-in-radiology-to-optimize-workflow> (2021).
Ghesu, F. C.et al. Quantifying and leveraging predictive uncertainty for medical image assessment. Medical Image Analysis 68, 101855 (2021).

Competing interest reported. The Data Science Office (DSO) at our institution is funded in part by monies and resources from Cognex Corporation.

Supplementalmaterial.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Successful creation of clinical AI without data scientists or software developers: radiologist-trained AI model for identifying suboptimal chest-radiographs

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods:

Study Design and Data Definitions

Data Partitioning

Standard of Reference

AI Model

Training

Statistical Evaluation

Results

Discussion

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1