Identification of Barrett's Esophagus in Endoscopic Images Using Deep Learning

doi:10.21203/rs.3.rs-698574/v1

Download PDF

Research Article

Identification of Barrett's Esophagus in Endoscopic Images Using Deep Learning

https://doi.org/10.21203/rs.3.rs-698574/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 01 Dec, 2021

Read the published version in BMC Gastroenterology →

You are reading this latest preprint version

Background: Development of a deep learning method to identify Barrett's esophagus (BE) scopes in endoscopic images.

Methods: 443 endoscopic images from 187 patients of BE were included in this study. The gastroesophageal junction (GEJ) and squamous-columnar junction (SCJ) of BE were manually annotated in endoscopic images by experts. Fully convolutional neural networks (FCN) were developed to automatically identify the BE scopes in endoscopic images. The networks were trained and evaluated in two separate image sets. The performance of segmentation was evaluated by intersection over union (IOU).

Results: The deep learning method was proved to be satisfying in the automated identification of BE in endoscopic images. The values of the IOU were 0.56 (GEJ) and 0.82 (SCJ), respectively.

Conclusions: Deep learning algorithm is promising with accuracies of concordance with manual human assessment in segmentation of the BE scope in endoscopic images. This automated recognition method helps clinicians to locate and recognize the scopes of BE in endoscopic examinations.

Gastroenterology & Hepatology

Barrett's esophagus

Esophagoscope

Deep learning

Fully convolutional networks

Segmentation.

Barrett's esophagus (BE) is a precancerous state caused by damages to the inner lining of the squamous esophageal mucosa, characterized by a change of the normal stratified squamous epithelium lining esophagus to a metaplastic columnar epithelium with goblet cells [1]. BE is the only known histological precursor of esophageal adenocarcinoma (EAC) associated with high mortality (5-year survival rate < 20%) and increasing incidences [2–5]. EAC patients with a prior diagnosis of BE normally have better outcomes than patients without a prior diagnosis of BE [6]. Therefore, early detection and appropriate treatment of BE are crucial for effective prevention of the development of EAC. At present, the most common screening method for BE is a pathological biopsy using samples obtained through esophagoscopy (ESO). However, due to the individual variations of the shapes, appearances, and textures of BE, accurate identification and location of the BE scope are still challenging. Moreover, the locating of BE relies on the individual experience of endoscopists, which might further introduce variations and bias. These problems might cause time consumption and misjudgments, probably led to delays in the identification or misdiagnosis of BE, and finally influenced the follow-up treatments. Therefore, to overcome these difficulties, it is necessary to further improve the efficiency and accuracy of BE identifying and locating under endoscopic examinations.

Recent years have witnessed tremendous development in artificial intelligence (AI), especially the emerging deep learning (DL) has achieved unprecedented successes in various domains with groundbreaking performance on par with human capabilities [7]. More recently, there is a trend of applying DL in healthcare and clinical applications [8, 9]. As a subbranch of AI, DL utilizes multiple layers of neurons to extract abstract patterns from data. In image analysis, DL shows encouraging potentials in tasks of segmentation, classification, and prediction [10–12]. A growing body of literature developed DL methods in analyses of medical images such as ultrasound, CT, MRI, X-ray [13–15].

Recently, DL has been gradually utilized in endoscopic image analysis of colon, stomach, and intestine, etc., with encouraging performance in identifying and diagnosing diseases such as tumors, polyps, and ulcers [16–18]. Meanwhile, several studies applied DL in the classification and segmentation of esophageal lesions [19–25]. However, there is still a lack of reports of developing DL methods dedicated to BE identification. In one study, a migration-based learning approach was adopted to segment endoscopic images included cancer and BE [19]. In another recent study, a convolutional neural network (CNN) was used to segment endoscopic images of cancer, BE, and inflammation [20]. Because of the importance of early diagnosis of BE in the prevention of EAC, it is worth further investigating DL in BE diagnosis with a sufficient BE sample size covering all BE types according to the Prague classification system [26].

The objective of this study was to develop a fully automated DL method for early-accurate segmentation and identification of BE in endoscopic images. We included 443 endoscopic images from 187 BE patients. The DL method could accurately identify and segment the scopes of BE, which could further facilitate the following endoscopic surveillance and treatment of BE.

Patient characteristics

In our retrospective study, a total of 187 patients examined using endoscopy between January 2015 and June 2019 were included in the Hospital of Chengdu Office of People’s Government of Tibetan Autonomous Region. The BE conditions were confirmed by pathological examinations. All data were anonymized, and an Ethics Approval was granted by the Ethics Committee of Hospital of Chengdu Office of People’s Government of Tibetan Autonomous Region (No. 201920).

Image acquisition

We obtained 443 endoscopic images from a total of 187 clinical cases, and the instruments used in the examinations were Olympus GIF-HQ290, GIF-Q260 gastroscope (Olympus Company, Japan). The esophagus was cleaned and examined with white light, narrow band imaging, and staining endoscopy. The BE scope was recorded according to the Prague classification system. The endoscope was positioned proximally to the GEJ, and the endoscopic image was taken. Meanwhile, the biopsy samples were obtained using biopsy forceps, and the final diagnoses were proved by pathologists.

Image annotation

To obtain the ground truth of the BE scopes in images, we invited two senior endoscopists with over 15 years’ experience to manually draw the outlines of the scopes using one in-house developed software. More specifically, the rims of the GEJ and SCJ were delineated to define the BE scopes. The experts were trained to follow the same quality standard before conducting the tasks. The first expert annotated all images, and the results were confirmed by the second expert. For any disagreement, the two experts discussed and made necessary new annotations until consensus was reached. The annotation information was later extracted to generate segmentations as ground truths for later DL algorithm training and evaluation.

Deep learning algorithm

In this study, we developed a DL algorithm in a neural network structure of fully convolutional networks (FCN) [27]. As shown in Figure 2, the neural network adopted several layers of fully convolutional neural network layers to extract abstract feature maps of an input image. After the downsampling, the deconvolutional neural network layers were appended to conduct the upsampling to generate the output image in the same size as the input image. Skip architectures were fused to both deep and shallow layers to achieve semantic segmentation at the pixel level. Furthermore, FCN is capable of processing images of any size, which allows FCN to be more suitable for medical images of various sizes. In the training stage, each image was input into the FCN, and a corresponding mask was generated to indicate the segmentation. The segmentation was compared against the ground truth obtained by experts. The loss formation was used to train the FCN. When all images in the training set were used to update the network, a trained FCN was obtained and passively used to generate segmentations for any inputs.

Since the BE scopes usually have two rims, namely GEJ and SCJ. We considered two approaches. Firstly, we trained two FCN networks independently to achieve the segmentation for GEJ and SCJ. In other words, two independent FCN networks were trained and evaluated using the annotations to obtain the rims of GEJ and SCJ. Secondly, we segmented the GEJ and SCJ using one single trained network. We reported and compared the performance of the two approaches. The obtained segmentations were further visualized for examinations.

To train and test the developed DL algorithm, we randomly divided the collected 443 images from 187 patients into two independent subsets according to patients. This approach ensured no images from a given individual patient appeared in both training and testing sets. In result, we obtained two subsets, namely a training set (n= 150, 354 images, 80%) and a test set (n = 37, 89 images, 20%). The DL algorithm was first trained using the annotated images in the training set. Afterward, the trained DL algorithm was evaluated in the test set.

The FCN neural network was implemented in the programming language of Python (3.7.3) using publicly available libraries of PyTorch (1.1.0), CUDA (10.1), and NumPy (1.16.2). The algorithm was trained and evaluated in a DL server equipped with a Tesla P40 graphic processing unit (GPU) running the operating system CentOS Linux (7.6.1810). Though a DL server was utilized in this study, it’s believed that a conventional workstation nowadays could be used to deploy the trained DL algorithm and generate segmentations within an acceptable time.

Statistical analysis

In line with previous studies of image segmentation, the metric of IOU was used to measure the performance of the DL algorithms. Intuitively, IOU indicated how well the predicted segmentation overlapped with the ground truth. A larger value of IOU closes to one indicates a favorable segmentation performance for a given algorithm. We also reported the Dice similarity coefficient (DSC), which is similar to IOU and widely appears in literature [28]. However, we used IOU as the major measurement for its simplicity and wide acceptance in literature. Therefore, the overall performance was reported as the averaged IOU and DSC in the test set.

Patient characteristics and images

A total of 187 BE patients (48 men and 139 women) were included in this study (Table 1), and 443 images with presences of BE were collected. Patients were randomly divided into one training set (80%, 150 patients, 354 images) and one test set (20%, 37 patients, 89 images) (Table 2). The two sets were independent without overlaps to avoid bias.

Table 1

Patient characteristics.
	Total	Male	Female
Number	187 (100%)	48 (25.67%)	139 (74.33%)
Age	53.96 ± 10.54	53.31 ± 10.66	54.18 ± 10.56

Table 2

Patients were randomly divided into one training set (80%) and one test set (20%) according to patients.
Dataset	Patients	Images
Training set	150 (80%)	354
Test set	37 (20%)	89

Performance

As described above, the DL algorithms were developed to obtain the GEJ and SCJ separately in two approaches. In Table 3, we reported the results of segmentation achieved by FCN for GEJ and SCJ in the test set. We found that the approach of separately segmenting GEJ and SCJ using two FCN networks was optimal than the approach of using one single network. To illustrate the results of DL in the identification of BE scopes, we visualized the segmentations of both DL and experts for representative samples in Fig. 3. As shown, DL was capable to accurately identify the GEJ and SCJ of BE scopes. Experts examined the DL results for all images in the test set and concluded that the agreement between DL results and expert annotations was satisfying. For those cases with smaller values of IOU, the overall shapes of GEJ and SCJ obtained by DL were also acceptable.

Table 3

Performance of DL algorithm achieved in the test set in the tasks of identifying the GEJ and the SCJ of the BE scopes.
GEJ/SCJ	IOU		DSC
GEJ/SCJ	Average	Std. Dev.	Average	Std. Dev.
GEJ	0.56	0.14	0.71	0.12
SCJ	0.82	0.12	0.90	0.08
GEJ + SCJ	0.66	0.13	0.79	0.11

EAC is the main histological type of esophageal cancer in the west [29], and BE is the only known histological precursor of EAC. Recently, several reports indicated that the incidences of BE and Barrett’s esophageal adenocarcinoma (BEA) were rising in Asia [30, 31]. Previous studies have shown that prior diagnosis, surveillance [6, 32–34], and appropriate treatment practices [35, 36] of BE can reduce the risk of EAC progression and improve survival. Endoscopic biopsy is the most commonly used method for diagnosis and monitoring of BE [37], and endotherapy such as endoscopic resection and esophageal ablation becomes the standard of care for BE [38]. These measures all require endoscopists to accurately identify the scopes of BE under endoscopic examination [39, 40]. This process relies on the experience of individuals with inevitable misjudgments, variations, and time consumption. Moreover, the diversities in shapes, appearances, and textures of BE contribute to the difficulties of accurate segmentation of BE scopes.

Therefore, in this study, we proposed and developed a DL method to automatically segment the BE scopes in endoscopy, which could further improve early-accurate diagnosis and treatments of BE. We collected 443 images from 187 patients and invited experts to manually annotate BE scopes. We constructed one training set and one test set to develop and evaluate the DL methods.

In one study, 100 endoscopic images, including 50 cancer cases and 50 BE cases of 39 patients were studied [19]. Using a migration-based learning approach, they reported a sensitivity of 0.94 and specificity of 0.88. However, the BE cases were relatively insufficient. In another research, neural networks were developed to segment 797 endoscopic images of cancer, BE, and inflammation cases [20]. Compared to the above works, our study dedicated to the automated identification and location of BE scopes in endoscopic images using DL. Especially we considered including all BE types according to the Prague classification system. Additionally, we developed DL methods to accurately identify both GEJ and SCJ. It is worth mentioning that our approach of separately segment GEJ and SCJ using two DL networks outperformed the traditional approach using one single DL network was inspiring for similar medical image analysis tasks. These efforts could assist endoscopists in the diagnosis of BE efficiently.

However, there are still several limitations in this study. Firstly, we only focused on developing DL to automated segment the scopes of BE under esophagoscopic examination but did not differentiate from other esophageal lesions. In the future, we would include other esophageal lesions and extend the present DL framework to classify and diagnose types of BE. Secondly, this is a retrospective study from a single center. The results could be further validated in prospective studies using external cohorts. Thirdly, the developed DL methods in this study are still in rapid evolution with more emerging advanced DL algorithms. It’s expectable to evaluate new DL algorithms to diagnose BE in endoscopic images to further improve the performance.

In this study, we carried out the recognition and segmentation of the BE scope in endoscopic images using DL. Specifically, FCN neural networks were developed and evaluated. The DL methods achieved satisfying performance in the segmentation of GEJ and SCJ, indicating their promising potentials in clinical BE evaluations.

Barrett's esophagus; EAC:esophageal adenocarcinoma; ESO:esophagoscopy; GEJ:gastroesophageal junction; SCJ:squamous-columnar junction; AI:artificial intelligence; DL:deep learning; FCN:fully convolutional networks; CNN:convolutional neural network; IOU:intersection over union; DSC:Dice similarity coefficient; BEA:Barrett’s esophageal adenocarcinoma

Conflict of interest

The authors report no conflicts of interest.

Authors’ contributions

Wen Pan, Xujia Li, Weijia Wang, Linjing Zhou, Jiali Wu, Tao Ren, Chao Liu, Muhan Lv, Song Su and Yong Tang conceived and designed the study, and were responsible for the final decision to submit for publication. All authors were involved in the development, review, and approval of the manuscript. All authors read and approved the final manuscript.

Funding

This study is supported by the Innovation Method Program of the Ministry of Science and Technology of the People’s Republic of China (M112017IM010700), The Key Research and Development Project of Science & Technology Department of Sichuan Province (20ZDYF1129), Natural Science Foundation of Tibet Autonomous Region (XZ2019ZR G-130), The Applied Basic Research Project of Science & Technology Department of Luzhou city (2018-JYJ-45).

Ethics approval and consent to participate

This was a retrospective study approved by the Ethics Committee of Hospital of Chengdu Office of People’s Government of Tibetan Autonomous Region (No. 201920).

Consent for publication

All authors read and approved the final manuscript.

Availability of data and materials

The datasets during and/or analyzed during the current study available from the corresponding author on reasonable request.

Acknowledgments

We appreciated our pathologists and endoscopists for their valuable support and discussions.

Iyer PG, Kaul V. Barrett Esophagus. Mayo Clin Proc 2019;94:1888–1901.
Launoy G, Bossard N, Castro C, et al. Trends in net survival from esophageal cancer in six European Latin countries: results from the SUDCAN population-based study. Eur J Cancer Prev 2017;26 Trends in cancer net survival in six European Latin Countries: the SUDCAN study:S24-S31.
Njei B, McCarty TR, Birk JW. Trends in esophageal cancer survival in United States adults from 1973 to 2009: A SEER database analysis. J Gastroenterol Hepatol 2016;31:1141–6.
Anderson LA, Tavilla A, Brenner H, et al. Survival for oesophageal, stomach and small intestine cancers in Europe 1999–2007: Results from EUROCARE-5. Eur J Cancer 2015;51:2144–2157.
Thrift AP. Global burden and epidemiology of Barrett oesophagus and oesophageal cancer. 2021.
Verbeek RE, Leenders M, Kate FJWt, et al. Surveillance of Barrett's esophagus and mortality from esophageal adenocarcinoma: a population-based cohort study. 2014;109:1215–1222.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436–444.
Faust O, Hagiwara Y, Hong TJ, et al. Deep learning for healthcare applications based on physiological signals: A review. Computer Methods and Programs in Biomedicine 2018;161:1–13.
Purushotham S, Meng C, Che Z, et al. Benchmarking deep learning models on large healthcare datasets. Journal of Biomedical Informatics 2018;83:112–134.
Aoki T, Yamada A, Aoyama K, et al. Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network. Gastrointestinal Endoscopy 2019;89:357–363.e2.
Coudray N, Ocampo PS, Sakellaropoulos T, et al. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nature Medicine 2018;24:1559–1567.
Jiang F, Grigorev A, Rho S, et al. Medical image semantic segmentation based on deep learning. Neural Computing and Applications 2018;29:1257–1265.
Ait Skourt B, El Hassani A, Majda A. Lung CT Image Segmentation Using Deep Neural Networks. Procedia Computer Science 2018;127:109–113.
Cheng J-Z, Ni D, Chou Y-H, et al. Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans. Scientific Reports 2016;6:24454.
Mittal M, Goyal LM, Kaur S, et al. Deep learning based enhanced tumor segmentation approach for MR brain images. Applied Soft Computing 2019;78:346–354.
Ling T, Wu L, Fu Y, et al. A deep learning-based system for identifying differentiation status and delineating the margins of early gastric cancer in magnifying narrow-band imaging endoscopy. Endoscopy 2021;53:469–477.
Zheng W, Zhang X, Kim JJ, et al. High Accuracy of Convolutional Neural Network for Evaluation of Helicobacter pylori Infection Based on Endoscopic Images: Preliminary Experience. Clin Transl Gastroenterol 2019;10:e00109.
Urban G, Tripathi P, Alkayali T, et al. Deep Learning Localizes and Identifies Polyps in Real Time With 96% Accuracy in Screening Colonoscopy. Gastroenterology 2018;155:1069–1078 e8.
Mendel R, Ebigbo, A., Probst, A., Messmann, H., Palm, C. Barrett’s esophagus analysis using convolutional neural networks. Bildverarbeitung für die Medizin 2017:80–85.
Wu Z, Ge R, Wen M, et al. ELNet:Automatic classification and segmentation for esophageal lesions using convolutional neural network. 2021;67:101838.
de Groof AJ, Struyvenberg MR, Fockens KN, et al. Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video). Gastrointest Endosc 2020;91:1242–1250.
Liu G, Hua J, Wu Z, et al. Automatic classification of esophageal lesions in endoscopic images using a convolutional neural network. 2020;8:486.
de Groof J, van der Sommen F, van der Putten J, et al. The Argos project: The development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy. United European Gastroenterol J 2019;7:538–547.
Jisu HB-Y, Park; Hyunjin, Park. Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference 2017;2017:2892–2895.
Gehrung M, Crispin-Ortuzar M, Berman AG, et al. Triage-driven diagnosis of Barrett's esophagus for early detection of esophageal adenocarcinoma using deep learning. Nat Med 2021.
Kinjo T, Kusano C, Oda I, et al. Prague C&M and Japanese criteria: shades of Barrett's esophagus endoscopic diagnosis. J Gastroenterol 2010;45:1039–44.
Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
Milletari F, Navab N, Ahmadi S-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation, In 2016 fourth international conference on 3D vision (3DV), IEEE, 2016.
Coleman HG, Xie SH, Lagergren J. The Epidemiology of Esophageal Adenocarcinoma. Gastroenterology 2018;154:390–405.
Wu JC. Gastroesophageal reflux disease: an Asian perspective. J Gastroenterol Hepatol 2008;23:1785–93.
Hongo M, Nagasaki Y, Shoji T. Epidemiology of esophageal cancer: Orient to Occident. Effects of chronology, geography and ethnicity. J Gastroenterol Hepatol 2009;24:729–35.
Bhat SK, McManus DT, Coleman HG, et al. Oesophageal adenocarcinoma and prior diagnosis of Barrett's oesophagus: a population-based study. Gut 2015;64:20–5.
Cooper GS, Yuan Z, Chak A, et al. Association of prediagnosis endoscopy with stage and survival in adenocarcinoma of the esophagus and gastric cardia. Cancer 2002;95:32–8.
Cooper GS, Kou TD, Chak A. Receipt of previous diagnoses and endoscopy and outcome from esophageal adenocarcinoma: a population-based study with temporal trends. Am J Gastroenterol 2009;104:1356–62.
Phoa KN, van Vilsteren FG, Weusten BL, et al. Radiofrequency ablation vs endoscopic surveillance for patients with Barrett esophagus and low-grade dysplasia: a randomized clinical trial. JAMA 2014;311:1209–17.
van Munster S, Nieuwenhuis E, Weusten B, et al. Long-term outcomes after endoscopic treatment for Barrett's neoplasia with radiofrequency ablation ± endoscopic resection: results from the national Dutch database in a 10-year period. 2021.
Eluri S, Shaheen NJ. Barrett's esophagus: diagnosis and management. 2017;85:889–903.
Prasad GA, Wu TT, Wigle DA, et al. Endoscopic and surgical treatment of mucosal (T1a) esophageal adenocarcinoma in Barrett's esophagus. Gastroenterology 2009;137:815–23.
Fitzgerald RC, Saeed IT, Khoo D, et al. Rigorous surveillance protocol increases detection of curable cancers associated with Barrett's esophagus. Dig Dis Sci 2001;46:1892–8.
Abela JE, Going JJ, Mackenzie JF, et al. Systematic four-quadrant biopsy detects Barrett's dysplasia in more patients than nonsystematic biopsy. Am J Gastroenterol 2008;103:850–5.

No competing interests reported.

Download PDF

Journal Publication

published 01 Dec, 2021

Read the published version in BMC Gastroenterology →

Editorial decision: Major revision
11 Oct, 2021
Reviews received at journal
20 Sep, 2021
Reviewers agreed at journal
05 Sep, 2021
Reviewers invited by journal
23 Jul, 2021
Editor assigned by journal
23 Jul, 2021
Editor invited by journal
20 Jul, 2021
Submission checks completed at journal
20 Jul, 2021
First submitted to journal
08 Jul, 2021

You are reading this latest preprint version

Identification of Barrett's Esophagus in Endoscopic Images Using Deep Learning

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Materials And Methods

Results

Patient characteristics and images

Performance

Discussion

Abbreviations

Declarations

References

Additional Declarations

Status:

Journal Publication

Version 1