Classification of Multi-Differentiated Liver Cancer Pathological Images Based on Deep Learning Attention Mechanism

doi:10.21203/rs.3.rs-1401152/v1

Purpose

Liver cancer is one of the most common malignant tumors in the world, ranking fifth in malignant tumors. The degree of differentiation can reflect the degree of malignancy. The degree of malignancy of liver cancer can be divided into three types: poorly differentiated, moderately differentiated, and well differentiated. Diagnosis and treatment of different levels of differentiation are crucial to the survival rate and survival time of patients. As the gold standard for liver cancer diagnosis, histopathological images can accurately distinguish liver cancers of different levels of differentiation. Therefore, the study of intelligent classification of histopathological images is of great significance to patients with liver cancer. At present, the classification of histopathological images of liver cancer with different degrees of differentiation has disadvantages such as time-consuming, labor-intensive, and large manual investment. In this context, the importance of intelligent classification of histopathological images is obvious.

Methods

Based on the development of a complete data acquisition scheme, this paper applies the SENet deep learning model to the intelligent classification of all types of differentiated liver cancer histopathological images for the first time, and compares it with the four deep learning models of VGG16, ResNet50, ResNet_CBAM, and SKNet. The evaluation indexes adopted in this paper include confusion matrix, Precision, recall, F1 Score, etc. These evaluation indexes can be used to evaluate the model in a very comprehensive and accurate way.

Results

Five different deep learning classification models are applied to collect the data set and evaluate model. The experimental results show that the SENet model has achieved the best classification effect with an accuracy of 95.27%. The model also has good reliability and generalization ability. The experiment proves that the SENet deep learning model has a good application prospect in the intelligent classification of histopathological images,

Conclusions

This study also proves that deep learning has great application value in solving the time-consuming and laborious problems existing in traditional manual film reading, and it has certain practical significance for the intelligent classification research of other cancer histopathological images.

histopathological images of liver cancer

SENet

Degree of differentiation of the whole type

Intelligent classification

Liver cancer is one of the malignant tumors with extremely high fatality rate and directly threatening human life. Hepatocellularcarcinoma (HCC) is a kind of primary liver cancer derived from hepatocytes, with a high degree of malignancy. There are approximately 750,000 patients worldwide each year. The mortality rate of HCC is as high as 93%, ranking third among all malignant tumors [1–5]. In recent years, the global morbidity and mortality of liver cancer have continued to increase, and liver cancer has seriously threatened human life and health. The main clinical features of liver cancer are low predictability, rapid deterioration, and easier death after cancer. The key to the treatment of liver cancer lies in timely diagnosis, and the diagnosis of liver cancer is very important [6–10]. At present, the main diagnostic methods for liver cancer include serum test, biopsy, and imaging diagnosis. Among them, the histopathological images of liver cancer can clearly see the location, size, number, cell and histological type, degree of differentiation, vascular and capsular invasion, satellite foci and metastases, and the lesions of the liver tissue adjacent to the cancer. Among them, the degree of differentiation can reflect the degree of malignancy. The higher the degree of differentiation, the closer to the normal tissue cells, the lower the degree of malignancy. The lower the degree of differentiation, the higher the degree of malignancy. The diagnosis and treatment of different degrees of differentiation is crucial to the survival rate and survival time of patients. [11, 12]. Therefore, the accurate classification of histopathological images of liver cancer has a decisive and irreplaceable role in the diagnosis of liver cancer with different degrees of differentiation. However, the classification of histopathological images with different degrees of differentiation has problems such as time-consuming, labor-intensive, and large manual investment. At the same time, due to the lack of experience of doctors or the fatigue caused by the doctors working for a long time and the individual's subjective consciousness are likely to cause misjudgment, which seriously affects the formulation of the patient's treatment plan and prognostic effect. It has important research value for the further study of histopathological image classification of liver cancer [13–15].

In recent years, with the rapid development of artificial intelligence, artificial intelligence algorithms have been widely used in the medical field. Deep learning, as one of artificial intelligence algorithms, is a machine learning method based on deep neural networks. Deep learning is widely used in computer vision and natural language processing tasks [14, 16–23]. Especially in medical image processing and early disease diagnosis; Zhou, LQ et al. introduced basic technical knowledge about AI, including traditional machine learning and deep learning algorithms, especially convolutional neural networks, and their use in liver disease medical imaging Clinical applications in the field, such as detecting and evaluating focal liver lesions, promoting treatment and predicting liver response to treatment. Finally, it is concluded that machine-assisted medical services will be a promising solution for liver medical services in the future [24]. Li, Jing designed a fully automated computer-aided diagnosis (CAD) system using a convolutional neural network (CNN) network structure to diagnose hepatocellular carcinoma (HCC). The study used a total of 165 venous phase CT images (including 46 diffuse tumors, 43 nodular tumors and 76 large tumors) to evaluate the performance of the proposed CAD system. Finally, the CNN classifier was obtained for the classification accuracy of diffuse, nodular and massive tumors of 98.4%, 99.7% and 98.7%, respectively [25]. However, the study still has sample types that do not use the gold standard, and it is difficult for doctors to design and arrange accurate treatment procedures based on the diagnosis results. Azer, Samy A studied the role of CNN in analyzing images and as a tool for early detection of HCC or liver masses. The results of this study prove that the accuracy of CNN for liver cancer image segmentation and classification has reached the best level [26]. However, this study still has the disadvantage of too few classification types, which has certain limitations. Liangqun Lu and Bernie J. Daigle, Jr et.al used the pre-trained CNN model VGG 16, Inception V3 and ResNet 50 to accurately distinguish normal samples from cancer samples using image features extracted from HCC histopathological images. However, the study still has the deficiency of a single classification sample category [27]. Sureshkumar, V et al. outlined various liver tumor detection algorithms and methods for liver tumor analysis. Proposed deep learning methods such as probabilistic neural networks to detect liver tumors and diagnose them through experimental results, and compare them with different methods. This study proposes a framework that can arrange CT images of the liver in expected or strange ways and distinguish unusual tumors with a reduced false positive rate and expanded accuracy [28]. However, the study still has insufficient sample types, which is not a deficiency of the diagnostic gold standard. In the study of Lin, HX, etc., 217 combinations of two-photon excitation fluorescence and second harmonic generation images were used to train a convolutional neural network based on the VGG-16 framework; the classification accuracy of liver cancer differentiation levels exceeded 90%. It is proved that the fusion of multiphoton microscope and deep learning algorithm can classify the differentiation of HCC, thereby providing an innovative computer-aided diagnosis method [29]. The research is relatively new, but there is still room for improvement in accuracy.

In order to ensure the authenticity and integrity of the data, we have developed a complete data acquisition scheme to obtain a higher quality data set for the classification experiment. Since histopathological images have more obvious and different characteristic information than other conventional natural images, the characteristics of histopathological images must be fully considered when establishing a deep learning model for histopathological image classification. Therefore, this paper adds the visual attention mechanism to the deep learning model. The attention mechanism increases the interpretation of the model by visualizing the attention, which can extract the feature information of medical images more effectively. On this basis, for the first time, we applied the SENet deep learning model to the intelligent classification study of all types of differentiated liver cancer histopathological images, and compared it with the four deep learning models of VGG16, ResNet50, ResNet50_CBAM, and SKNet. Finally, SENet was obtained the classification accuracy rate is 95.27%. The experiment has proved that the SENet deep learning model has a good application prospect in the intelligent classification of histopathological images. The effective classification of different levels of differentiation is convenient for doctors to formulate treatment plans more effectively, so that patients can get treatment in time. At the same time, it can effectively reduce the workload of medical workers and reduce the reading time of histopathological images of liver cancer. This also provides a certain reference value for deep learning in the research of intelligent classification of medical images.

2.1 Obtaining slice samples

The Institutional Review Committee of the Cancer Hospital Affiliated to Xinjiang Medical University approved this study with the ethical approval number K-2021050. Male and female patients who were treated in the Cancer Hospital of Xinjiang Medical University from August 2020 to August 2021 participated in this study with informed consent. Fresh liver cancer tissue samples were collected by professional surgeons. All patients did not receive hormone therapy, radiotherapy or chemotherapy before surgery. In the end, without collecting the personal information of these patients, we obtained 74 liver cancer tissue samples from these patients (including: 24 poorly differentiated pathological slice samples, 28 moderately differentiated pathological slice samples, and 22 highly differentiated pathological slice samples). The following is the specific process of slice sample preparation:

The first step is fixation. Fresh liver cancer specimens are quickly fixed with 10% neutral buffered formalin at room temperature for 18–24 hours. The second step is dehydration, by immersing the formalin-fixed sample in a series of alcohol solutions that increase with the alcohol concentration to remove water from it. After removing the dehydrated sample with an organic solvent, the third step of sample preparation is to bury it in molten paraffin. After cooling, each paraffin-embedded block (1.5x1.5x0.3cm³) was sliced into 4 µm consecutive slices, and each slice was placed on an albumin glass slide. The fourth step dyeing starts after the paraffin is dissolved. The histological section of liver cancer tissue is stained with H&E, which is the most commonly used staining method in medical diagnosis. Finally, 3 skilled pathologists with more than 10 years of pathology experience examined histological sections under a light microscope and selected representative H&E sections. The diagnosis results were consistent. Table 1 shows the basic information of all patients.

Table 1

Basic patient information
Differentiation	Age	Gender		Staging				Alcoholism		Smoking
Differentiation	Age	M	W	Ⅰ	Ⅱ	Ⅲ	Ⅳ	Y	N	Y	N
Poorly differentiated	35–50	7	3	4	1	3	2	2	8	5	5
	51–66	6	1	4	1	1	1	2	5	3	4
	67–82	4	3	2	4	1	0	0	7	0	7
Moderate differentiation	35–50	8	0	2	3	3	0	2	6	3	5
	51–66	12	2	8	4	2	0	6	8	8	6
	67–82	5	1	1	0	5	0	2	4	3	3
Well differentiated	35–50	2	0	1	0	0	1	2	0	2	0
	51–66	12	1	11	2	0	0	1	12	5	8
	67–82	4	1	1	1	3	0	0	5	2	3

2.2 Data set preparation

In this paper, 444 liver cancer histopathological images (There are 144 poorly differentiated histopathological images, 168 moderately differentiated histopathological images, and 132 highly differentiated histopathological images.) were obtained to form the research data set, and every image acquisition was done in the digital pathology scanner (Model: PRECICE 500B, Shanghai, China) with size of 1665×1393 pixels. The specificflow chart as is shown in Fig. 1.

The data set preparation process is as follows:

①Putting 5 liver cancer pathological slices with the same degree of differentiation into the card slot of the digital pathology scanner simultaneously to complete the high-definition scanning of the pathological slices.

②The dense area of Cancer cell in the high-definition image should be marked based on pathological expertise by a professional pathologist and cutting off 6 images evenly in a 2×3 matrix order in it after scanning, and then save the current images.

③Repeat step ② for the 5 pathological slices in the card slot in turn, and repeat the above process until all images are acquired.

④The pathology medical workers should verify again whether all the acquired images meet the classification of different degrees of differentiation, exclude all images without normal cells, and perform the next classification experiment after confirming that they are correct.

2.3 Experimental environment

All program codes in the study, are developed based on the Python language. The software and hardware conditions of the specific experimental environment are shown in Table 2.

Table 2

experimental environment
Category	Name	Version
CPU	IntelCore I5 9600KF	--
RAM	32G 2666 MHz	--
GPU	NVIDIA GTX 1070	--
Development language	Python	V3.7.0
Image processing library	Sci-kit image	V0.16.2
Data processing library	Numpy	V1.17.2
Deep learning framework 1	Sci-kit learn	V0.22
Machine learning library	TensorFlow	V2.1.0
Deep learning framework 2	Keras	V2.3.1

2.4 Experimental methods

2.4.1 Image enhancement

There are 444 original liver cancer histopathological images used in this paper, which are divided into three categories: poorly differentiated, moderately differentiated, and well differentiated. Because a large number of samples are needed for training to achieve better generalization in the subsequent deep learning model training. Obviously, due to raw data set used in this paper can not meeting training conditions, so we have to enhance the raw data set. However, each piece of the histopathological image of liver cancer contains rich pathological diagnosis information, and many image enhancement methods cannot well retain its pathological diagnosis information. At the same time, in order to restore the complete process of medical workers in actual reading as much as possible, this research adopts image rotation enhancement method to enhance the original image, the principle is as follows:

Image rotation: Suppose that $\left({x}_{0},{y}_{0}\right)$ is the coordinate after rotation, $\left(x,y\right)$ is the coordinate before rotation. $\left(m,n\right)$ is the center of rotation.$\theta$ is the angle of rotation.$\left(left,top\right)$ is the coordinates of the upper left corner of the rotated image. the realization formula is as shown in (1):

$$\left[\begin{array}{ccc}{x}_{0}& {y}_{0}& 1\end{array}\right]=\left[\begin{array}{ccc}x& y& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ -m& n& 1\end{array}\right]\left[\begin{array}{ccc}\text{cos}\theta & -\text{sin}\theta & 0\\ \text{sin}\theta & \text{cos}\theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ left& top& 1\end{array}\right] \left(1\right)$$

The rotation angle in this paper is set to 10°, and each original image is rotated at an angle of 10° each time. A picture is rotated 36 times, and a total of 36 images are obtained after each original image is rotated. The enhanced data set contains a total of 15,984 histopathological images.

2.4.2 Model training

In this paper, five deep learning models which are VGG16, ResNet50, ResNet_CBAM, SENet, and SKNet are selected for classification experiments [30, 31]. The above model and specific training process will be described in detail as follows, and the experimental frame diagram is shown in Fig. 2.

VGG16 network is relatively simple and does not have so many hyperparameters. It is a simple network that only needs to focus on building a convolutional layer. First, a 3×3 filter with a stride of 1 is used to construct a convolutional layer, and the padding parameter is the parameter in the same convolution. Then use a 2×2 filter with a stride of 2 to build the maximum pooling layer. Therefore, a major advantage of VGG network is that it does simplify the neural network structure. However, the amount of parameters is too large, which consumes a lot of computing resources, so it is not suitable for image learning with complex features [32, 33].

Traditional convolutional networks or fully connected networks will more or less have problems such as information loss and loss when information is transmitted, and at the same time, they can cause gradients to disappear or explode, making deep networks unable to train. ResNet solves this problem to a certain extent. Its main idea is to add the idea of Highway Network to the network, which allows to retain a certain proportion of the output in the previous network layer. The input information is directly detoured to the output to protect the integrity of information in ResNet. The entire network only needs to learn the part of the difference between input and output to simplify the learning objectives and difficulty [34, 35]. Based on the above advantages, this article chooses ResNet50 as one of the training models.

CBAM is a simple and effective convolutional neural network attention module. Given an intermediate feature map arbitrarily in the convolutional neural network, CBAM injects the Attention Mapping along two independent dimensions of the channel and space of the feature map, and then multiplies the Attention by the input feature map to perform Adaptive feature refinement on the input feature map. CBAM is an end-to-end universal module [36]. Based on the advantages of CBAM combined with the ResNet50 model, this paper selects the ResNet_CBAM model for classification experiments.

The idea of SENet is very simple, and it is easy to extend to the existing network structure. The SENet model proposes a new block structure, called the SE block, which uses each feature layer in the squeeze compression model structure, and uses excitation to capture feature channel dependence [37, 38], so the SE block can be combined with many existing models. The SE module mainly includes two operations, Squeeze and Excitation, which can be applied to any mapping ${F}_{tr}:X\to U$,$X\in {R}^{{H}^{\text{＇}}\times {Y}^{\text{＇}}\times {M}^{\text{＇}}}$,$U\in {R}^{H\times Y\times M}$, take convolution as an example, the kernel of convolution is $V=\left[{v}_{1},{v}_{2},\cdots {v}_{m}\right]$,where v_m represents the${m}^{th}$ convolution, ernel. Then output$U=\left[{u}_{1},{u}_{2},\cdots {u}_{M}\right]$;

$${u}_{m}={v}_{m}\text{*}X=\sum _{n=1}^{{M}^{\text{＇}}}{v}_{m}^{n}\text{*}{x}^{n}$$

2

* represents the convolution operation, and ${v}_{m}^{n}$represents an n channel 2-D convolution kernel, which inputs the spatial characteristics of a channel and learns the feature space relationship, but because the convolution results of each channel are added, the feature relationship of the channel is mixed with the spatial relationship learned by the convolution kernel. The SE block is to remove this confounding, so that the model directly learns the feature relationship of the channel.

SKNet and SENet are lightweight modules that can be directly embedded in the network. Feature maps of different convolution kernelsizes are fused through the Attention Mechanism, and the size of the Attention is based on the deterministic information extracted by convolution kernels of different sizes [39]. Intuitively, SKNet assimilate a soft attention mechanism into the network, so that the network can obtain information of different Receptive Field, which may become a network structure with better generalization ability [40].

After selecting the experimental model, this article randomly divides the ratio of the training set and the test machine according to the ratio of 8:2, and the specific division is shown in Table 3.

Table 3

Data partition
Differentiation type	Raw data	Enhanced data	Training set	Test set
Poorly differentiated	144	5184	4176	1008
Moderate differentiation	168	6048	4860	1188
Well differentiated	132	4752	3816	936

In this paper, all the deep model training batches are set to 300 Epoch, and the configuration of the optimizer and the setting of the learning rate are also crucial in model training. Based on the advantages of the (Stochastic gradient descent) SGD method in the application of large data sets, the training speed is very fast and as long as the noise is not very large, it can converge well. This paper chooses the SGD method as the optimizer configuration algorithm. The algorithm principle is as follows:

Suppose there are n samples in a batch, and randomly select one ${i}_{s}$. The model parameter is set to W, the cost function is set to $\text{J}\left(\text{W}\right)$, the gradient is $\varDelta \text{J}\left(\text{W}\right)$, and the learning rate is ${{\epsilon }}_{\text{t}}$, then the method update parameter expression is:

$${W}_{t+1}={W}_{t}-{\epsilon }_{t}{g}_{t} \left(3\right)$$

Among them; ${g}_{t}=\varDelta {J}_{{i}_{s}}\left({W}_{t};{X}^{\left({i}_{s}\right)};{X}^{\left({i}_{s}\right)}\right),{i}_{s}\in \text{1,2}\cdots n$ represents a gradient direction randomly selected, and ${W}_{t}$ represents the model parameter at the moment, $E\left({g}_{t}\right)=\varDelta J\left({W}_{t}\right)$, the expectation is the correct gradient descent.

In the above formula (6), the learning rate ε_t is one of the most important parameters in training, and the learning rate is set to 0.001 in the training of this article.

2.4 Evaluation index

There are many commonly used evaluation criteria for medical image classification, such as accuracy, error rate, false positive rate, true negative rate, false negative rate, true negative rate, etc. The evaluation indexes adopted in this paper include confusion matrix, Precision, recall, F1 Score, etc. These evaluation indexes can be used to evaluate the model in a very comprehensive and accurate way. The detailed description is given below.

1.Confusion matrix: All evaluation indexes are directly or indirectly related to the confusion matrix. All evaluation indexes can be directly calculated from the confusion matrix, and their specific forms are shown in Table 4:

Table 4

Confusion matrix
		Actual class
		positive class	negative class
Predicted class	positive class	True Positive (TP) False negative (FN)	False Positive (FP) True negative (TN)
	Negative class

In the confusion matrix shown in the table above:

·TP represents that the actual class is positive, and the predicted class of the model is also positive.

·FP represents that the predicted class is positive, but the actual class is negative. The actual class is inconsistent with the predicted class.

·FN represents that the predicted class is negative, but the actual class is positive. The actual class is inconsistent with the predicted class.

·TN represents that the actual class is negative, and the predicted class of the model is also Negative.

By using the confusion matrix, we can intuitively see the classification of a model in positive and negative categories.

2.Precision and recall: Formula (4) and formula (5) are used to calculate the precision and recall of the positive class:

Precision represents the proportion of samples whose actual class is positive in the samples whose predicted class is positive; The recall represents the percentage of samples that are successfully predicted in a truly positive sample by the model.

3.F1 Score: F1 Score is calculated by Formula (6):

F1 Score takes both the precision and recall of the classification model into account. It can be regarded as a harmonic average of precision and recall of model. It has a maximum value of 1 and a minimum value of 0.

3.1 The classification results

In this section, the classification results of different models are analyzed in detail. The training accuracy curves of all classification models under the same training conditions including optimizer algorithm, loss function, learning rate, Epoch and other conditions are shown in Fig. 4. Table 5 shows the classification accuracy of different differentiation types.

In Fig. 3(a) and Fig. 3(c), we respectively show the accuracy curves of training set and test set in the five classification models. In order to observe the convergence process in the training process more intuitively, we intercept respectively the accuracy curves of training set and test set in 160–300 Epoch, as shown in Fig. 3(b) and Fig. 3(d).In the training set, the models except VGG16 began to converge when the Epoch was 50 and completely converged when the Epoch was 120, while VGG16 model tended to converge completely when the Epoch was 215.In the test set, the models began to converge at 150 Epoch, and the VGG16 model began to converge at 220 Epoch. This also proves that the simple network model focusing on the construction of convolutional layer is not suitable for the classification task of histopathological images of liver cancer.

Table 5

Classification accuracy of histopathological images of liver cancer unit:%
Classifier	Poorly differentiated	Moderate differentiation	Well differentiated	Accuracy
VGG16	77.98	79.12	73.08	76.95
ResNet50	96.43	90.99	95.94	94.22
ResNet50_CBAM	95.24	94.28	94.76	94.73
SENet	98.41	92.59	95.30	95.27
SKNet	96.43	93.94	95.30	95.15

Through horizontal comparison of the accuracy of different models in Table 5, it was found that SENet model achieved the best classification effect with a classification accuracy of 95.27%. The ResNet50 model and resnet50_CPAM model achieved respectively 95.94% accuracy in the classification of well differentiated histopathological images and 94.28% accuracy in those of moderate differentiation histopathological images. It also confirmed the superiority of ResNet50 model in the classification of images with relatively simple features. At the same time, the SENet model achieved a 98.41% classification accuracy in the classification of poorly differentiated histopathological images.

3.2 Model evaluation

In this paper, the confusion matrix of the training model is used as the evaluation index of the model. From the confusion matrix of the model (0 represents poorly differentiated, 1 represents moderate differentiation, 2 represents well differentiated) we can very clearly see the liver cancer histopathological image classification accuracy of different models in different differentiation degree. From the confusion matrix of different models, it can be found that the VGG16 model has a general classification effect on the images with three different degrees of differentiation. It is worth noting that, in the moderate differentiation image classification, the number of images which are misjudged as poor differentiation was much larger than the number of images which are misjudged as well differentiation. The main cause of this phenomenon lies in the relatively complex image characteristics of histopathology images. So the VGG16 model whose structure is relatively simple can not learn more from the image characteristics and it causes the classification effect is poor. Since ResNet50 and ResNet50_CBAM models have deeper structures, their image classification effect with different degrees of differentiation is better than VGG16 model.ResNet50_CBAM model is slightly better than ResNet50 model in the classification of moderately differentiated images, and the number of correct classification of poorly differentiated images and well differentiated images are relatively consistent. SENet and SKNet models have achieved good classification results in the classification of three types of images, and they are completely consistent in the classification of well differentiated images. SENet model has more correct judgments in the classification of poorly differentiated images than SKNet model, while SKNet model has a slight advantage in the classification of moderate differentiation images. The main reason for this phenomenon is that SENet model pays more attention to strengthening useful information and compressing useless information, so it is more suitable for histopathological images with complex features.

Through further analysis and calculation of the confusion matrix of different models, we can use more representative Precision, Recall and F1 Score to prove the reliability and generalization ability of different models. These indexes can comprehensively describe the performance of the model from different perspectives. Table 6 shows the evaluation indexes of different models in detail.

Table 6

Evaluation indexes of different models unit:%
Differentiation type	Metrics	VGG16	ResNet50	ResNet50_CBAM	SENet	SKNet
Poorly differentiated	Precision	70.43	90.08	90.48	92.02	92.22
	Recall	77.98	96.43	95.24	99.41	96.43
	F1 Score	74.01	93.15	92.80	95.11	94.28
Moderate differentiation	Precision	77.56	96.60	98.42	98.21	96.21
	Recall	79.12	90.99	94.28	92.59	93.94
	F1 Score	78.33	93.71	96.30	95.32	95.06
Well differentiated	Precision	85.07	96.15	95.07	95.50	97.17
	Recall	73.08	95.94	94.76	95.30	95.30
	F1 Score	78.62	96.04	94.92	95.40	96.22

Through the horizontal comparison of the above five classification models and their three different evaluation indexes, it was found that the VGG16 model did not achieve ideal results in any classification of the differentiation degree. In the classification of poorly differentiated histopathological images, SENet model achieved good scores both in recall and F1 Score, and its precision was next only to SKNet model. This indicates that SENet model is more suitable for distinguishing poorly differentiated histopathological images with more complex image features. We analyze the main reason for this phenomenon and think this is because the difference in image features of histopathological images is mainly reflected in channel dimension. While SENet model can pay more attention to channel features with the most information and suppress the channels features that are not important. Resnet50_CBAM model achieved the best score in the classification of moderate differentiation histopathological images in all three indexes. This is thanks to two independent sub-modules of CBAM, Channel Attention Module (CAM) and Spartial Attention Module (SAM). It is more suitable for moderate differentiation images that appear to be clearly abnormal from normal cells, but retain traces of some tissue source. The SKNet model achieved the best score in Precision and F1 Score for image classification of well differentiated histopathological images. For well differentiated histopathological images, they contain more normal cells, and their image features are relatively simple and easy to learn. SKNet carries out an attention mechanism for the convolutional nucleus, that is, the network selects the appropriate convolutional nucleus by itself, without too many changes to the overall structure of the model. Therefore, SKNet model is more stable and has better generalization ability in the classification of well differentiated histopathological images.

In this paper, a complete data acquisition scheme was developed for the intelligent classification of histopathological images with different degrees of differentiation, and histopathological images of liver cancer with full types and degrees of differentiation were obtained. Because histopathological images have more obvious and different feature information than other conventional natural images, the characteristics of histopathological images must be fully considered when establishing the deep learning model of histopathological image classification. Therefore, this article adds visual attention mechanism to some deep learning model, which can increase interpretation forms of the model through attention visualization and can more effectively extract the characteristic information of the medical image. Five different deep learning classification models are applied to collect the data set and evaluate model. The experimental results show that the SENet model has achieved the best classification effect with an accuracy of 95.27%. The model also has good reliability and generalization ability.

This study also proves that deep learning has great application value in solving the time-consuming and laborious problems existing in traditional manual film reading, and it has certain practical significance for the intelligent classification research of other cancer histopathological images.

Ethics approval and consent to participate

The Institutional Review Committee of the Cancer Hospital Affiliated toXinjiang Medical University approved this study with the ethical approval numberK-2021050. All subjects in this study gave informed consent.The data and materials used in this study were collected, processed, and analyzed in accordance with appropriate guidelines and regulations.

Consent for publication

(Not applicable)

Availability of data and materials

The data that support the findings of this study are available from the Cancer Hospital of Xinjiang Medical University but restrictions apply to the availability of these data, which were used under license for the current study. We do not have the right to disclose data and so are not publicly available.Data are however available from the authors upon reasonable request and with permission of the Cancer Hospital of Xinjiang Medical University.

Competing interests

The authors declare that they have no relevant financial and/or non-financial conflict of interests in this article and no potential conflicts of interest to disclose.

Funding

This research was funded by the special scientific research project for young medical science (2019Q003) and Karamay City Key R&D Project - Molecular Mechanism and Application of DNA Methylation Liquid Biopsy in the "Prevention, Diagnosis and Treatment" of Malignant Tumors.

Authors’ contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Chen Chen, Cheng Chen and MingruiMa. The first draft of the manuscript was written by Chen Chen and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

This work was supported by the special scientific research project for young medical science (2019Q003) and Karamay City Key R&D Project - Molecular Mechanism and Application of DNA Methylation Liquid Biopsy in the "Prevention, Diagnosis and Treatment" of Malignant Tumors.

[1] D. M. Parkin, F. Bray, J. Ferlay, and P. Pisani, “Global cancer statistics, 2002,” Ca-a Cancer Journal for Clinicians, vol. 55, no. 2, pp. 74-108, Mar-Apr, 2005.

[2] A. Jemal, F. Bray, M. M. Center, J. Ferlay, E. Ward, and D. Forman, “Global Cancer Statistics,” Ca-a Cancer Journal for Clinicians, vol. 61, no. 2, pp. 69-90, Mar-Apr, 2011.

[3] J. Ferlay, H. R. Shin, F. Bray, D. Forman, C. Mathers, and D. M. Parkin, “Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008,” International Journal of Cancer, vol. 127, no. 12, pp. 2893-2917, Dec, 2010.

[4] A. Jemal, “Global Cancer Statistics (vol 61, pg 69, 2011),” Ca-a Cancer Journal for Clinicians, vol. 61, no. 2, pp. 134-134, Mar-Apr, 2011.

[5] N. M. Niehoff, E. C. Zabor, J. Satagopan, A. Widell, T. R. O'Brien, M. Zhang, N. Rothman, T. K. Grimsrud, S. K. Van den Eeden, and L. S. Engel, “Prediagnostic serum polychlorinated biphenyl concentrations and primary liver cancer: A case-control study nested within two prospective cohorts,” Environmental Research, vol. 187, pp. 11, Aug, 2020.

[6] S. Parikh, and D. Hyman, “Hepatocellular cancer: A guide for the internist,” American Journal of Medicine, vol. 120, no. 3, pp. 194-202, Mar, 2007.

[7] L. Ali, I. Wajahat, N. A. Golilarz, F. Keshtkar, and S. A. C. Bukhari, “LDA-GA-SVM: improved hepatocellular carcinoma prediction through dimensionality reduction and genetically optimized support vector machine,” Neural Computing & Applications, pp. 10.

[8] D. P. Sheng, F. C. Xu, Q. Yu, T. T. Fang, J. J. Xia, S. R. Li, and X. Wang, “A study of structural differences between liver cancer cells and normal liver cells using FTIR spectroscopy,” Journal of Molecular Structure, vol. 1099, pp. 18-23, Nov, 2015.

[9] H. M. Li, “Microcirculation of liver cancer, microenvironment of liver regeneration, and the strategy of Chinese medicine,” Chinese Journal of Integrative Medicine, vol. 22, no. 3, pp. 163-167, Mar, 2016.

[10] F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, “Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” Ca-a Cancer Journal for Clinicians, vol. 68, no. 6, pp. 394-424, Nov-Dec, 2018.

[11] H. T. Liao, T. Y. Xiong, J. J. Peng, L. Xu, M. H. Liao, Z. Zhang, Z. R. Wu, K. F. Yuan, and Y. Zeng, “Classification and Prognosis Prediction from Histopathological Images of Hepatocellular Carcinoma by a Fully Automated Pipeline Based on Machine Learning,” Annals of Surgical Oncology, vol. 27, no. 7, pp. 2359-2369, Jul, 2020.

[12] X. Cui, T. Liu, X. Xu, Z. Zhao, Y. Tian, Y. Zhao, S. Chen, Z. Wang, Y. Wang, D. Hu, S. Fu, G. Shan, J. Sun, K. Song, and Y. Zeng, “Label-free detection of multiple genitourinary cancers from urine by surface-enhanced Raman spectroscopy,” Spectrochim Acta A Mol Biomol Spectrosc, vol. 240, pp. 118543, Oct 15, 2020.

[13] B. B. Xia, H. Y. Jiang, H. L. Liu, and D. H. Yi, “A Novel Hepatocellular Carcinoma Image Classification Method Based on Voting Ranking Random Forests,” Computational and Mathematical Methods in Medicine, pp. 8, 2016.

[14] C. L. Sun, A. Xu, D. Liu, Z. W. Xiong, F. Zhao, and W. P. Ding, “Deep Learning-Based Classification of Liver Cancer Histopathology Images Using Only Global Labels,” Ieee Journal of Biomedical and Health Informatics, vol. 24, no. 6, pp. 1643-1651, Jun, 2020.

[15] M. Barone, A. Di Leo, C. Sabba, and A. Mazzocca, “The perplexity of targeting genetic alterations in hepatocellular carcinoma,” Medical Oncology, vol. 37, no. 8, pp. 5, Jul, 2020.

[16] B. Sakthisaravanan, and R. Meenakshi, “OPBS-SSHC: outline preservation based segmentation and search based hybrid classification techniques for liver tumor detection,” Multimedia Tools and Applications, pp. 27.

[17] S. Naeem, A. Ali, S. Qadri, W. K. Mashwani, N. Tairan, H. Shah, M. Fayaz, F. Jamal, C. Chesneau, and S. Anam, “Machine-Learning Based Hybrid-Feature Analysis for Liver Cancer Classification Using Fused (MR and CT) Images,” Applied Sciences-Basel, vol. 10, no. 9, pp. 22, May, 2020.

[18] V. Hemalatha, and C. Sundar, “Automatic liver cancer detection in abdominal liver images using soft optimization techniques,” Journal of Ambient Intelligence and Humanized Computing, pp. 10.

[19] N. A. Mat-Isa, M. Y. Mashor, and N. H. Othman, “An automated cervical pre-cancerous diagnostic system,” Artificial Intelligence in Medicine, vol. 42, no. 1, pp. 1-11, Jan, 2008.

[20] H. T. Liao, Y. X. Long, R. J. Han, W. Wang, L. Xu, M. H. Liao, Z. Zhang, Z. R. Wu, X. Q. Shang, X. F. Li, J. J. Peng, K. F. Yuan, and Y. Zeng, “Deep learning-based classification and mutation prediction from histopathological images of hepatocellular carcinoma,” Clinical and Translational Medicine, pp. 8.

[21] F. Kanavati, G. Toyokawa, S. Momosaki, M. Rambeau, Y. Kozuma, F. Shoji, K. Yamazaki, S. Takeo, O. Iizuka, and M. Tsuneki, “Weakly-supervised learning for lung carcinoma classification using deep learning,” Scientific Reports, vol. 10, no. 1, pp. 11, Jun, 2020.

[22] H. Sun, X. X. Zeng, T. Xu, G. Peng, and Y. T. Ma, “Computer-Aided Diagnosis in Histopathological Images of the Endometrium Using a Convolutional Neural Network and Attention Mechanisms,” Ieee Journal of Biomedical and Health Informatics, vol. 24, no. 6, pp. 1664-1676, Jun, 2020.

[23] A. Kumar, S. K. Singh, S. Saxena, K. Lakshmanan, A. K. Sangaiah, H. Chauhan, S. Shrivastava, and R. K. Singh, “Deep feature learning for histopathological image classification of canine mammary tumors and human breast cancer,” Information Sciences, vol. 508, pp. 405-421, Jan, 2020.

[24] L. Q. Zhou, J. Y. Wang, S. Y. Yu, G. G. Wu, Q. Wei, Y. B. Deng, X. L. Wu, X. W. Cui, and C. F. Dietrich, “Artificial intelligence in medical imaging of the liver,” World Journal of Gastroenterology, vol. 25, no. 6, pp. 672-682, Feb, 2019.

[25] J. Li, Y. R. Wu, N. Y. Shen, J. W. Zhang, E. L. Chen, J. Sun, Z. Q. Deng, and Y. C. Zhang, “A fully automatic computer-aided diagnosis system for hepatocellular carcinoma using convolutional neural networks,” Biocybernetics and Biomedical Engineering, vol. 40, no. 1, pp. 238-248, Jan-Mar, 2020.

[26] S. A. Azer, “Deep learning with convolutional neural networks for identification of liver masses and hepatocellular carcinoma: A systematic review,” World Journal of Gastrointestinal Oncology, vol. 11, no. 12, pp. 1218-1230, Dec, 2019.

[27] L. Q. Lu, and B. J. Daigle, “Prognostic analysis of histopathological images using pre-trained convolutional neural networks: application to hepatocellular carcinoma,” Peerj, vol. 8, pp. 25, Mar, 2020.

[28] V. Sureshkumar, V. Chandrasekar, R. Venkatesan, and R. K. Prasad, “Improved performance accuracy in detecting tumor in liver using deep learning techniques,” Journal of Ambient Intelligence and Humanized Computing, pp. 8.

[29] H. X. Lin, C. Wei, G. X. Wang, H. Chen, L. S. Lin, M. Ni, J. X. Chen, and S. M. Zhuo, “Automated classification of hepatocellular carcinoma differentiation using multiphoton microscopy and deep learning,” Journal of Biophotonics, vol. 12, no. 7, pp. 7, Jul, 2019.

[30] R. V. M. da N?brega, P. P. Rebou?as, M. B. Rodrigues, S. P. P. da Silva, C. Dourado, and V. H. C. de Albuquerque, “Lung nodule malignancy classification in chest computed tomography images using transfer learning and convolutional neural networks,” Neural Computing & Applications, vol. 32, no. 15, pp. 11065-11082, Aug, 2020.

[31] H. Q. Zhang, L. Han, K. Chen, Y. L. Peng, and J. L. Lin, “Diagnostic Efficiency of the Breast Ultrasound Computer-Aided Prediction Model Based on Convolutional Neural Network in Breast Cancer,” Journal of Digital Imaging, pp. 6.

[32] B. S. A. Gazioglu, and M. E. Kamasak, “Effects of objects and image quality on melanoma classification using deep neural networks,” Biomedical Signal Processing and Control, vol. 67, pp. 9, May, 2021.

[33] M. Takeuchi, T. Seto, M. Hashimoto, N. Ichihara, Y. Morimoto, H. Kawakubo, T. Suzuki, M. Jinzaki, Y. Kitagawa, H. Miyata, and Y. Sakakibara, “Performance of a deep learning-based identification system for esophageal cancer from CT images,” Esophagus, vol. 18, no. 3, pp. 612-620, Jul, 2021.

[34] Y. Celik, M. Talo, O. Yildirim, M. Karabatak, and U. R. Acharya, “Automated invasive ductal carcinoma detection based using deep transfer learning with whole-slide images,” Pattern Recognition Letters, vol. 133, pp. 232-239, May, 2020.

[35] N. Wu, J. Phang, J. Park, Y. Q. Shen, Z. Huang, M. Zorin, S. Jastrzebski, T. Fevry, J. Katsnelson, E. Kim, S. Wolfson, U. Parikh, S. Gaddam, L. L. Y. Lin, K. Ho, J. D. Weinstein, B. Reig, Y. M. Gao, H. Toth, K. Pysarenko, A. Lewin, J. Lee, K. Airola, E. Mema, S. Chung, E. Hwang, N. Samreen, S. G. Kim, L. Heacock, L. Moy, K. Cho, and K. J. Geras, “Deep Neural Networks Improve Radiologists Performance in Breast Cancer Screening,” Ieee Transactions on Medical Imaging, vol. 39, no. 4, pp. 1184-1194, Apr, 2020.

[36] J. Niu, H. Li, C. Zhang, and D. A. Li, “Multi-scale attention-based convolutional neural network for classification of breast masses in mammograms,” Medical Physics, pp. 15.

[37] F. Shahidi, S. M. Daud, H. Abas, N. A. Ahmad, and N. Maarop, “Breast Cancer Classification Using Deep Learning Approaches and Histopathology Image: A Comparison Study,” Ieee Access, vol. 8, pp. 187531-187552, 2020.

[38] X. Li, X. Shen, Y. X. Zhou, X. H. Wang, and T. Q. Li, “Classification of breast cancer histopathological images using interleaved DenseNet with SENet (IDSNet),” Plos One, vol. 15, no. 5, pp. 13, May, 2020.

[39] A. A. Jeny, A. N. M. Sakib, M. S. Junayed, K. A. Lima, I. Ahmed, and M. B. Islam, "SkNet: A Convolutional Neural Networks Based Classification Approach for Skin Cancer Classes," ICCIT 2020 - 23rd International Conference on Computer and Information Technology, Proceedings.

[40] M. Xu, Z. Liu, Z. Wang, L. Sun, and Z. Liang, "The diagnosis of Alzheimer's disease based on enhanced residual neutral network," Proceedings - 2019 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2019. pp. 405-411.

No competing interests reported.

Classification of Multi-Differentiated Liver Cancer Pathological Images Based on Deep Learning Attention Mechanism

Status:

Version 1

Abstract

Purpose

Methods

Results

Conclusions

Figures

1 Background

2 Materials And Methods

2.1 Obtaining slice samples

2.2 Data set preparation

2.3 Experimental environment

2.4 Experimental methods

2.4.1 Image enhancement

2.4.2 Model training

2.4 Evaluation index

3 Results Analysis And Discussion

3.1 The classification results

3.2 Model evaluation

4 Conclusion

Declarations

References

Additional Declarations

Status:

Version 1