Tongue Image Quality Assessment Based on Deep Convolutional Neural Network

doi:10.21203/rs.3.rs-91687/v1

Download PDF

Research article

Tongue Image Quality Assessment Based on Deep Convolutional Neural Network

https://doi.org/10.21203/rs.3.rs-91687/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 05 May, 2021

Read the published version in BMC Medical Informatics and Decision Making →

You are reading this latest preprint version

Background: With the wide application of digital tongue diagnosis instrument, massive tongue images will be produced. Adequate image quality is the prerequisite to ensure accurate tongue image analysis. In the process of tongue image collection, improper operation may lead to many poor-quality images (fogging, underexposure, overexposure, blurred focus, wrong tongue posture, etc.), which seriously affect the image processing and the accuracy of image analysis. However traditional pattern recognition is difficult to evaluate the quality of tongue images by extracting features and manual removal of tongue images with bad quality consumes a lot of labor and has a high error rate. In this research, we utilized a deep convolutional neural network to automatically select bad quality tongue images.

Methods: The present study was conducted to identify the most appropriate CNN model for Tongue Image Quality Assessment based on deep CNN. The CNN model was evaluated by using Residual neural network and compared with VGGNet and DenseNet. Evaluation metrics such as accuracy, precision, recall, and F1-score were used for CNN model performance.

Results: A detection model is established for tongue image quality control based on deep residual network, with an average accuracy of 99.04%, accuracy of 99.05%, recall of 99.04%, and F1-score of 99.05%, which can be used for quality screening of massive tongue images.

Conclusions: Our research findings demonstrate various CNN models in the decision-making process for the selection of tongue image quality assessment and prove that applying deep learning methods, specifically deep CNN, to evaluate bad quality tongue images is feasible.

Medical Informatics

Tongue diagnosis

Quality assessment

Deep learning

ResNet

DenseNet

Tongue inspection with a long history is the most intuitive, simple and effective diagnostic methods in traditional Chinese medicine (TCM) [1, 2]. However, traditional tongue diagnosis is affected by factors such as the external environment and doctors’ subjective clinical experience. Digital tongue images can objectively reflect tongue characteristics. Computerized tongue diagnosis systems are gradually being accepted by more and more clinicians as a medical application for health assessment and diagnosis of diseases, such as type 2 diabetes mellitus [3–6], breast cancer [7], colorectal cancer [8], appendicitis [9], gastritis [10], etc.

With the popularization of clinical application of digital tongue picture, massive tongue images data will be produced. The quality of tongue image cannot be ignored, and it is closely related to the accuracy of tongue diagnosis and clinical treatment [1].

The quality of tongue image is an important prerequisite for clinical application of tongue diagnosis, see Fig. 1.

A normal quality tongue image should have the following characteristics: the tongue surface is centered, no fog, no light leakage, no overexposure or underexposure, no focus unevenness, and complete tongue surface, see Fig. 2.We found that in the process of using tongue diagnosis equipment, despite having received standardized tongue image acquisition training, abnormal tongue images are still common in the clinical tongue image acquisition process, mainly from two aspects: from the operators and participants.

On the one hand, due to the operator, there is a standard setting for the abnormal parameter settings of the tongue diagnosis instrument, such as the size of the aperture and the shutter speed. If the tongue is dark and cannot truly reflect the color of the tongue, there will be underexposure (Fig. 2-B-a); If the parameter settings of the tongue diagnosis instrument are changed, if the image of the tongue is too bright, this is an overexposure (Fig. 2-B-b); or the operator may not operate correctly, this may produce tongue images with blurry focus (Fig. 2-B-c), or light leakage (Fig. 2-B-d), etc.

On the other hand, from the participants. The breathe and fog when collecting tongue image (Fig. 2-B-e), does not have good tongue extension training, and has abnormal tongue extension posture (Fig. 2-B-g, Fig. 2-B-h), eat food before collecting tongue image, especially food with pigment, appear stained tongue coating, see (Fig. 2-B-a) and other tongue images of foreign objects on the tongue (Fig. 2-B-f).

The images of these situations will affect the process and results of data analysis, and will severely interfere with the results of subsequent tongue image analysis. These tongue images bring interference bias to the later analysis and processing of tongue image data, which will cause misclassification of pattern differentiation of syndrome in TCM and erroneous clinical decision on prevention, diagnosis and treatment of disease.

The evaluation of tongue image quality has also attracted more and more attention from researchers. Image Quality Assessment (IQA) mainly evaluates the quality of images. Both manual and automatic methods can be used to evaluate images quality. At present, the main approach to removing pictures in these situations is manual. The manual method is based on TCM diagnosis and clinical experts' perception assessment of the quality of tongue image, including the sharpness of the tongue image, tongue extension posture, noise, etc. This method is costly, labor-intensive, error-prone and inefficient, and cannot be automated in real time. Therefore, an efficient and accurate quality control model of tongue image is essential to clinical use of tongue diagnosis instruments.

It is easy for human viewers to perceive a significant difference in the quality of a set of tongue images, while it is not so easy for a computer to automatically recognize tongue images of abnormal quality. In recent years, with the improvement of deep neural network models and the development of deep learning algorithms, the classification accuracy and efficiency of image technology based on CNNs have been greatly improved. It has been widely used in image segmentation, image classification, face recognition, etc., and has become the current Mainstream algorithm [11, 12]. Convolutional Neural Networks (CNNs), a representative of deep learning methods, has gradually become a research hotspot in the field of tongue diagnosis objectification.

In this research, we focus on the model construction method of automatically rejecting unqualified tongue images based on the deep CNN model to evaluate the quality of tongue images.

2.1 Tongue image acquisition and preprocessing

To make a relatively stable tongue image dataset, tongue images were collected by uniformed equipment (TFDA-1, equipment number: ER17005-201809) developed by Shanghai University of Traditional Chinese Medicine [13]. This equipment has applied for a medical device registration certificate and mainly includes CCD equipment, standard D50 light source, hood, base, curved reflector, see Fig. 3. The color temperature of the LED lamp is 5003K, and the color rendering index is 97. The device has a high color rendering index LED light source, and a curved reflector is set in front of it to ensure the uniformity of the illumination of each part when the tongue image is collected, which effectively solves the stability and authenticity of the tongue image collection process.

Three steps were performed in this study to ensure the interpretation principles of tongue image quality. First, professionals unanimously agree the tongue image diagnosis criteria for good quality and bad quality. Second, at least 8 of 10 experts confirm that the same label is included in the dataset and all 2531 tongue images were labeled to ‘‘good” or ‘‘bad” folder by two professionals. The other eight professionals checked the labeled folders respectively. Third, if inconsistency occurs, the controversial tongue images are not included in this study. Unanimously agreed tongue images were included in the dataset for building CNN model.

According to the above interpretation principles, the dataset, contained 1238 bad quality tongue images and 1293 good quality tongue images, was constructed. Among them, 1,238 bad quality tongue images were captured in clinical research centers, including 189 cases of underexposure, 192 cases of overexposure, 168 cases of fogging, 190 cases of light leakage, 146 cases of blurred focus, 197 cases of tongue posture errors, and 156 cases of foreign objects on the tongue. Remaining 1293 tongue images with no fogging, no underexposure, and no overexposure were selected as good quality tongue images. Raw tongue image size is 5568 × 3711 pixels. In addition, in order to control the noise of the face and background areas around the tongue region, all available raw tongue images were isolated and cropped manually to the same size (400 × 400 pixels) with tongue region before model training. Finally, we constructed the dataset, including good quality tongue region images and bad quality tongue region images.

The schematic of the process of acquired raw tongue image dataset construction and tongue region image preprocessing were shown in Fig. 4.

2.2 CNN network architecture

In order to evaluate the tongue image quality, this paper uses the deep convolutional neural network model ResNet (152) based on the Residual Network (Residual Network, ResNet) [11]. It improves the efficiency of information dissemination by adding a shortcut connection to the nonlinear convolutional layer. The residual network increases the depth of the neural network by connecting multiple residual units. This method has improved the prediction accuracy and the training speed and performs more excellent than traditional neural network model. The schematic diagram of the tongue image quality assessment based on ResNet152 model was shown in Fig. 5.

Two data layers are used for training data respectively. Put the tongue image into the CNN network for training. The input tongue image quality inspection image is first propagated to the output layer through the forward direction, and then the output is obtained, and the error between the output value and the true value is calculated. The calculated error is propagated back layer by layer through backpropagation according to the principle of minimum error, and the weight between the networks is automatically updated during the backpropagation process.

2.3 Model training, validation, and testing

CNN models were trained using PyTorch(version1.3.1) Python(version 3.6) framework on Ubuntu system(version 14.04).CNN models were developed on the hardware DEVTOP AIX4750 produced by OMNISKY with 4 NVIDIA GTX 1080Ti GPU, i7-6850 K CPU, 64 GB DDR4 RAM, and 512G SDD. The initialized weights of CNN network were pretrained on ImageNet datasets. Then, the CNN network was fine-tuned on tongue image dataset for 30 epochs using a batch size of 32. We use stochastic gradient descent (SGD) method as an optimizer with learning rate of 0.01 and momentum of 0.9.

Then, 80% of the dataset is divided into training set, 10% is divided into validation set, and the remaining 10% is divided into testing set. For each set of parameters, iterate 30 epochs for model training on the training set, input the validation set to the trained model, observe the training process and the changes in the loss and accuracy on the verification set. Based on the fine-tuning results, select a set of parameters with relatively good results on the validation set, and use the model trained under the parameters as the final model for testing. Input the test set into the model and get the test result. Finally, a tongue image quality detection model with high accuracy and not over-fitting is obtained.

To prove the effectiveness of the CNN model in the present research, three types of experiments were conducted.

First, VGG16 was used for comparative experiments. VGGNet is a deep CNN developed by researchers from the Visual Geometry Group of Oxford University and Google DeepMind [14]. VGG16, which contains 13 convolutional layers and 3 fully connected layers, is used to improve performance by continuously deepening the network structure. For comparative analyses and reducing training time, the VGG16 has also been pretrained on ImageNet datasets and the training parameter settings were in accordance with ResNet152 aforementioned.

Then, DenseNet169 was also used for comparative experiments. Crucially, in contrast to ResNets, ResNets bypass signal from one layer to the next via identity connections combine features by concatenating them. Due to the design of the dense connectivity pattern with dense block and transition layer, DenseNets also alleviate the vanishing-gradient problem, and achieve high performance in competitive object recognition benchmark tasks[15]. The DensNet169 have also been pretrained on ImageNet datasets.

Finally, testing in new datasets by other clinical research center with different kinds of tongue diagnosis instrument. The new testing dataset including 700 tongue images acquired by TFDA-1 and TDA-1 instrument, were constructed. They have different CCD and illumination. Besides, we also cropped each raw tongue image into the tongue region image with same size. The tongue image dataset was also classified into 350 bad quality tongue images and 350 good quality tongue images by the same professionals as mentioned above in Sect. 2.1. 350 bad quality tongue images including 7 categories (example: underexposure, overexposure, fogging, light leakage, blurred focus, wrong tongue posture, and foreign objects on the tongue, etc. 50 tongue images of each type). Then, we use all tongue images in this new dataset to build the aforementioned 3 CNN models. The main procedures were shown in Fig. 4.

2.4 Model evaluation metrics

Accuracy is one of the most commonly used model evaluation metrics in machine learning. It is the average classification effect describing the overall performance of all categories. In addition, this study also uses three metrics, Precision, Recall, and F1-Score, to evaluate and analyze the performance of the model. The accuracy (Eq. (1)), precision (Eq. (2)), recall (Eq. (3)) and F1-score (Eq. (4)) were used to evaluate the performance of the CNN model. True positive (TP) means the number of correctly classified as bad quality tongue images, true negative (TN) represents the number of correctly classified as good quality tongue images, false positive (FP) means the number of incorrectly classified as bad quality tongue images, and false negative (FN) represents the number of incorrectly classified as good quality tongue images. It is, therefore, a complementary metric to the overall accuracy. Macro-averaging is used for models with more than 2 target classes. Macro-averaging is performed by first computing the precision, recall, F1-score of each class, and then taking the average of all precisions, recalls, and F1-scores.

In the current study, we use 10-fold cross-validation for evaluating the performance of the model. First, the tongue image dataset was randomly divided into 3 subsets, 80% for training, 10% for validation, and rest 10% for testing. Then, validate the model on the one reserved subset and record the evaluation metrics and calculate the testing performance of the 10 testing Folders’ evaluation metrics. Finally, the average metrics and standard deviation (SD) of the 3 CNN models’ accuracy, precision, recall, and F1-score were calculated using Python 3.6 software.

3.1 Testing results on tongue image dataset with ResNet152

The accuracy of the model in the training set and the validation set is close to 100%, and the training loss gradually decreases as the epoch increases. The tongue image quality classification results by ResNet152 architecture on 2531 raw tongue images are shown in Table 1. As expected, the classification performance of the ResNet152 model is still stable and excellent. The macro average accuracy is 98.82%, which proves the effectiveness of the CNN method. the macro precision is 98.83% and macro recall is 98.81%, revealing that the ResNet152 models have a relatively high level of precision and recall.

Table 1

Results of the ResNet152 architecture.
Class	Pre	Rec	F1-score	ACC
Fold1	99.21%	99.21%	99.21%	99.21%
Fold 2	98.04%	98.02%	98.03%	98.03%
Fold 3	98.45%	98.41%	98.43%	98.43%
Fold 4	98.83%	98.81%	98.82%	98.82%
Fold 5	98.42%	98.42%	98.42%	98.43%
Fold 6	98.81%	98.83%	98.82%	98.82%
Fold 7	99.21%	99.21%	99.21%	99.21%
Fold 8	98.83%	98.81%	98.82%	98.82%
Fold 9	99.24%	99.19%	99.22%	99.21%
Fold 10	99.24%	99.19%	99.22%	99.21%
Average(SD)	98.83%(0.42%)	98.81%(0.42%)	98.82%(0.42%)	98.82%(0.41%)

3.2 Compare With Vgg16 And Densenet 169

To further study whether the CNN architecture may influence the experimental results, VGG16 and DenseNet169 were used for comparison analysis. The results are shown in Table 2 and Table 3. The macro average accuracy is 96.89% by VGG16 and 98.82% DenseNet169 on the same testing tongue images subset, respectively. However, ResNet152 and DenseNet169 architecture can increase the accuracy of bad quality tongue image classification nearly by 2% on the same testing tongue images subset. As we expected, the macro average precision is 96.91% and recall is 96.88% by VGG16, and the macro average precision is 98.83% and recall is 98.82% by DenseNet, indicating that the models have impressively high precision and recall.

Table 2

Results of the VGG16 architecture.
	Pre	Rec	F1-score	ACC
Fold1	95.27%	95.27%	95.27%	95.28%
Fold 2	96.50%	96.43%	96.47%	96.46%
Fold 3	95.35%	95.24%	95.29%	95.28%
Fold 4	96.84%	96.87%	96.86%	96.85%
Fold 5	96.85%	96.85%	96.85%	96.85%
Fold 6	98.04%	98.02%	98.03%	98.03%
Fold 7	98.04%	98.02%	98.03%	98.03%
Fold 8	96.09%	96.04%	96.07%	96.06%
Fold 9	98.45%	98.41%	98.43%	98.43%
Fold 10	97.63%	97.66%	97.64%	97.64%
Average(SD)	96.91%(1.13%)	96.88%(1.14%)	96.89%(1.13%)	96.89%(1.14%)

Table 3

Results of the DenseNet169 architecture.
	Pre	Rec	F1-score	ACC
Fold1	99.21%	99.23%	99.22%	99.21%
Fold 2	98.83%	98.81%	98.82%	98.82%
Fold 3	98.08%	98.00%	98.04%	98.03%
Fold 4	97.64%	97.64%	97.64%	97.64%
Fold 5	98.81%	98.83%	98.82%	98.82%
Fold 6	98.87%	98.79%	98.83%	98.82%
Fold 7	99.21%	99.23%	99.22%	99.21%
Fold 8	99.21%	99.21%	99.21%	99.21%
Fold 9	98.81%	98.83%	98.82%	98.82%
Fold 10	99.60%	99.62%	99.61%	99.61%
Average(SD)	98.83%(0.58%)	98.82%(0.60%)	98.82%(0.59%)	98.82%(0.59%)

3.3 Robustness Testing

To further evaluate the robustness of our CNN models, we also conducted comparative experiments on new testing dataset. The new tongue image dataset consisted of 350 bad tongue images from different tongue diagnosis instrument. The overall accuracy of the trained CNN models is 97.71% with VGG16, 99.04% with ResNet152, and 98.89% with DenseNet169 for the same testing dataset, respectively (Table 4). In addition, since the tongue images from this testing dataset were acquired in different light conditions, the macro accuracy of the testing new dataset is also higher than 97%, revealing that the CNN models have good robustness and can be generalized to images from different instruments with various CCDs and illuminations.

Table 4

Testing results on a new dataset.
Model	Pre	Rec	F1-score	ACC
VGG16	97.77%(0.47%)	97.71%(0.51%)	97.74%(0.49%)	97.71%(0.51%)
ResNet152	99.05%(0.20%)	99.04%(0.20%)	99.05%(0.20%)	99.04%(0.20%)
DenseNet169	98.89%(0.17%)	98.89%(0.18%)	98.89%(0.17%)	98.89%(0.18%)

In sum, the average accuracy of our ResNet152, VGG16, and DenseNet169 with different CNN models and datasets are shown in Fig. 6. It can be seen from the above chart that the tongue image quality control model based on ResNet has a better test effect, with an average accuracy of 99.04%, precision of 99.05%, recall of 99.04%, and F1-score of 99.05%, as shown in Fig. 6, based on the results of the test, it is believed that the CNN model can be used for quality inspection and screening of massive tongue images.

It is a common consensus that standardized tongue image acquisition criterion is an important support for objective tongue diagnosis in TCM clinical research. The quality of tongue image is a crucial indicator in artificial intelligent tongue diagnosis system. Limited by the individual differences of the operators of the tongue diagnosis instruments, in the process of advancing the "Research and Development of the Intelligent Tongue Diagnosis System" project, we found that even though we have already conducted multiple standardized tongue image collection trainings, the obtained tongue image quality is still unqualified, and yet there are many bad quality tongue images mentioned in Section 1. Especially when facing a massive number of tongue image images, manual selection will consume a lot of labor, and the demand for an efficient and intelligent tongue image quality screening model is urgently needed.

To the best of our knowledge, this is the first study using deep convolutional neural networks for tongue image quality assessment. Here, we proposed a CNN framework for the classification of bad quality tongue image. First, we collected 2531 raw tongue images by uniformed instruments, and categorized these tongue images into 1238 bad quality tongue images and 1293 good quality tongue images by 10 TCM professionals. We also preprocessed the tongue image to the same size, generated the dataset from each tongue region image. Then, we used different CNN models, ResNet152, VGG16, and DenseNet169, to extract features and perform binary classifications. Interestingly, the macro average accuracy of the CNN models was impressively over 96% both on the testing subset and on the new testing dataset. Moreover, ResNet152 and DenseNet169 perform a better classification accuracy than VGG16, mainly due to the deeper depths and powerful feature extraction capabilities of the networks. Finally, we collected 700 tongue images from other clinical research centers by different instruments to verify our CNN models. Even in the new dataset, our models with ResNet152, VGG16, and DenseNet169 can achieve a macro average accuracy, precision, recall, and F1-score over 97%. It indicates that the CNN models can be effective and adaptable to tongue images acquired by instrument with different illuminations and CCDs.

In summary, these testing results demonstrate that the CNN models in the present study performed excellent in classification of bad quality tongue images. In addition, our study may become the basic foundation work of the intelligent tongue diagnosis system.

Recently, CNN have become the most widely used and successful models in medicine. In general, CNN architectures can avoid feature selection manually and automatically extract features, which are key elements to enable the intelligent tongue diagnosis system into the TCM clinical practice. Although several previous studies have reported encouraging results using CNN methods to extract tongue image features for tongue color (tongue body and tongue coating) classification [16-18], tongue image characteristic recognition (tooth-marked tongue [19-21], cracked tongue [22,23]) and tongue image segmentation[24-30], but they usually ignore the quality of tongue images, which is strongly related to the accuracy of TCM diagnosis. Thus, the medical application of deep learning methods to the field of tongue diagnosis has not achieved much so far.

Since the bad quality tongue image includes a variety of abnormalities, it is difficult for traditional pattern recognition methods to quickly identify a variety of abnormal images. Therefore, the emergence of deep learning methods such as CNN provides new possibilities. With the assistance of deep learning methods, the proposed CNN method on our tongue image dataset for binary classification performed especially high accuracy, so the tongue image quality assessment can be easily achieved. We efficiently and accurately eliminate poor-quality tongue images, which is conducive to follow-up research and accurate analysis of the tongue diagnosis intelligent analysis system. It provides reliable premises, guarantees the stability of later data analysis, and meets the clinical research needs of tongue diagnosis.

Yet, there are still several shortcomings in this study. The quality assessment of tongue images can be further improved in the future.

First, the training of standardized acquisition techniques, subjects should be given sufficient guidance before collecting tongue images. Those tongue images originated from the operators can be avoided.

Second, despite of the good performance of CNN models, the problem of poor interpretability of CNN [31], some scholars have also conducted visual analysis and research on the process of convolution, pooling, and prediction classification of CNNs [32,33]. Our results show that the deeper architectures (ResNet152, DenseNet169) outperformed the shallower architectures (VGG16) for all the evaluation metrics, including accuracy, precision, recall, and F1-score. However, it is usually more computationally intensive when the CNN layer gets deeper. Therefore, to balance the computation cost and model performance well, it is essential to develop lighter convolutional neural network models for tongue image quality.

Third, further researches are required to investigate the tongue image quality assessment model for more diverse scenes, such as smartphones [34, 35], aiming at various kinds of bad quality tongue image appearances. Constructing multiple classification models to distinguish bad tongue quality images into more groups may increase the clinical applicability to expand the clinical application level of tongue diagnosis.

Our research findings demonstrate various CNN models in the decision-making process for the selection of tongue image quality assessment and prove that applying deep learning methods, specifically deep CNN, to evaluate bad quality tongue images is feasible.

Acc, accuracy;

Rec, recall;

Pre, precision;

SD, standard deviation.

Acknowledgements

Not applicable.

Funding

This study was conducted through the National Key Technology R&D Program of China (No. 2017YFC1703301), the National Natural Science Foundation of China (No. 81574058; No.81904094; No.81973750), Shanghai Key Research Program supported by Shanghai Municipal Commission of Health and Family Planning (ZY (2018-2020)-CCCX-2001-01), and Health Industry Clinical Research Project supported by Shanghai Municipal Commission (201940117).

Availability of data and materials

The experimental data will not be shared as it involved in privacy conditions.

Authors’ Contributions

Tao Jiang and Xiao-juan Hu contributed equally to this work.

Xu-xiang Ma, Xing-hua Yao and Jing-bin Huang enrolled study subjects and collected data. Ji Cui and Li-Ping Tu provided support on data analysis and interpretation and wrote the manuscript. Qing-feng Wu and Jia-tuo Xu designed the study. All authors had read the final version of this manuscript and approved the manuscript before submission.

Ethics approval and consent to participate

This study was reviewed and approved by the institutional research ethics committee of Shuguang hospital affiliated to Shanghai University of Traditional Chinese Medicine (2018-626-55-01).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no conflict of interests.

Author details

1 Basic Medical College, Shanghai University of Traditional Chinese Medicine, 1200 Cailun Road, Shanghai, 201203, China

2 Shanghai Collaborative Innovation Center of Health Service in Traditional Chinese Medicine, Shanghai University of Traditional Chinese Medicine, 1200 Cailun Road, Shanghai, 201203, China

3 School of Information Science and Engineering, Xiamen University, Xiamen, 361005, China

Xu J-T: Clinical map of traditional Chinese medicine tongue diagnosis. BeiJing:Chemical Industry Press; 2017.
D Z HZ. B Z: Tongue image analysis. New York, NY: Springer, Berlin Heidelberg; 2017:207–223.
Li JX, Zhang B, Lu GM, You JE, Zhang D. Body surface feature-based multi-modal Learning for Diabetes Mellitus detection. Inf Sci. 2019;472:1–14.
Bob Z, Kumar BVKV, David Z. Detecting diabetes mellitus and nonproliferative diabetic retinopathy using tongue color, texture, and geometry features. IEEE Trans Bio-med Eng. 2014;61(2):491–501.
Zhang J, Xu J, Hu X, Chen Q, Tu L, Huang J, Cui J. Diagnostic Method of Diabetes Based on Support Vector Machine and Tongue Images. Biomed Res Int. 2017;2017:7961494.
Hsu PC, Huang YC, Chiang JY, Chang HH, Liao PY, Lo LC. The association between arterial stiffness and tongue manifestations of blood stasis in patients with type 2 diabetes. BMC Complement Altern Med. 2016;16(1):324.
Lo LC, Cheng TL, Chen YJ, Natsagdorj S, Chiang JY. TCM tongue diagnosis index of early-stage breast cancer. Complement Ther Med. 2015;23(5):705–13.
Han S, Chen Y, Hu J, Ji Z. Tongue images and tongue coating microbiome in patients with colorectal cancer. Microb Pathog. 2014;77:1–6.
Pang B, Zhang D, Wang KQ. Tongue image analysis for appendicitis diagnosis. Inf Sci. 2005;175(3):160–76.
Meng D, Cao G, Duan Y, Zhu M, Tu L, Xu D, Xu J. Tongue Images Classification Based on Constrained High Dispersal Network. Evidence-Based Complementary Alternative Medicine. 2017;2017:7452427.
He K, Zhang X, Ren S, Jian S: Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision & Pattern Recognition: 2016; 2016:770–778.
Minaee S, Boykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D. Image Segmentation Using Deep Learning: A Survey 2020; arXiv preprint arXiv:2001.05566.
Qi Z, Tu LP, Chen JB, Hu XJ, Xu JT, Zhang ZF: The Classification of Tongue Colors with Standardized Acquisition and ICC Profile Correction in Traditional Chinese Medicine. Biomed Res Int 2016, 2016:3510807.
Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2014:1409.1556.
Huang G, Liu Z, Maaten LVD, Weinberger KQ: Densely Connected Convolutional Networks. In Proceedings of the IEEE conference on computer vision and pattern recognition: 2017; 2017:4700–4708.
Yang J-D, Zhang P. Tongue image classification method based on transfer learning and fully connected neural network. Academic Journal of Second Military Medical University. 2018;39(08):897–902.
Tang Y-P, Wang L-R, He X, Chen P, Yuan G-P. Classification of tongue image based on multi-task deep convolutional neural network. Computer Science. 2018;45(12):255–61.
Xiao Q-X, Zhang J, Zhang H, Li X-G, Zhuo L. Tongue coating color classification based on shallow convolutional neural network. Measurement Control Technology. 2019;38(03):26–31.
Li X, Zhang Y, Cui Q, Yi X, Zhang Y. Tooth-Marked Tongue Recognition Using Multiple Instance Learning and CNN Features. Ieee Transactions on Cybernetics. 2019;49(2):380–7.
Wang X, Liu J, Wu C, Liu J, Chen J. Artificial intelligence in tongue diagnosis: using deep convolutional neural network for recognizing unhealthy tongue with tooth-mark. Computational Structural Biotechnology Journal. 2020;18:973–80.
Sun Y, Dai S, Li J, Zhang Y, Li X. Tooth-Marked Tongue Recognition Using Gradient-Weighted Class Activation Maps. Future Internet. 2019;11(2):45.
Li XQ, Wang D, Cui Q. WLDF: Effective Statistical Shape Feature for Cracked Tongue Recognition. Journal of Electrical Engineering Technology. 2017;12(1):420–7.
Liu M, Wang X-T, Zhou L, Tan L-B, Li J, Guan J, Li F. Study on extraction and recognition of Traditional Chinese Medicine tongue manifestation: based on deep learning and migration learning. J Tradit Chin Med. 2019;60(10):835–40.
Huang X, Zhang H, Zhuo L, Li X, Zhang J: TISNet-Enhanced Fully Convolutional Network with Encoder-Decoder Structure for Tongue Image Segmentation in Traditional Chinese Medicine. Computational and Mathematical Methods in Medicine 2020, 2020: 6029258.
Zhou C, Fan H, Li Z. Tonguenet: Accurate Localization and Segmentation for Tongue Images Using Deep Neural Networks. Ieee Access. 2019;7:148779–89.
Lin B, Xle J, Li C, Qu Y. Deeptongue: Tongue Segmentation Via Resnet. In: 2018; 2018: 1035–1039.
Li L, Luo Z, Zhang M, Cai Y, Li C, Li S. An iterative transfer learning framework for cross-domain tongue segmentation. Concurrency Computation-Practice Experience. 2020;32(14):e5714.
Yuan W, Liu C: Cascaded CNN for Real-time Tongue Segmentation Based on Key Points Localization. In IEEE International Conference on Big Data Analytics:2019; 2019: 303–307.
Xu Q, Zeng Y, Tang W, Peng W, Xia T, Li Z, Teng F, Li W, Guo J. Multi-task Joint Learning Model for Segmenting and Classifying Tongue Images Using a Deep Neural Network. IEEE journal of biomedical health informatics. 2020;24(9):2481–9.
Zhang X-F, Guo Y-T, Cai Y-H, Sun M. Tongue image segmentation algorithm based on deep convolutional neural network and fully conditional random fields. Journal of Beijing University of Aeronautics Astronautics. 2019;45(12):2364–74.
Attia ZI, Noseworthy PA, Lopez-Jimenez F, Asirvatham SJ, Deshmukh AJ, Gersh BJ, Carter RE, Yao X, Rabinstein AA, Erickson BJ, Kapa S, Friedman PA. An artificial intelligence-enabled ECG algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction. Lancet. 2019;394(10201):861–7.
Wang ZJ, Turko R, Shaikh O, Park H, Das N, Hohman F, Kahng M, Chau DH: CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization. ArXiv 2020, abs/2004.15004.
Selvaraju RR, Cogswell M, Das A, Vedantam R, Batra D: Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In Proceedings of the IEEE international conference on computer vision 2017; 2017: 618–626.
Zhang Q, Shang HL, Zhu JJ, Jin MM, Wang WX, Kong QS: A new tongue diagnosis application on Android platform. In IEEE International Conference on Bioinformatics and Biomedicine: 2013; 2013: 334 – 327.
Hu MC, Lan KC, Fang WC, Huang YC, Ho TJ, Lin CP, Yeh MH, Raknim P, Lin YH, Cheng MH, He YT, Tseng KC. Automated tongue diagnosis on the smartphone and its applications. Comput Methods Programs Biomed. 2019;174:51–64.

Download PDF

Journal Publication

published 05 May, 2021

Read the published version in BMC Medical Informatics and Decision Making →

Editorial decision: Major revision
04 Jan, 2021
Reviewer #4 agreed at journal
16 Dec, 2020
Review #4 received at journal
16 Dec, 2020
Review #3 received at journal
11 Dec, 2020
Reviewer #3 agreed at journal
08 Dec, 2020
Review #2 received at journal
18 Nov, 2020
Reviewer #2 agreed at journal
24 Oct, 2020
Reviewer #1 agreed at journal
23 Oct, 2020
Review #1 received at journal
23 Oct, 2020
Reviewers invited by journal
22 Oct, 2020
Editor assigned by journal
19 Oct, 2020
Submission checks completed at journal
12 Oct, 2020
Editor invited by journal
12 Oct, 2020
First submitted to journal
11 Oct, 2020

You are reading this latest preprint version

Tongue Image Quality Assessment Based on Deep Convolutional Neural Network

Status:

Journal Publication

Version 1

Abstract

Figures

1. Background

2. Methods

2.1 Tongue image acquisition and preprocessing

2.2 CNN network architecture

2.3 Model training, validation, and testing

2.4 Model evaluation metrics

3. Results

3.1 Testing results on tongue image dataset with ResNet152

3.2 Compare With Vgg16 And Densenet 169

3.3 Robustness Testing

4. Discussion

5. Conclusions

Abbreviations

Declarations

References

Status:

Journal Publication

Version 1