In this study, we aim to determine the multimodal data associations between radiological, pathologic, and molecular characteristics in bladder cancer. Furthermore, we tried to find the differential features between high- and low-grade bladder cancer, and improve the prediction accuracy of molecular subtype by multimodal feature fusion. 127 patients with bladder cancer are analyzed in this study. Multimodal associations analysis of CT and pathologic WSI were performed. Structural equation modeling (SEM) was used to measure the structural relationships between multimodal data. Shannon entropy was calculated to evaluate the heterogeneity of bladder cancer. A convolutional neural network was constructed for molecular subtyping based on multimodal features. Feature contribution in model decision-making was explored by discriminative localization method. Cox regression and Kaplan-Meier survival analysis were used to explore the relevance of multimodal features to the prognosis. There are extensive associations between CT and WSI features and parts of these features were related to prognosis. 77 densely associated blocks of feature pairs between CT and WSI were identified. The first block were confirmed with relation to tumor grade. Block 2 and 3 reflect the association of plasm and nucleus of cancer cells with CT features, respectively. SEM detected significant relation between pathological features and molecular subtype. High-grade bladder cancer showed heterogeneity of significance across different scales and higher disorder in microscopic level. Entropy level of WSI can be used to predict the prognosis of patients. For the molecular subtype prediction, an AUC of 0.95±0.09 was achieved based on multimodal feature fusion. The fused CT and WSI features achieved a more accurate molecular subtype prediction. At last, 13 prognosis-related features from CT and WSI were identified.
Article
Integrated analysis of multimodal bladder cancer data based on CT, Whole Slide Image and Transcriptome
https://doi.org/10.21203/rs.3.rs-628673/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
Multimodal Data Integration
Association Analysis
Structural Equation Model
Bladder Cancer
With the advancements in medical diagnostic technologies, the abundance of multimodal data in clinics, pathology, radiology, and molecular characteristics enable us to obtain multi-dimensional information of diseases and greatly deepen the understanding of the nature of diseases. Integrative analysis of multimodal data helps to discover new features of complex diseases such as cancer[1, 2].
Bladder cancer is the ninth common tumor worldwide and with the highest incidence in the urinary system. About 75% of newly diagnosed patients are non-muscle-invasive bladder cancer (NMIBC) characterized by painless haematuria and ~25% have muscle-invasive bladder cancer (MIBC) with a 5-year survival rate as low as 40%~60%[3-5].
Features of bladder cancer have been extensively studied at the subcellular level and comprehensive molecular analysis has identified four molecular subtypes, basal, luminal neuronal, and squamous[6, 7]. Recent studies have shown that molecular typing is correlated with the prognosis of patients and substance in the prediction of response to neoadjuvant chemotherapy[8-11]. However, the confirmatory procedure requires sequencing the cancer tissues after surgery, and the difficulty in specimen accessibility and sequencing expense limits its clinical application. An alternative way is to infer the molecular types by other easily accessible features, such as radiological imaging features. Recently, histopathological slides have been used to predict the molecular subtypes in MIBC by convolution neural network, providing a potential way to confirm the molecular subtypes[12].
CT images and histopathological slices are the conventional diagnosis methods for tumor patients. CT images primarily record a pattern of densities of our body and histopathological slides usually acts as the gold standard for diagnosis. Currently, the association among molecular features, pathological and radiological feature in bladder cancer remain obscure and rare studies have been involved. In this work, we collect data from the molecular, cellular, and tissue level respectively and investigate the relevance of quantitative imaging features including pathology and radiology to the molecular signatures. Additionally, we build a deep learning model to learn the latent association among different modalities and try to predict the molecular subtypes of patients. Lastly, we analyze the contributions of features in the model decision for a certain type using visualization strategies.
Patient cohorts and data preparation
The participants in this study were from three cohorts. For the patients from the TCGA-BLCA cohort, received transurethral resection or cystectomy from 2010 to 2014, the original 120 raw CT images in DICOM format and the digital pathologic slices WSI (40 × magnification) were gathered from the open-source The Cancer Imaging Archive (TCIA) database, and the corresponding clinicopathologic, survival information and raw-counts files measuring gene expression level in bladder cancer cell were gathered in The Cancer Genome Atlas (TCGA) database. After selection, 76 participants (17 females and 59 males) were involved in the following analysis. The second and third cohorts of 19 (19 males) and 32 (4 females and 28 males) patients are from two first-class hospitals separately in Shenzhen and Beijing, China. All the patients received surgeries from 2018 to 2020. CT images, WSI, and clinicopathologic diagnostic reports were gathered from the local hospital information system. For the 23 patients in the third cohorts, raw counts data of RNA-sequencing of bladder cancer tissues were available. All data used were acquired following the institutional review board-approved protocols. Informed consent documents were obtained from the patients of the local cohort.
The basic and clinical characteristics of these participants are shown in Table 1. In summary, the mean age (standard deviation) is 65.6 (10.1) years. There are 104 (81%) participants diagnosed with high-grade bladder cancer, 79 (62%) participants diagnosed with muscle-invasive bladder cancer.
Table 1: Basic and clinical characteristics of participants.
Data preprocessing and feature extraction
For the CT images in these cohorts, we performed additional quality screening to exclude the CT images with low resolution (slice thickness > 5mm), secretion period of enhancing CT images, and CT images that do not contain bladder. After filtering, the bladder and cancer region in CT images were manually annotated by two radiologists (been in clinical practice for more than 5-year experience) and two well-trained clinical graduate students. To speed up the annotation process, we trained a deep learning model based on 3D U-Net and performed automatic semantic segmentation on rest fo the data. Then the segmentation results were examined and amended by radiologists who further delineated the accurate bladder and cancer boundaries. After the annotation, 200 radiological features of the bladder and caner voxel were extracted by PyRadiomics (v 3.0.1)[13], which roughly consist of texture, morphological and statistical features.
The pathologic slices of cancer tissues were processed and digitalized at various institutions. Each slice was split into tiles of 1024×1024 pixels and then normalized to the same reference tile as described by Anand et al [14]. The cancer cell phenotypes were quantified from the normalized tiles by CellProfiler software (v 4.1.3)[15]. In brief, 1029 quantitative features were generated, containing the mean, median, and standard deviations of shape, intensity, texture, and radial distribution features of the nucleus and plasma of cells.
Levels of cancer gene expression were obtained by RNA sequencing in 23 cases in the third cohorts. Raw gene counts were calculated by alignment to human reference genome GRCh38. TPM (transcripts per million) was used as the measurement to normalize the raw counts. We combined the molecular subtypes (Luminal/Luminal-papillary/Luminal-infiltrated/Neuronal) as one class (Luminal) and Basal as the other (Basal). Molecular subtyping (luminal/basal) in bladder cancer was performed by hierarchical cluster with a panel of 48 genes according to the TCGA subtype approach[6]. The molecular classification of participants is shown in Table 1.
The diagram of the integrative analysis framework is shown in Fig. 1. In brief, features of CT and WSI were used to explore the multimodal associations, molecular subtype prediction and prognosis.
Multimodal association analysis
To explore the associations between CT and WSI features, feature selection was performed by removing the highly correlated (Pearson Correlation > 0.7) features in WSI features. After selection, 200 features of CT and 428 features of WSI were used in the association analysis. Firstly, modal correlation matrices were calculated by HAllA(v 0.8.18)[16]. Then blocks of associated features with significant correlations were determined by hierarchical clustering and Gini impurity of the splits.
To further explore the statistical dependancy between CT, WSI, clinicopathological characteristics and molecular subtypes, we constructed a structural equation model (SEM) using the aforementioned features by lavaan (v0.6-8) [17]. The standardized beta coefficient (β) was used to quantify the influence of the change in the independent variables on the dependent variables. Other indices such as root mean square error of approximation (RMSEA), compare fit index (CFI) were used to measure the model fitting effect.
Statistical analysis
The differential features of high- and low-grade cancer were identified by Welch’s two-sample t-test. The Cox regression analysis and survival analysis were used to assess the impact of multimodal features upon overall prognosis. All these machine learning experiments and statistical analyses were conducted in python (v3.8) and R (v3.6.3). The statistic analysis is considered of significance when the P-value < 0.05. The uncertainty of the estimate such as accuracy, AUC was quantified at the 95% confidence interval. To quantitatively measure the heterogeneity of bladder cancer across different scales, we calculate the Shannon entropy of its frequency distribution for each modality.
Deep learning algorithm and model interpretability analysis
For the classification experiments, cases were randomly split into training (80%) and testing (20%) sets. A convolutional neural network was constructed to predict the molecular subtype by a multimodal feature fusion approach. Performance metrics include accuracy, precision, F1 score, recall, and the area under the curve (AUC) of the receiver operating characteristic (ROC).
To explore the contributions of different features in model decision-making, the post-hoc explanation method called Score-Class Activation Maps (Score-CAM)[18] was conducted. A large score defined by Score-CAM suggests that the feature affects the substance of the model decision.
Strong associations among multimodal features
A total of 200 CT and 428 WSI features from 127 participants were processed with HAllA to investigate the associations within modalities. 77 densely associated blocks were identified in the modal correlation matrix between CT and WSI features (Supplementary material Fig.S1). Block 1 has the largest number of notable feature pairs that were clustered together (Fig. 2a). The significance level of feature pairs in block 1 was verified by the Pearson correlation test. One feature pair of significance was illustrated in Fig.2b. Positive correlation (R= 0.537 ± 0.113, P < 0.001) was found between cancer_firstorder_TE (CT feature: cancer first-order Total Energy)and nucleus_mean_Texture_CORH3 (WSI feature:nucleus Texture Correlation Hematoxylin)(Fig.2b). The second and third blocks reflect the associations of plasmic and nucleic features of cancer cells with CT features, respectively (Supplementary material Fig.S2). Texture features of cancer cell plasma are associated with grey level matrix relevant features in CT (P<1.49 10-6). On the other side, nucleus-related features are accompanied by First_order features which describe the distribution of voxel intensities within the bladder (P<4.08 10-6).
We also performed a supervised analysis by a two-sample t-test to explore the features with significant differences between the high- and low-grade bladder cancer. Texture, Intensity, Area Shape related features are distinctive in high- and low-grade WSI. Among the radiological features, shape-related features, grey level matrix (glcm, glrlm, and glszm) exhibits the significant difference between high- and low-grade bladder cancer (Fig.2c). Moreover, most features in Block 1 are in line with the differential features between high- and low-grade bladder cancer (Odds Ratio = 21.0 ± 10.8, P < 0.001) (Fig.2d), suggesting that the features in block 1 reveal the pathological grade characteristics.
High-grade bladder cancer typically presents a high level of heterogeneity than the low-grade[19]. We quantify this heterogeneity by Shannon entropy, which is a specific measure of randomness. Fig.2e shows the entropy of high- and low-grade bladder cancer in different modalities and the high-grade get a higher entropy in all scales. Additionally, bladder cancer has higher entropy at the microscopic level than the macro level, which reflects the disorder of cancer at the microscopic scale. We assessed the efficacy of Shannon entropy in predicting the outcome of patient and results showed that the entropy level in cancer cell nucleus and plasma has a significant impact on clinical prognosis (Fig.S4).
Statistical dependancy among multimodal data
According to previous results and related studies, we hypothesized that both CT and WSI features were related to clinicopathology and luminal/basal molecular subtype[12]. To further explore the statistical dependency among multimodal data, we constructed a hypothetical path diagram (Fig.3) and validate the hypothesis by SEM.
The SEM analysis confirmed significant associations (P < 0.05) along the proposed path. A strong association is observed from WSI Features to Pathologic Subtype (β=0.806, P<0.001), which is consistent with pathological diagnosis. A moderate association is observed from WSI Features to luminal/basal Molecular Subtype (β=0.396, P<0.001), which agreed on the conclusion that molecular subtype of bladder cancer could be inferred by WSI features[12]. Furthermore, a weak association was detected between CT Features and Pathologic Subtype (β=0.148, P=0.007), Molecular Subtype (β=-0.20, P=0.031), which indicated molecular changes might also have a certain influence on CT texture features. Fitting indices indicate the proposed path fit the data well (CFI: 0.942, TLI: 0.905, RMSEA: 0.099, SRMR: 0.103)
Molecular subtype prediction by multimodal feature fusion
Considering that the molecular signature of cancer had an impact on both CT and WSI, we constructed a deep learning model using features of CT and WSI to predict luminal/basal subtype.
As described in Fig.4a, a convolutional neural network for molecular subtype classification was constructed. 97 cases, each containing 200 CT features and 1029 WSI features were randomly split into a training set (79%, 77 cases) and a testing set (21%, 20 cases). The AUC (area under the curve) of the ROC (receiver operating characteristic curve) curve is 0.95 0.09 for luminal/basal molecular subtype classification on the testing set (Fig.4b), while the accuracy is 90% (78.6%-100% at the 95% confidence level). Our model works better than the unimodal models by using either WSI or CT (Supplementary material Fig.S3).
To explore the contribution of each feature to the model prediction, Score-CAM was established for the deep learning model. We calculated the score of each input feature for all the cases and took the features with positive median scores as model-activated features (Fig.4c). In summary, the convolution neural network lays particular emphasis on luminal/basal molecular classification. Features of WSI exhibited the highest scores and shown strong relations with molecular subtype which is consistent with results of SEM analysis. The value of CT features in predicting molecular subtype is relatively lower than the WSI.
Relevance of imaging features to prognosis
Considering the molecular subtype of bladder cancer has a significant impact on clinical prognosis, we performed Cox regression analysis on the features with positive median scores that influence the prediction of molecular subtypes (Fig.5a). 13 features of substance were identified. CT texture feature such as first-order statistics of voxel intensities (bladder_firstorder_10Percentile) and WSI feature such as the measurement in nucleus area (nucleus_median_AreaShape_MinFeretDiameter) indicate a poor prognosis. The prognostic relevance of the 13 features was also verified by log-rank tests. Fig.5b illustrates the relevance of the nucleic WSI feature nucleus_StDev_Intensity_MeanIntensity_Hematoxylin to the overall survival.
In this paper, we delineate the multimodal characteristics of bladder cancer collected from 127 individuals on three different scales, namely molecular, cellular, and tissue level. We identified novel associations in pathological and radiological features that distinguish high- and low-grade bladder cancer. The texture features of cancer cell plasma are related to the gray level matrix features in CT, while the nucleic features reflect the intensity level of voxels in CT. In addition, high-grade bladder cancer showed heterogeneity of significance across different scales, which is validated by Shannon entropy, a generalizable indicator across all scales for grading and prognosis. We further explored the statistical dependancy of modality features by SEM and validate our main findings. A deep learning-based model was built to predict the molecular subtype and achieved an AUC of 0.95 .
Specifically, Texture, Intensity, Area Shape related features are discriminative characteristics in high- and low-grade bladder cancer slides. High-grade samples tend to manifest higher intensity and more variations in texture. The area shape of the nucleus in the high-grade cancer cells is changeable, and a high coefficient of variation was observed in radial intensity distribution in the nucleus. A similar phenomenon was observed in CT images. First-order statistics (describe the distribution of voxel intensities) in intensity level, Nonuniformity, and Zone Variance in the grey level matrix (glcm, glrlm, and glszm) exhibited a significant difference between high- and low-grade patients. These results are in line with the noteworthy pattern identified by HAllA, which demonstrates that the variation in texture and intensity distribution of bladder cancer cells in slides correlate with the grey level matrix variation of bladder cancer at the CT level. In summary, the patterns in different scales in the same pathological grade are similar. Additionally, we notice that nucleus-related features are associated with basal molecular subtype in Score-CAM analysis, such as the radial intensity distribution. The pathological features are in a dominant position in predicting the molecular subtype and prognosis. Interestingly, bladder shape-related features in CT played an important role in predicting luminal subtype. We observed that unsupervised analysis obtained a similar feature association as supervised pattern detection, which indicates that these features are relevant to the transition in pathological type and can be utilized as an imaging biomarker. Moreover, molecular features are crucial in today’s personalized treatment. The complex interactions between individuals’ molecular characteristics and phenotype typically require diverse and large-scale data to be identified. A systematic multimodal data analysis approach is essential in the identification of novel signatures for diseases. Our work proffers a comprehensive framework for the signature recognition of multimodal data and is helpful in personalized medical management.
The present framework still has several limitations that require further improvement. Firstly, there is a certain sample imbalance such as the number of patients with different tumor grade, which might influence the statistical results in this study. Secondly, these findings remain to be replicated in a large population-based cohort before they are applied in the clinic. Additional efforts are needed to take more participants in this study. In conclusion, high-grade bladder cancer showed heterogeneity of significance across different scales. The Shannon entropy can be used as an indicator to differentiate high- and low-grade bladder cancer and predict the outcome. The characteristics of the plasm and nucleus of cancer cells in the pathological slides are correlated to gray level matrix characteristics and voxel intensity characteristics at the CT level. CT features also contribute to the molecular subtype prediction, though not as much as WSI features do. Our work demonstrates the potential of multimodal data analysis in personalized medicine and related clinical application.
Competing interests
All authors declare no competing interests.
Data Availability
All of the code generated or used during the study are available in the Github repository (https://github.com/wukaiyeah/Multimodal_analysis_in_bladder_cancer). The original images and data used in this study are available from the corresponding author by request.
1. Yan, R., et al. Integration of multimodal data for breast cancer classification using a hybrid deep learning method. in International Conference on Intelligent Computing. 2019. Springer.
2. Gao, J., et al., A Survey on Deep Learning for Multimodal Data Fusion. Neural Comput, 2020. 32(5): p. 829-864.
3. Kamat, A.M., et al., Bladder cancer. The Lancet, 2016. 388(10061): p. 2796-2810.
4. Witjes, J.A., et al., Updated 2016 EAU guidelines on muscle-invasive and metastatic bladder cancer. European urology, 2017. 71(3): p. 462-475.
5. Saginala, K., et al., Epidemiology of Bladder Cancer. Med Sci (Basel), 2020. 8(1).
6. Robertson, A.G., et al., Comprehensive molecular characterization of muscle-invasive bladder cancer. Cell, 2017. 171(3): p. 540-556. e25.
7. Network, C.G.A.R., Comprehensive molecular characterization of urothelial bladder carcinoma. Nature, 2014. 507(7492): p. 315.
8. McConkey, D.J. and W. Choi, Molecular subtypes of bladder cancer. Current oncology reports, 2018. 20(10): p. 1-7.
9. Seiler, R., et al., Impact of molecular subtypes in muscle-invasive bladder cancer on predicting response and survival after neoadjuvant chemotherapy. European urology, 2017. 72(4): p. 544-554.
10. McConkey, D.J. and W. Choi, Molecular Subtypes of Bladder Cancer. Curr Oncol Rep, 2018. 20(10): p. 77.
11. Tan, T.Z., et al., Molecular subtypes of urothelial bladder cancer: results from a meta-cohort analysis of 2411 tumors. European urology, 2019. 75(3): p. 423-432.
12. Woerl, A.-C., et al., Deep learning predicts molecular subtype of muscle-invasive bladder cancer from conventional histopathological slides. European urology, 2020. 78(2): p. 256-264.
13. Van Griethuysen, J.J., et al., Computational radiomics system to decode the radiographic phenotype. Cancer research, 2017. 77(21): p. e104-e107.
14. Anand, D., G. Ramakrishnan, and A. Sethi. Fast gpu-enabled color normalization for digital pathology. in 2019 International Conference on Systems, Signals and Image Processing (IWSSIP). 2019. IEEE.
15. Jones, T.R., et al., Scoring diverse cellular morphologies in image-based screens with iterative feedback and machine learning. Proc Natl Acad Sci U S A, 2009. 106(6): p. 1826-31.
16. Rahnavard, G., et al., High-sensitivity pattern discovery in large multi’omic datasets. URL https://huttenhower. sph. harvard. edu/halla, 2017.
17. Rosseel, Y., lavaan: An R Package for Structural Equation Modeling. 2012, 2012. 48(2): p. 36.
18. Wang, H., et al. Score-CAM: Score-weighted visual explanations for convolutional neural networks. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.
19. Cheng, L., et al., Cancer heterogeneity and its biologic implications in the grading of urothelial carcinoma. Cancer, 2000. 88(7): p. 1663-1670.
There is NO Competing Interest.
- supplementarymaterial.docx
supplementary material