DNA-Methyaltion-Based Deep Learning for Precision Classification of Central Nervous System Tumors: A Comparative Study

doi:10.21203/rs.3.rs-3897766/v1

Download PDF

Article

DNA-Methyaltion-Based Deep Learning for Precision Classification of Central Nervous System Tumors: A Comparative Study

https://doi.org/10.21203/rs.3.rs-3897766/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 02 Oct, 2024

Read the published version in npj Precision Oncology →

You are reading this latest preprint version

As part of the advancement in therapeutic decision-making for brain tumor patients at St. Jude Children’s Research Hospital (SJCRH), we develop and compare the performance of three classification models: a deep learning neural network (NN), an exact bootstrap k-nearest neighbor (kNN), and a random forest classifier (RF) model to predict the 82 molecularly distinct central nervous system (CNS) tumor classes based on DNA-methylation profiles of 2,801 patients. We validate their classification accuracy, precision, and recall with 2,054 samples from two independent cohorts. Although all models perform robustly to missing data, the NN model achieves the highest classification accuracy and maintains better balance between precision and recall than kNN and RF. Average precision and recall of NN reduce to that of RF and kNN only when tumor purity was less than 50%. In conclusion, DNA-methylation based deep learning approach provides the most potential advancement toward precision medicine for brain tumors.

Biological sciences/Computational biology and bioinformatics

Health sciences/Oncology/Cancer/CNS cancer

deep learning

machine learning

neural network

central nervous system tumor

DNA methylation

Brain tumor classification has historically relied on morphologic examination of tumor specimens under a light microscopy ¹. Refinement of the process has occurred through recognition of additional tumor-specific histologic patterns and by integrating testing modalities such as cytogenetics, immunohistochemistry, and nucleic acid sequencing findings into the classification schemes. More recently, it has been recognized that normal and neoplastic tissues have inherent epigenetic signatures encoded in their methylome ^2–5. The pattern of CpG methylation has been established as a stable and reliable biomarker for tumors and normal tissues ^2,6. The DNA methylation signature is considered a combined surrogate of the cell of origin and genomic driver abnormality. It is retained even after tumor recurrence or passage of tumors as an orthotopic xenograft ^7–9. Histologically-defined tumor types often consist of heterogeneous molecular subtypes with distinct biological and clinical behavior ^10,11. Due to the lack of recurrent defining mutations, some of these molecular subtypes may only be recognizable by their distinct methylation or transcriptomic signatures ^5,12.

The adoption of high-density methylation arrays (Illumina Infinium 450K and 850K EPIC arrays) has allowed for genome-wide evaluation of DNA methylation from large cohorts of human tumors. These arrays have favorable characteristics, including relatively low cost, comparable performance on both fresh or formalin-fixed and paraffin-embedded (FFPE) tissues, and stability of the methylation mark even in material stored for multiple decades ¹³. In addition, the ability to assay FFPE has facilitated the accumulation of large tumor cohorts and allows for easy integration into standard clinical workflows ^5,6,14.

The initial utility of methylation profiling to refine the classification of brain tumors relied on an unsupervised analysis comparing specific tumor cohorts of interest to reference brain tumor types ^5,12,14,15. However, the introduction of supervised classification models based on methylation data has significantly improved clinical diagnostics. In addition, these models are amenable to the types of performance metrics typically utilized for clinical tests ^6,16. The current state-of-the-art model for methylation-based classification relies on a random forest (RF) classifier trained on a reference cohort containing all tumor entities represented in the 2016 WHO Classification of Tumours of the Central Nervous System ⁶. While showing relatively good performance, certain model features remain suboptimal, yielding either misclassification or a substantial number of subthreshold scores for selected tumor types ^6,17.

Several models utilizing simple and advanced machine learning techniques have been proposed to improve classification accuracy. For example, the well-received k-nearest neighbor (kNN) algorithm was adapted and showed high capability and notable sensitivity in the acceptable discrimination of classes ^18–21. The kNN approach was also employed in a hybrid classification model combining genetic algorithms and artificial neural networks ²². This hybrid model showed more effectiveness in classifying performance than expectation and maximization classifiers. Recently, artificial neural networks (NN) have been considered by researchers as one of the most useful and applicable constructs in artificial intelligence ^23–25. Specifically, deep NNs have been widely adopted in the diagnostic process of various biomedical fields and provided many opportunities to improve health care and heighten precision in the oncologic pathology ^23,26–28. Despite their effectiveness and popularity, neither k-NN nor deep learning NN has been explored in the epigenetic field to classify the complex molecular entities among central nervous system (CNS) tumors. Currently, no commercial models are available for CNS tumor classification based on DNA methylation data.

As part of our clinical implementation of DNA methylation profiling at St. Jude Children’s Research Hospital (SJCRH), we developed a multi-layer perceptron NN model (NNmod) based on DNA methylation profiles of 2,801 samples from a reference tumor cohort to predict the 82 histologically and/or molecularly distinct CNS tumor classes and 9 normal controls. We also compared the performance characteristics of this NNmod to the current state-of-the-art RF classifier ⁶ (RFmod) and another alternative model using an exact bootstrap k-nearest neighbor (kNNmod) algorithm. We validated the performance of these models with two independent brain tumor cohorts consisting of 1104 samples from GSE109379 and 950 samples from the St. Jude Children’s Research Hospital. Our results showed that although all models performed robustly to missing data, the deep NN model had the highest CNS classification accuracy and the most favorable performance characteristics, especially in minimizing the proportion of subthreshold score during testing and validation. Average precision and recall of the NNmod started reduce to similar levels of kNNmod and RFmod when tumor purity was less than 50%. This suggests that a deep NN model can be implemented in clinical laboratories as a reliable and essential diagnostic tool to assist in precision therapy for brain tumors.

Model performance on train and test set:

We developed three models, i.e., a k-nearest neighbors model (kNNmod), a random forest model (RFmod), and a multi-layer perceptron neural network model (NNmod) (Fig. S1), to classify human CNS tumors based on methylation signatures of the comprehensive reference set (GSE90496, n = 2801). This set comprises 91 methylation classes grouped into 75 methylation class families based on their histological and biological closeness (Capper et al., 2008). The RF model represented a recapitulation of the previous random forest produced by Capper et al., representing the best in the current model. Here, we compared the performance of kNNmod and NNmod to the RFmod. The three models were evaluated with 1000 leave-out-25% cross-validations in predicting methylation classes and families (Fig. S2). All models produced accuracies above 0.95 for both class and family prediction (Table 1). Among the three models, classification accuracy and its Kappa statistic were highest in NNmod (above 0.98) and lowest in kNNmod (0.90 and 0.95 for class and family prediction) (Table 1). These accuracies were statistically significantly different from the null accuracy, i.e. the accuracy could be achieved by predicting the most frequent class (McNeMar's p-values < 10^–16). These results suggest that all models produced useful predictions with high accuracy.

Table 1

Overall performance of leave-out-25% train-test process for each classifier on the GSE90496
	RF		kNN		NN
	Class	Family	Class	Family	Class	Family
Accuracy	0.98	0.99	0.95	0.96	0.99	0.99
Accuracy Null	0.05	0.11	0.06	0.11	0.06	0.12
Kappa	0.96	0.99	0.90	0.95	0.98	0.99
Recall	0.97	0.98	0.86	0.88	0.98	0.99
Precision	0.96	0.985	0.90	0.92	0.98	0.995
Specificity	0.99	0.99	0.99	0.99	0.99	0.99

Cross-validation misclassifications by RFmod and NNmod focused on a few methylation classes while miss-classifications by kNNmod spread into many classes. Cross-validation of RFmod, kNNmod, and NNmod resulted in an average accuracy of 98%, 95%, and 99% for class prediction, respectively (Table 1). Notably, NNmod produced the best accuracy in predicting methylation class in all 1000 cross-validation rounds (Table S1). kNNmod, compared to RFmod and NNmod, had the lowest precision (90% vs. 96% and 98%, respectively) and recall (86% vs. 97% and 98%). All models had comparable specificity (around 99%). The majority of miss-classifications among three models occurred within the six histologically and biologically closely related tumor classes (pituitary adenomas - PITAD and PITUI) and myxopapillary ependymomas (EPN-MPE) (Fig. 1A). However, kNNmod misclassification expanded to other methylation classes such as ependymomas (EPN), supratentorial subependymomas (SUBEPN), low-grade gliomas (LGG), melanomas (MELAN), melanocytomas (MELCYT), and plexus tumors (PLEX) (Fig. 1Aii). On the other hand, NNmod had the narrowest ranges in accuracy, precision, and recall of predicting 91 subclasses with a median value around 0.98 for each metric (Fig. 1B-C and Table S1). Minimal F1 scores for RFmod, kNNmod, and NNmod were 0.729, 0.359, and 0.761, respectively (Fig. 1D and Table S1), suggesting NNmod had the best balance between precision and recall.

All models, in general, performed better at predicting methylation families. The classification metrics of these 75 methylation families are shown in Table 1 and Fig. S3. The cross-validation accuracies for the clinically relevant groupings were improved in all models. Compared to kNNmod, NNmod showed higher accuracy (99% vs 96%), precision (99% vs 88%), and recall (99.5% vs 93%). Compared to RFmod, the NNmod showed higher recall (99% vs. 98%) and comparable accuracy (~ 99%), precision (~ 98%), and specificity (~ 99.9%) (Table 1). Among the 1000 cross-validation rounds in predicting methylation family, NNmod produced the best accuracy 604 times, while RFmod produced the best accuracy 280 times. The rest of the cross-validation rounds, NNmod and RFmod had the same accuracy that was higher than kNNmod (Table S2). The misclassification of RFmod and NNmod among the CNS tumor classes shown in Fig. 1A appeared to be dissolved but retained in kNNmod (Fig. S3A). Although accuracy was improved for all models, the gap between precision (88%) and recall (93%) for kNNmod (Fig. S3B-D and Table S1) remained the same in predicting methylation families. Minimal F1 scores for RFmod and NNmod were increased to 0.878 and 0.883, while this score was reduced to 0.318 by kNNmod (Table S2). In conclusion, these results indicated that although RFmod and NNmod had very comparable performance, NNmod still had the highest accuracy and the best balance between precision and recall among the three models, suggesting that it had the highest discriminating power for both methylation class and family.

Model performance on two independent validation sets:

The classification performance of the three models was additionally tested on two independent data sets (GSE 109379) and the SJCRH data sets. To objectively assign each independent test sample to the reference methylation class group, we performed a semi-supervised learning approach ²⁹ to assign labels to the two validation data sets. The 1,104 samples were assigned to 65 methylation classes and 50 families (Table S3), while the 950 SJCRH samples were grouped into 49 methylation classes (Table S4). This result was then used as the ground truth to measure the accuracy of the prediction results from our classifiers. We evaluated the performance of each classifier at multiple probabilistic prediction cutoffs ranging from 0 to 0.9 with an 0.1 increment. Figure 2 shows the overall average precision and recall at each cutoff for each classifier when validating on GSE109379 and SJCRH data sets. Although all models had their prediction precision increase as the threshold increased for both class (red line) and family (blue line) prediction, the recalls that met the cutoff dropped quickly to around 65% in RFmod and kNNmod, but it stayed above 0.75 in NNmod (Fig. 2 and Table S5-S8). For application to diagnostic tumor samples, an optimal calibrated score threshold of ≥ 0.9 was selected ⁶. For subclasses within methylation class families, a threshold value of ≥ 0.5 was defined as sufficient for a valid prediction, as long as all family member scores add up to a total score of ≥ 0.9. Single class specificity and sensitivity are provided in Table S5-S8. While RFmod and kNNmod had average balanced accuracies below 90% (82% and 86%, respectively), NNmod achieved 91% accuracy for methylation class prediction of GSE109379. At the 0.9 threshold for methylation class prediction, NNmod maintained good recalls (> 82% for class and > 84% for family) with > 90% precision, while RFmod and kNNmod produced recalls of less than 60% (Fig. 2A and Table 2). When predicting SJCRH at the 0.9 threshold, NNmod and RFmod achieved comparable results with balanced accuracies of 94% and 93%, recalls of 89% and 87%, and precisions of 96% and 92%, respectively. Meanwhile, kNNmod got 84% accuracy with 70% recall and 80% precision, respectively (Fig. 2B and Table 2). These results suggested that NNmod could identify the most positive calls with higher accuracy and precision.

Table 2

Performance of each classifier when predicting methylation class in the independent test sets at 0.9 threshold
Data set	RF			kNN			NN
Data set	Precision	Recall	Accuracy	Precision	Recall	Accuracy	Precision	Recall	Accuracy
GSE109379	0.87	0.73	0.86	0.76	0.64	0.82	0.91	0.82	0.91
SJCRH	0.92	0.87	0.93	0.8	0.7	0.84	0.96	0.89	0.94

Model robustness

CNS tumor classification of our classifiers is based on features that measure DNA methylation at different CpG sites in the human genome using probes on Illumina BeadChip arrays. This microarray technology is easy to use, time-efficient and cost-effective. However, it keeps evolving, and in each new release, more probes are printed to cover more diverse genomic regions, and some probes are purposely removed for efficiency. Other potential applications, such as detection of tumors by cell free DNA testing, may also have uneven or missing values. Because the missing probes could differentially affect model performance, we investigated whether the performance of the three classifiers was robust in producing consistent outputs in class labels and their corresponding prediction scores even when a proportion of input probes were not present. We performed an experiment in which we randomly dropped 10% of the probes in the independent test data sets GSE109379 and SJCRH. We repeated this process 10 times to create 10 different missing probes scenarios. The robustness of each classifier was accessed based on the Theil’s U uncertainty coefficients between the two sets of predicted labels and Spearman’s correlation coefficients between prediction scores with and without missing probes. Table 3 shows that RFmod and NNmod have Theil’s U uncertainty coefficients greater than 0.94, suggesting that the predicted labels by RFmod and NNmod before probe drop-out were as similar as those produced after probe drop-out. In contrast, kNNmod has the lowest uncertainty coefficient among the three classifiers with Theil’s U ranging from 0.889 to 0.908 for methylation family and class prediction. These results indicate that the two sets of predicted labels are strongly associated. All models have Pearson correlation coefficients > 0.928 with p-values < 2.2e^-16, suggesting a strong and statistically significant linear correlation between prediction scores produced when 10% of probes were missing and when there were no missing probes (Table 4). Figure 3 shows the regression analysis of the two sets of classification scores. Scores produced by NNmod and kNNmod with drop-out data set were generally higher than those that were output using all probes as indicated with positive y-intercepts (Fig. 3). RFmod, when using data with missing probes, produced lower classification scores (negative y-intercepts). All models had the goodness-of-fit R-squared of at least 86%, indicating a strong correlation between the two sets of prediction scores (Fig. 3). These results suggest that missing probes do not affect the prediction outcomes of any classifiers.

Table 3

Theil’s U uncertainty coefficient with a 95% confidence interval of each classifier with and without dropping 10% of probes in the GSE109379 and SJCRH independent test sets
Data set	RFmod		kNNmod		NNmod
Data set	Class	Family	Class	Family	Class	Family
GSE109379	0.969 (0.967, 0.972)	0.974 (0.970, 0.978)	0.892 (0.887, 0.898)	0.908 (0.902, 0.914)	0.973 (0.970, 0.9796)	0.980 (0.976, 0.983)
SJCRH	0.945 (0.940, 0.949)	0.964 (0.960, 0.968)	0.889 (0.883, 0.894)	0.899 (0.894, 0.905)	0.970 (0.966, 0.974)	0.979 (0.975, 0.982)

Table 4

Pearson's correlation coefficients of prediction scores with and without 10% missing probes
Data set	RFmod		kNNmod		NNmod
	Class	Family	Class	Family	Class	Family
GSE109379	0.98	0.99	0.93	0.94	0.94	0.94
SJCRH	0.99	0.99	0.95	0.96	0.99	0.99

Model assessment based on sample purity

Infiltrating of normal cells such as epithelial, stromal, and immune cells in tumor tissue can perturb the tumor signal in molecular studies. In our application, this contamination can affect the methylation level measured by microarray chips, leading to possible degradation in the performance of a classifier. Therefore, we developed an in silico experiment in which different fractions of the normal control cells were mixed with the tumor tissue to answer these questions: (1) whether a classifier produces unexpected methylation class/family prediction (2) if yes, would the prediction have a suprathreshold score, and (3) approximately at what percentage of control contamination, the classification accuracy starts to degrade. We first observed the overall performance of the three classifiers based on their average recall and precision for methylation class and family prediction at different thresholds and purity fractions (Fig. 4). NNmod started to perform the best, as seen in previous sections. RFmod degraded at a comparable rate with kNNmod after the sample purity was less than 65%. As the purity of tumor samples was less than 40%, NNmod started to yield lower precision and recall compared to the other two classifiers (Fig. 4, threshold = 0). At the 0.9 clinical threshold and 0.95 purity, NNmod had twice the average recalls and a much higher average precision than RFmod and kNNmod. The performance of NNmod did not start to degrade at a similar rate to RFmod and kNNmod until the purity of tumor samples was less than 50%. As the contamination increased, RFmod had the lowest performance among the three classifiers (Fig. 4, threshold = 0.9). NNmod maintained the highest average precision and recall among the three classifiers. Its performance reduced to a comparable level with RFmod and kNNmod only when the tumor purity was less than 50%.

Next, we observed the prediction results of each classifier for each methylation class and family. Figures 5 and 6 show the performance of RFmod, kNNmod, and NNmod at different control fractions in the tumor sample for methylation class diffused midline glioma H3 K-27 mutant (DMG, K27) and glioblastoma, IDH wildtype, H3.3 G34 mutant (GBM, G34), respectively. When the high-grade DMG, K27 tumors got contaminated with control, RFmod and NNmod did not produce unexpected methylation classes besides DMG, K27 and its corresponding mixed control cerebella hemisphere (CONTR, CEBM) class (Fig. 5A, B, G, and H) or family (Fig. S4A, C, D, and F). On the other hand, kNNmod unexpectedly predicted these high-grade gliomas to be low-grade pilocytic astrocytoma (LGG, PA PF) with scores above the clinical threshold (0.9) (Fig. 5D, E and Fig. S4B, E). kNNmod and RFmod could not accurately predict the methylation class of the DMG, K27 tumors if the purity of these samples was less than 70% (Fig. 5C, F and Fig. S4G, H). Meanwhile, NNmod was able to maintain its prediction accuracy prediction for DMG, K27 samples unless the sample’s purity dropped below 40% (Fig. 6I). Figure 6 shows that when predicting the GBM, G34 methylation class, RFmod did not provide any suprathreshold results if more than 30% of control tissues were present in the mixture (Fig. 6A, B, and C). On the contrary, kNNmod accurately predicted these samples until the contamination was up to 60%. At this fraction, kNNmod unexpectedly classified these grade IV glioblastomas as grade I dysembryoplastic neuroepithelial gliomas (LGG, DNT) (Fig. 6D, E and F). NNmod did not provide any suprathreshold classification to GBM, G34 samples besides their corresponding normal hemispheric cortex (CONTR, HEMI) starting at 70% contamination (Fig. 6G, H and I). Similar results were shown in Fig. S5 for GBM, G34 samples at the methylation family.

We developed a deep neural network model to predict CNS tumor classification based on a large DNA-methylation data set from 2801 patients of 82 distinct CNS tumors and 9 controls. Our multilayer perceptron neural network classifier achieved high performance, as demonstrated in 3 different evaluation settings. Compared with RFmod, a current-state-of-the-art CNS tumor classifier based on DNA-methylation ⁶, our NNmod showed higher overall accuracy (99% vs. 98%), precision (98% vs. 97%) and recall (98% vs. 96%) and comparable specificity (~ 99%) in methylation class prediction (Table 1). Among the three developed models, the kNN model produced the lowest accuracy (95%), precision (86%), and sensitivity (90%) (Table 1). In addition, we showed that our DNN model is highly robust and generalizable as evaluated in an independent testing dataset of 1104 GSE109379 samples (65 tumor classes) and 700 classifiable SJCRH samples (45 tumor classes), with an overall accuracy of 91% and 94%. Among these results, NNmod showed the highest accuracy and the best balance between precision and recall compared to RFmod and kNNmod (Table 1, Fig. 1–2).

All classifiers were trained on the reference data (GSE90496) generated from the Illumina Human Methylation 450K chips. These chips featured 485,577 CpG sites throughout the human genome, but they became obsolete and have been replaced by the Ilumina HumanMehtylationEPIC BeadChip (EPIC). EPIC measures methylation at > 850,000 CpG sites and covers approximately 90% of the same sites represented on the 450K chip. EPIC eliminates sites reported to be poorly performed ³⁰ and features more CpGs that cover more regulatory elements. When using classifiers trained on data produced by 450K chips to predict samples ran on EPIC chips, it is possible that some probes used for prediction are no longer present on EPIC chips and could hinder the classifier performance. As such, we performed a random probes drop-out experiment to evaluate the classification performance of RFmod, kNNmod, and NNmod (Tables 3 and 4). Although having the probes dropped out randomly may be adequate, it would be additionally useful to know in the future whether dropping all the poorly performed probes in the 450K training data set would enhance the performance and increase the robustness of all classifiers.

Diagnosis of CNS tumors is a complex multiclass classification problem as the number of diagnostic classes in which patients are stratified is not limited to a few selected classes but rather to a very high list of entities represented in the 5th edition of the WHO Classification of CNS Tumors ⁶. It has been shown that diagnostic accuracy can be improved by utilizing a robust machine-learning classification algorithm based on DNA-methylation profiles obtained from formalin-fixed, paraffin embedded (FFPE) or frozen tissue samples ⁶.The preparation of FFPE samples is one of the most widely used procedures to preserve and archive specimens in clinical oncology. This workflow requires an invasive tissue biopsy to be performed on patients. Recently, the use of liquid biopsies, a less invasive method for cancer detection has rapidly gained prominence ³¹. Particularly, plasma cell-free DNA methylation profiles have been shown to be highly sensitive, cost effective, and accurate in early tumor detection for cancer interception, and for multi-cancer classification ^32,33. Our study demonstrated that NNmod was the top stand-alone classifier among the three developed classifiers using DNA-methylation signatures from FFPE samples. The 11-layer perceptron NNmod maintained high recalls (> 82% for GSE109379 and > 90% for SJCRH) above an 0.9 clinical threshold with > 0.92 precision when validated with two independent data sets (Fig. 2 and Table S5-S8). With these improvements over RFmod, NNmod represents a viable method that could be used in conjunction with clinical, histopathologic, and molecular data to aid in the diagnosis and classification of CNS tumors. Future study would be to apply this machine learning modeling with the DNA-methylation profiles from plasma cell-free DNA obtained through the less invasive liquid biopsy procedure.

Patient material. FFPE or frozen tumor samples representing pediatric patient samples encountered on the typical pathology service were evaluated. The samples represented 650 samples expected to be present in the reference series and 300, true negative samples representative of non-brain solid tumors known to be absent from the reference series (Table S9).

Training and independent testing data sets

All supervised models were trained on the genome-wide DNA methylation profiles from the CNS tumor reference cohort (GSE90496), consisting of 2,801 samples from 91 methylation classes ⁶. All classifiers were independently validated with two methylation data sets, including the 950 CNS tumor samples from the St. Jude Children's Research Hospital (SJCRH) and 1,104 CNS tumor samples from GSE109379 ⁶.

Data generation and methylation array processing. We analyzed the 950 independent test samples using Illumina Methylation BeadChip (EPIC) arrays according to the manufacturer's instructions. In summary, DNA was isolated from formalin-fixed paraffin-embedded (FFPE) tumor tissue using the Maxwell® Clinical Sample Concentrator system (Promega, Madison, WI). Following extraction, DNA was quantified using a Qubit fluorometer and quantitation reagents (Thermo Fisher Scientific, Waltham, MA), and bisulfite converted using the Zymo EZ DNA methylation kit (Zymo Research, Irvine, CA). The overall DNA input amount was approximately 250 ng. DNA methylation profiling was carried out with the Infinium HumanMethylationEPIC BeadChip (850K) array (Illumina Inc., San Diego, CA) on the Illumina iScan platform.

All methylation data analyses, including those from GSE90496 and GSE109379, were performed in R (http://www.r-project.org, version 3.5.3), using several packages from Bioconductor and other repositories. Specifically, array data were preprocessed using the minfi package (v.1.28.4) ³⁴. Background correction with dye-bias normalization was performed for all samples using noob (normal-exponential out-of-band) with the "single" dye method ³⁵. Batch effects such as hybridization time and other technical variables were removed using removeBatchEffect from the limma package (v.3.38.3) ³⁶. Probe filtering was performed after normalization. Specifically, we removed probes located on sex chromosomes, probes containing nucleotide polymorphism (dbSNP132 Common) within five base pairs, including the targeted CpG-site, or mapping to multiple sites on hg19 (allowing for one mismatch), as well as cross-reactive probes.

Semi-supervised analysis. We developed a combination approach including a self-training with editing using a support vector machine (SETRED-SVM) as the base learner model with an L2-penalized, multinomial logistic regression model to obtain high confidence labels from a few reference instances ²⁹. We applied this approach on GSE109379 and the SJ samples to get labels for the independent validation purpose of the supervised models. The ssc R package (v2.1-0) was used to build and train the SETRED-SMV semi-supervised model. First, the standard deviation for each probe across all 2,801 samples from GSE90496 was calculated. Input features for SSL models were the 5072 probes with a standard deviation greater than 0.3 across all 2801 samples. We used the best SETRED-SVM model to predict the methylation class for 1104 GSE109379 and 950 SJ samples. The SSL scores were calibrated with an L2-penalized, multinomial logistic regression. Scores above the 0.8 threshold were considered correctly classifiable ²⁹.

The random forest algorithm and development. The random forest algorithm was reconstructed from Capper's algorithm ⁶ using the randomForest R package (v.4.6–14) ^37,38. This model was trained based on the 408,862 overlapping probes of the 450K and 850K array probes. First, the 10,000 features (or probes) with the highest importance scores were selected by splitting the 408,862 intersecting probes into 43 sets of approximately 9500 probes. Next, one hundred trees were fitted for each set using 639 randomly sampled candidate features at each split (mtry = square root of 408,862). The subclass labels, stratified subsampling methods, and the number of trees in the forest were followed as in ⁶. This framework can produce a model that either predicts the methylation class or the methylation "family" scores ⁶ that represent clinically-equivalent families on which Capper et al. witnessed their best error rates. Next, a multinomial logistic regression was used to calibrate the prediction scores from all cross-validation splits as previously described ⁶. The family scores were then generated as the sum of all methylation class scores from the trained random forest.

The k-nearest neighbor algorithm and development (kNNmod)

An exact bootstrap k-nearest neighbor model (kNNmod) was built as described in ³⁹. The model was trained on score vectors constructed based on the difference in median beta values of the top 100 hyper- and hypo-methylated probes. Each set of 100 top probes was selected based on the mean ß values in a methylation group and the absolute z-scores computed by taking the differences between mean beta values of two CNS methylation groups divided by the square root of the sum of the variance in each group. Hence, each methylation group had a list of 200 probes that were either most hypo- or hypermethylated based on the absolute z-scores. Each sample had a vector of scores, i.e. one score per methylation group. Each score was computed by taking the median ß values of the top 100 hypermethylated probes and subtracting that from the top 100 hypomethylated probes. Euclidean distance on these vectors was used to measure the distance between each pair of samples. The entire Euclidean distance matrix on the methylation group score vectors was computed for all pairwise samples.

To classify a new sample, kNNMod ordered all other samples by their distance from the new observation and derived the probability that those neighbors would be included among the k nearest neighbors in the binomial distribution. We used k = 5 neighbors for classification because some subgroups were very rare. For each new sample, the exact bootstrap probability of assignment to each methylation group can be conditionally computed on the training data set and the resulting probe selection and group score definition.

The multilayer sparse perceptron architecture and development (NNmod)

The design of the multilayer sparse perceptron is shown in Figure S1. This design is based on two primary assumptions, (i) the methylation data from central nervous system tumors and normal brain is embedded in low dimensional space, and (ii) random combinations of important probes can predict methylation class. The first assumption is typical of high dimensional data and is supported by examination of the singular value decomposition (SVD) of previously published reference data ⁶ (data not shown). In addition, the ability of combined methylation probes to predict methylation class is supported by previous implementations of random forest classifiers ⁶.

We constructed an 11-layer perceptron neural net. The input dimension is 51,108, composed of probes selected with feature extraction described in the network training section immediately following this section. The first layer is sparse, while the remaining ten layers are fully connected. The sparse layer maps 139,264 uniformly random sets of 256 features to the space [0,1]⁵¹². In other words, the sparse layer computes cosines of angles of vectors of length 256 drawn with uniform probability without replacement from the 51,108 input probes, which were selected by a LASSO model. This sparse feature layer can be considered a forest of random decision stumps ⁴⁰, the output of which is fed through a standard perceptron.

Stochastic gradient descent was performed with a batch size of 32 on logarithms of output scores from the network using a learning rate of 0.001. These gradient descent parameters were obtained via a random search of the parameter space. The log-likelihood loss was minimized over the three-fold cross-validation. Using the evaluation partitions from the cross-validation splits, model calibration was performed with a multinomial logistic regressor. The final model was trained on the complete 2,801 samples using identical parameters following cross-validation.

Classifier cross-validation. To reduce the overfitting problem when training classifiers on high-dimensional data, all classifiers were cross-validated based on 1000 leave-out-25% cross-validations. We randomly selected 75% of the data used to train the classifiers (GSE90496), while the remaining 25% of the data were used for predictions. Stratified random sampling was performed for each methylation class or family to ensure the number of categories remained the same in each iteration. This validation process was repeated 1000 times (Fig. S2).

Model calibration. Calibration of machine learning methods may be necessary because the scores output by the classifier may have different scales when broken down by class, even when the scores are normalized so that they sum to 1. This poses problems for comparing the uncertainty in class or family calls between cases or even in the same case. Thus, the scores must be rescaled to form a well-calibrated multinomial distribution with minimal differences between expected values and variances between the class call groups.

Both RF and NN models were calibrated with the same multinomial logistic regression approach described by ⁶. The glmnet package (v-4.1-3) ⁴¹ was used with R bindings for the random forest and python bindings for the neural net.

Model robustness. To test whether missing methylation probes (features) affect our machine learning models, we randomly dropped 10% of the probes from the testing data (GSE109379 and SJCRH) and calculated the accuracy. The same probes at each round were used for all models. This process was repeated ten times to create 10 different missing sets of probes. Pearson’s correlation and Theil’s U uncertainty coefficients were computed using the ggpubr R package (v.0.4.0) and the DescTools R package (v.99.44), respectively. Pearson’s correlation coefficients with p-values were calculated to examine the linear relationship between the two sets of prediction scores (with and without missing probes). Theil’s U uncertainty coefficients were calculated to measure the nominal association between the two sets of labels predicted by the three classifiers on samples of GSE109379 and SJCRH data with and without missing probes.

Purity analysis. We performed an in silico simulated impurity experiment using different fractions of control and positive samples in GSE109379 and SJCRH test sets. The experiment was performed based on m-values. The in silico mixed m-values (m_mixed) for each positive sample were computed as follows

$${m}_{\text{m}\text{i}\text{x}\text{e}\text{d}}=(1-p){m}_{\text{t}\text{e}\text{s}\text{t}}+p{m}_{\text{c}\text{o}\text{n}\text{t}\text{r}\text{o}\text{l}}$$

where ${m}_{\text{t}\text{e}\text{s}\text{t}}$ is the input m-value from the positive samples, and ${m}_{\text{c}\text{o}\text{n}\text{t}\text{r}\text{o}\text{l}}$ is the average m-values of up to 19 appropriate control (i.e. normal) tissue samples in the test sets, p is the proportion of normal control tissues contaminated in a tumor sample (ranging from 0 to 1 with 0.05 increment). The control samples were selected based on their control methylation class corresponding to the methylation class tumor as described in Table S10. The final measurement of the mixed sample was then converted back to beta values for classifier inputs.

Model performance metrics. All models were evaluated based on accuracy, precision, specificity, recall, and F1 score. Classification accuracy is the number of correct predictions (true positives and true negatives) divided by the total number of predictions. Precision is the ratio of true positives to all the total positives predicted by a classifier. Specificity measures the proportion of true negatives correctly identified by a classification model. Recall or sensitivity is the ratio of true positives to all the ground truth positives. The F1-score is the harmonic mean of precision and recall and a good metric to measure the results in imbalanced classification problems. The higher the F1 score, the better the performance of a model. All measurements were computed using the caret R package (v.6.0–90).

CNS

central nervous system

kNN

k–nearest neighbors

RF–random forest

NN–multi–layer perceptron neural net

SOFTWARE AVAILABILITY: The generated code is available from the corresponding authors upon reasonable request for non-commercial use.

AUTHOR CONTRIBUTIONS: AB developed the MLPNet framework and implemented the RF model. QTT modified and maintained the models, analyzed the results, produced figures and tables, and drafted the manuscript. BAO conceptualized the project, interpreted the results and drafted the manuscript. TL and SP implemented the KNN model. RT, SJA extracted the DNA and produced methylation data. MC, LVF, GR, PN, AG, EA, SS, and DWE provided samples and interpreted the results. MC, DH, and TM modified and maintained the MLPNet. All authors reviewed and edited the manuscript.

COMPETING INTERESTS: All authors declare no competing interests.

MATERIALS and CORRESPONDENCE: All correspondence and material requests should be addressed to Dr. Brent Orr at [email protected] .

Ferguson, S. & Lesniak, M. S. Percival Bailey and the classification of brain tumors. Neurosurg Focus 18, e7 (2005). https://doi.org:10.3171/foc.2005.18.4.8
Kumar, R., Liu, A. P. Y., Orr, B. A., Northcott, P. A. & Robinson, G. W. Advances in the classification of pediatric brain tumors through DNA methylation profiling: From research tool to frontline diagnostic. Cancer 124, 4168–4180 (2018). https://doi.org:10.1002/cncr.31583
Moran, S. et al. Epigenetic profiling to classify cancer of unknown primary: a multicentre, retrospective analysis. Lancet Oncol 17, 1386–1395 (2016). https://doi.org:10.1016/S1470-2045(16)30297-2
Pajtler, K. W. et al. Molecular Classification of Ependymal Tumors across All CNS Compartments, Histopathological Grades, and Age Groups. Cancer Cell 27, 728–743 (2015). https://doi.org:10.1016/j.ccell.2015.04.002
Sturm, D. et al. New Brain Tumor Entities Emerge from Molecular Classification of CNS-PNETs. Cell 164, 1060–1072 (2016). https://doi.org:10.1016/j.cell.2016.01.015
Capper, D. et al. DNA methylation-based classification of central nervous system tumours. Nature 555, 469–474 (2018). https://doi.org:10.1038/nature26000
Smith, K. S. et al. Patient-derived orthotopic xenografts of pediatric brain tumors: a St. Jude resource. Acta Neuropathol 140, 209–225 (2020). https://doi.org:10.1007/s00401-020-02171-5
He, C. et al. Patient-derived models recapitulate heterogeneity of molecular signatures and drug response in pediatric high-grade glioma. Nat Commun 12, 4089 (2021). https://doi.org:10.1038/s41467-021-24168-8
Kumar, R. et al. Clinical Outcomes and Patient-Matched Molecular Composition of Relapsed Medulloblastoma. J Clin Oncol 39, 807–821 (2021). https://doi.org:10.1200/JCO.20.01359
Pugh, T. J. et al. Medulloblastoma exome sequencing uncovers subtype-specific somatic mutations. Nature 488, 106–110 (2012). https://doi.org:10.1038/nature11329
Robinson, G. et al. Novel mutations target distinct subgroups of medulloblastoma. Nature 488, 43–48 (2012). https://doi.org:10.1038/nature11213
Northcott, P. A. et al. Medulloblastoma comprises four distinct molecular variants. J Clin Oncol 29, 1408–1414 (2011). https://doi.org:10.1200/JCO.2009.27.4324
Hovestadt, V. et al. Robust molecular subgrouping and copy-number profiling of medulloblastoma from small amounts of archival tumour material using high-density DNA methylation arrays. Acta Neuropathol 125, 913–916 (2013). https://doi.org:10.1007/s00401-013-1126-5
Northcott, P. A. et al. The whole-genome landscape of medulloblastoma subtypes. Nature 547, 311–317 (2017). https://doi.org:10.1038/nature22973
Broniscer, A. et al. Gliomatosis cerebri in children shares molecular characteristics with other pediatric gliomas. Acta Neuropathol 131, 299–307 (2016). https://doi.org:10.1007/s00401-015-1532-y
Danielsson, A. et al. MethPed: a DNA methylation classifier tool for the identification of pediatric brain tumor subtypes. Clin Epigenetics 7, 62 (2015). https://doi.org:10.1186/s13148-015-0103-3
Fukuoka, K. et al. Clinical impact of combined epigenetic and molecular analysis of pediatric low-grade gliomas. Neuro Oncol 22, 1474–1483 (2020). https://doi.org:10.1093/neuonc/noaa077
Weinberger, K. Q. & Saul, L. K. Distance Metric Learning for Large Margin Nearest Neighbor Classification. J. Mach. Learn. Res. 10, 207–244 (2009).
Zeng, Y., Yang, Y. & Zhao, L. Nonparametric classification based on local mean and class statistics. Expert Syst. Appl. 36, 8443–8448 (2009). https://doi.org:10.1016/j.eswa.2008.10.041
Mylonas, P., Wallace, M. & Kollias, S. in Methods and Applications of Artificial Intelligence Vol. 3025 Lecture Notes in Computer Science (eds G.A. Vouros & T. Panayiotopoulos) (Springer, Berlin, Heidelberg, 2004).
Sakr, S. et al. Comparison of machine learning techniques to predict all-cause mortality using fitness data: the Henry ford exercIse testing (FIT) project. BMC Med Inform Decis Mak 17, 174 (2017). https://doi.org:10.1186/s12911-017-0566-6
Aci, M., İnan, C. & Avci, M. A hybrid classification method of k nearest neighbor, Bayesian methods and genetic algorithm. Expert Syst. Appl. 37, 5061–5067 (2010). https://doi.org:10.1016/j.eswa.2009.12.004
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. Artificial intelligence in radiology. Nat Rev Cancer 18, 500–510 (2018). https://doi.org:10.1038/s41568-018-0016-5
Kotsiantis, S. B., Zaharakis, I. D. & Pintelas, P. E. Machine learning: a review of classification and combining techniques. Artificial Intelligence Review 26, 159–190 (2007). https://doi.org:10.1007/s10462-007-9052-3
Libbrecht, M. W. & Noble, W. S. Machine learning applications in genetics and genomics. Nat Rev Genet 16, 321–332 (2015). https://doi.org:10.1038/nrg3920
Balyen, L. & Peto, T. Promising Artificial Intelligence-Machine Learning-Deep Learning Algorithms in Ophthalmology. Asia Pac J Ophthalmol (Phila) 8, 264–272 (2019). https://doi.org:10.22608/APO.2018479
Currie, G., Hawk, K. E., Rohren, E., Vial, A. & Klein, R. Machine Learning and Deep Learning in Medical Imaging: Intelligent Imaging. J Med Imaging Radiat Sci 50, 477–487 (2019). https://doi.org:10.1016/j.jmir.2019.09.005
Peiffer-Smadja, N. et al. Machine learning for clinical decision support in infectious diseases: a narrative review of current applications. Clin Microbiol Infect 26, 584–595 (2020). https://doi.org:10.1016/j.cmi.2019.09.009
Tran, Q. T., Alom, M. Z. & Orr, B. A. Comprehensive study of semi-supervised learning for DNA methylation-based supervised classification of central nervous system tumors. BMC Bioinformatics 23, 223 (2022). https://doi.org:10.1186/s12859-022-04764-1
Pidsley, R. et al. Critical evaluation of the Illumina MethylationEPIC BeadChip microarray for whole-genome DNA methylation profiling. Genome Biol 17, 208 (2016). https://doi.org:10.1186/s13059-016-1066-1
Diaz, L. A., Jr. & Bardelli, A. Liquid biopsies: genotyping circulating tumor DNA. J Clin Oncol 32, 579–586 (2014). https://doi.org:10.1200/JCO.2012.45.2011
Shen, S. Y. et al. Sensitive tumour detection and classification using plasma cell-free DNA methylomes. Nature 563, 579–583 (2018). https://doi.org:10.1038/s41586-018-0703-0
Stackpole, M. L. et al. Cost-effective methylome sequencing of cell-free DNA for accurately detecting and locating cancer. Nature Communications 13, 5566 (2022). https://doi.org:10.1038/s41467-022-32995-6
Aryee, M. J. et al. Minfi: a flexible and comprehensive Bioconductor package for the analysis of Infinium DNA methylation microarrays. Bioinformatics 30, 1363–1369 (2014). https://doi.org:10.1093/bioinformatics/btu049
Triche, T. J., Jr., Weisenberger, D. J., Van Den Berg, D., Laird, P. W. & Siegmund, K. D. Low-level processing of Illumina Infinium DNA Methylation BeadArrays. Nucleic Acids Res 41, e90 (2013). https://doi.org:10.1093/nar/gkt090
Smyth, G. K. & Speed, T. Normalization of cDNA microarray data. Methods 31, 265–273 (2003). https://doi.org:10.1016/s1046-2023(03)00155-5
Ho, T. K. in IEEE. 278–282.
Liaw, A. & Wiener, M. Classification and Regression by randomForest. R news 2, 18–22 (2002).
Steele, B. M. Exact bootstrap k-nearest neighbor learners. Machine Learning 74, 235–255 (2009). https://doi.org:10.1007/s10994-008-5096-0
in Encyclopedia of Machine Learning (eds Claude Sammut & Geoffrey I. Webb) 262–263 (Springer US, 2010).
Friedman, J., Hastie, T. & Tibshirani, R. Regularization Paths for Generalized Linear Models via Coordinate Descent. J Stat Softw 33, 1–22 (2010).

(Not answered)

FigS1MLPNetArchitecture.png
Fig. S1. Architecture of the multi-layer perceptron neural network. We constructed an 11-layer perceptron neural net. The input dimension is 51,108, composed of probes selected with feature extraction described in the Methods section. The first layer is sparse, while the remaining ten layers are fully connected. The sparse layer maps 139,264 uniformly random sets of 256 features to the space [0,1]⁵¹². This layer is a forest of random decision stumps ⁴⁰ that computes cosines of angles of vectors of length 256 drawn with uniform probability without replacement from the 51,108 input probes, which were selected by a LASSO model. The output of this layer is then fed through a standard perceptron.
FigS2TrainTestValidationscheme.gif
Fig. S2. Training, testing, and validation scheme of all classifiers. To reduce the overfitting problem when training classifiers on high-dimensional data, all classifiers were cross-validated based on 1000 leave-out-25% samplings. We randomly selected 75% of the data used to train the classifiers (GSE90496), while the remaining 25% of the data were used for predictions. Stratified random sampling was performed for each methylation class or family to ensure the number of categories remained the same in each iteration. This training and testing process was repeated 1000 times. The final models were validated with two independent data sets: GSE109379 and the St. Jude data set.
FigS3HeatmapAccuracyPRF1family.png
Fig. S3. Leave-out-25% testing results of each methylation family. (A) Heat map showing methylation family prediction results after 1000 stratified random samplings (i)RF, (ii) kNN, and (iii) NN classifier incorporating information of n = 2,801 reference tumor samples allocated to 75 methylation class families (GSE90496). Deviations from the bisecting line represent misclassification errors (using the maximum calibrated score for class prediction). Boxplots showing (B) the accuracy, (C)precision and recall, and (D) F1-score for each classifier with outliers.
FigS4PurityDMGK27family.tif
Fig. S4. Classification results of RF, kNN, and NN model for high-grade diffused midline glioblastoma with K-27 mutant (DMG, K27) methylation family at different contamination levels. (A-C) Density plots of all calls (blue curve) and calls over the 0.9 clinical threshold (orange curve) at each possible methylation family predicted by RF, kNN, and NN when the ground truth is DMG, K27 at different fractions of control tissue contamination. (D-F)Box plots show the score distribution for each methylation family predicted by RF, kNN, and NN models. (G-I) Accuracy of each classifier at each purity level.
FigS5PurityGBMG34family.tif
Fig. S5. Classification results of RF, kNN, and NN model for grade IV glioblastoma, IDH wildtype, H3.3 G34 mutant (GBM, G34)methylation family at different levels of contamination. (A-C) Density plots of all calls (blue curve) and calls over the 0.9 clinical threshold (orange curve) at each possible methylation family predicted by RF, kNN, and NN when the ground truth is GBM, G34 at different fractions of control tissue contamination. (D-F) Box plots show the score distribution for each methylation family predicted by RF, kNN, and NN models. (G-I)Accuracy of each classifier at each purity level.
TableS1GSE90496caretmetricssubclass.xls
Table S1. Cross-validation performance metrics per methylation class for kNN, RF, and NN
TableS2GSE90496caretmetricsfamily.xls
Table S2. Cross-validation performance metrics per methylation family for kNN, RF, and NN
TableS3GSE109379SSL.xls
Table S3. Semi-supervised labeling results with top calibration scores for GSE109379 (n=1104)
TableS4SJ950casesSSL.xls
Table S4. Semi-supervised labeling results with top calibration scores for 950 SJCRH samples
TableS5ACCclassGSE109379.xls
Table S5. Performance of kNN, RF, and NN when predicting methylation class in GSE109379 validation data set at 0.9 classification probabilistic threshold
TableS6ACCfamilyGSE109379.xls
Table S6. Performance of kNN, RF, and NN when predicting methylation family in GSE109379 validation data set at 0.5 classification probabilistic threshold
TableS7ACCsubClasssjindep.xls
Table S7. Performance of kNN, RF, and NN when predicting methylation class in SJCRH validation cohort at 0.9 classification probabilistic threshold
TableS8ACCfamilysjindep.xls
Table S8. Performance of kNN, RF, and NN when predicting methylation family in SJCRH validation cohort at 0.5 classification probabilistic threshold
TableS9clinicaldataSJCRH.xlsx
Table S9. St. Jude Children’s Research Hospital (SJCRH) validation cohort (n=950) characteristics
TableS10.docx
Table S10. Control tissues used for the in silico mixing experiment

Download PDF

Journal Publication

published 02 Oct, 2024

Read the published version in npj Precision Oncology →

Editorial decision: revise
19 Mar, 2024
Review #1 received at journal
24 Feb, 2024
Review #3 received at journal
20 Feb, 2024
Reviewer #3 agreed at journal
19 Feb, 2024
Reviewer #2 agreed at journal
19 Feb, 2024
Reviewer #1 agreed at journal
11 Feb, 2024
Reviewers invited by journal
07 Feb, 2024
Editor assigned by journal
26 Jan, 2024
Submission checks completed at journal
26 Jan, 2024
First submitted to journal
25 Jan, 2024

You are reading this latest preprint version

DNA-Methyaltion-Based Deep Learning for Precision Classification of Central Nervous System Tumors: A Comparative Study

Status:

Journal Publication

Version 1

Abstract

Figures

INTRODUCTION

RESULTS

Model performance on train and test set:

Model performance on two independent validation sets:

Model robustness

Model assessment based on sample purity

DISCUSSION

MATERIALS and METHODS

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1