Validation of new, circulating biomarkers for gliomas

doi:10.21203/rs.3.rs-5314367/v1

Download PDF

Research Article

Validation of new, circulating biomarkers for gliomas

https://doi.org/10.21203/rs.3.rs-5314367/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background: Biomarkers are useful clinical tools but only a handful of them are used routinely for patient care. Despite intense efforts to discover new, clinically useful biomarkers, very few new circulating biomarkers were implemented in clinical practice in the last 40 years. This is mainly due to rather poor clinical performance.

Methods: Here, our goal was to validate the performance of a group of newly discovered circulating biomarkers for glioma by comparing our data with data from a paper recently published in Science Advances. We analyzed our own sets of clinical samples (gliomas (n=30), meningiomas (n=20)) and a different analytical assay (Proximity Extension Assay, OLINK Proteomics) to compare the results of Shen and colleagues.

Results: Despite the sophistication of the utilized discovery method by the original investigators, we found that the newly proposed biomarkers for glioma (the best one presumably being SERPINA6) did not perform as originally claimed.

Conclusions: Scientific irreproducibility has been extensively discussed in the literature. A large proportion of newly discovered candidate biomarkers likely represent “false discovery” and significantly contribute to the propagation of irreproducible results between investigators. One of the best ways to assess the value of any new biomarker is by independent and extensive validation. Based on our previous classification of irreproducible results, we believe that this new work likely represents another example of biomarker false discovery.

biomarkers

glioma

false discovery

biomarker failures

SERPINA6

biomarker validation

Recently, Shen et al. completed a large study aiming to reveal mechanistic insights into glioma formation and identify circulating (plasma-based) biomarkers for disease diagnosis (1). They performed multidimensional proteomics of many tissues and fluids, some of which were collected at the vicinity of the tumor, and compared within and between patients to spot mechanistically and diagnostically interesting molecules related to gliomas. The authors generated long lists of differentially expressed candidate proteins. A number of these proteins were variably present in the peripheral blood of glioma and non-glioma patients, presumably opening the window for biomarker-based clinical applications, such as diagnostics. The authors’ confidence about this possibility for their most promising biomarker is summarized as follows: ‘The results further demonstrate the promise of SERPINA6 as a blood biomarker for rapid initial screening for gliomas.’(1).

Among the plethora of candidate biomarkers originating from this study, were proteins associated with ROS metabolic processes, mismatch repair-related, platelet-related, and glycolysis-related proteins, tumor suppressors, immunoglobulins, and protease inhibitors. Among all candidates, members of the SERPIN family of serine protease inhibitors, namely SERPINA2, 4, 6 and 7, were given heightened attention, with SERPINA6 appearing to be lower in the blood of glioma vs non-glioma patients (some data are shown in their Fig. 6, where the sensitivity and specificity of SERPINA6 to detect glioma vs facial paralysis was approximately 88% (1)).

Despite the potential diagnostic claim, such sensitivity/specificity values are not suited for rapid initial screening for gliomas since, given the low prevalence of gliomas in the general population (about 2–5 cases per 100,000), the expected positive predictive value (PPV) of this test would be < 2%, as per our earlier calculations for many other candidate biomarkers (2). The PPV represents the chance of somebody having the disease if the test is positive and is the ratio of the true positives over all positive results (for more details and additional explanations please see our previous contributions (2–4)). Furthermore, the overlap of SERPINA6 plasma concentration between glioma and non-glioma patients was quite significant. This is a notable weakness of any diagnostic biomarker (see the authors’ Figs. 5 and 6 (1)).

Here, we validate some diagnostic findings of the Shen et.al. paper by using independent patient groups from gliomas and meningiomas and a different (orthogonal) assay (PEA, proximity extension assay) to analyze our own plasma samples.

We analyzed a cohort of the following plasma samples, provided by the Northwestern University Brain Tumor Biobank that were collected at diagnosis but before therapy, from patients with gliomas (N = 30), and meningiomas (as benign controls) (N = 20). Analysis of these samples was performed at OLINK Proteomics facilities using the PEA technology, as described in detail elsewhere (5–7). The list of the 3,000 proteins that are included in the OLINK panel can be found on the OLINK Website (https://olink.com/). As per our previous validation (7), PEA is reproducible, with coefficients of variation (CVs) of < 20% for > 99% of the proteins. Our own efforts to discover novel glioma biomarkers by using PEA have been published elsewhere (8).

To successfully compare our results with those of Shen et al., an initial analysis of our data for the 15 proteins that were also discovered by Shen et al. was performed. SERPINA6 was selected for an in-depth analysis, due to the proposed diagnostic strength of this marker by the group (1). In particular, re-analysis of the SERPINA6 data from Shen et al. was conducted, followed by student-t tests and Wilcoxon signed-rank tests for both our analysis and the re-analysis.

Several proteins that were quantified by Shen et al. by multidimensional mass spectrometry were also present in the OLINK panel of 3,000 proteins. The concentration of these common proteins were compared between our series of glioma and meningioma patients to identify agreements and differences with the proteins identified by Shen et al. (1).

Shen et al. used 4 cohorts of patients for their discovery and validation experiments (1). These cohorts included serum and tissues, and can be seen in Fig. 1 of their paper. Below, we will show the re-analyzed Shen et al. data from all four cohorts, and our own data, for several biomarker candidates.

Figure 1 illustrates the differences in relative protein intensity between patients with glioma vs meningioma (our patient cohorts) for all the proteins included in the OLINK panel and that were also considered promising biomarkers in the Shen et.al paper. As mentioned above, we compared the protein values through t-tests. Our obtained data (Fig. 1) do not confirm the Shen et al findings. None of the examined proteins, including the most promising candidate biomarker, SERPINA6, was different between gliomas and meningiomas. SERPINA6 was slightly lower in the glioma group, but the difference was not statistically significant (p = 0.47).

Re-analysis of the data for SERPINA6

We separately re-analyzed the data for SERPINA6, which was the most promising glioma biomarker identified in the study of Shen et al. (raw data were available by the authors through their publication: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10871530/bin/sciadv.adk1721_data_s1_to_s4.zip).

Blood samples from patients with glioma were collected by the authors at three different locations (Cohort 1 from (1)). Although it is stated in the paper that SERPINA6 is decreased in glioma-derived peripheral serum (pV), in comparison to glioma artery (gA) and glioma vein (gV) serum, the plotted concentrations of SERPINA6 are slightly increased in the peripheral serum (pV) of some patients but the differences were not statistically significant (paired t-tests) (Fig. 2A).

For the second cohort (1), tissue samples were collected from normal brain, perineural and glioma patients. There were no measured SERPINA6 values from the normal tissues. A Wilcoxon test looking for differences in medians between the groups was not significant. The distribution of SERPINA6 in peritumoral tissue and glioma tissue is shown in Fig. 2B (Cohort 2).

We then compared the concentration of SERPINA6 in peripheral blood of controls (facial paralysis) (red dots) vs glioma patients (blue dots) (Cohort 3 data (1)). The glioma group had lower values than the controls (Wilcoxon p < 0.001) (Fig. 2C). This observation agrees with the reported data by Shen et al. (1).

We finally compared SERPINA6 concentration in peripheral blood of controls (facial paralysis) vs glioma patients, from cohort 4 data (1) (Fig. 2D). Indeed, the glioma patients had slightly lower values, in agreement with the authors’ claim, but the differences were barely significant by Wilcoxon test (p = 0.046).

In receiver operating characteristic (ROC) analysis for predicting glioma from controls by using serum SERPINA6, we found, in agreement with the authors, some discriminatory potential, which, however, was not sufficient for reliably diagnosing glioma with their blood test.

ROC curves to predict glioma vs non-glioma with SERPINA6 as biomarker (cohort 4 (1)) were constructed. The data were similar to the author’s reported results for cohort 4 (data not shown).

There is a considerable interest in developing disease-related biomarkers and applying them to improve patient care. Despite major investments in this area by granting agencies and commercial organizations, the yield has been disappointingly poor. The handful of cancer biomarkers that are used in the clinic today were discovered more than 40 years ago (9).

Previously, we and others commented on several reasons that contribute to newly discovered biomarker failures and identified pre-analytical, analytical, and post-analytical shortcomings which may affect a biomarker’s performance (10–17). In short, most of the failures are due to unrecognized biases/differences between the diseased and controlled clinical samples, the groups and numbers of patients and their clinical information, the analytical method used, and more recently, the way the data are interpreted (black box approach) (17, 18).

Recently, considerable efforts were made, to find ways to better reproduce published and seemingly promising biomarkers. It has been realized that a large number of manuscripts, published in even top-rated journals, describe false discovery (14). The number of retractions of manuscripts published in highly reputable journals is at the all-time high (19, 20).

One of the ways to decrease false discovery is to reproduce the findings, preferably independently, from the original investigators. This is not an easy task, since specific reagents and techniques may not be available to the validators. Also, this process is time-consuming and expensive. We proposed a simpler way to tackle the irreproducibility problem that we coined “the 5-year reflection” (21). Reproducing at least a fraction of high impact studies may reveal weaknesses which can lead to improved outcomes in the future.

In the paper under discussion, Shen et al. tried to identify new biomarkers for glioma by collecting blood samples from the peripheral circulation and from glioma arteries and veins in the vicinity of the tumor. Tissue samples were also collected. To validate their discovery data, they collected samples that were not used in the discovery phase.

In our previous work, we described the identification of glioma biomarkers by a different proteomic technique, the PEA (6, 8). Taking these results, we then examined which of the proteins identified by Shen et al. and showed biomarker promise, were in the PEA panel. Surprisingly, we found no overlap between the candidates in the two studies. Even the best putative biomarker selected by Shen et al. (SERPINA6) showed no differences between gliomas and meningiomas in our patients. Additional analyses have shown that SERPINA6 concentration was no different between the tumoral and peritumoral tissue. To explain some of these discrepancies, we suggest that stricter quantitative technologies are likely needed to identify differences between glioma and non-glioma patients in the discovery phase, with emphasis to be given to clinical utility.

Regarding the diagnostic power of SERPINA6, the sensitivity and specificity of the test was about 88%. This led the authors to conclude that it may be suitable for clinical use. However, even these seemingly high sensitivities and specificities are likely not enough for clinical use, as we exemplified elsewhere (2–4). Moreover, for a biomarker to be promising it needs to have clinical utility that complements statistical significance. A clinically useful biomarker needs to have either very high specificity + good sensitivity or very high sensitivity + good specificity. Simply put, having a significant difference in median/mean value between groups, does not confer clinical utility.

We conclude that the biomarkers identified by Shen et al. (1) were not successfully validated with our own sets of data from glioma and meningioma patients and the characteristics of the test, as published, is not sufficient to warrant any clinical applications at present.

Ethics approval and consent to participate

We previously obtained ethics certificates from the Research Ethics Board at Mount Sinai Hospital, Toronto, Canada (REB #21-0114-E), which has been renewed until August 4, 2025. Samples were provided by the Northwester University Brain Tumor Biobank.

Consent for publication

Not applicable

Availability of data and materials

The excel file with the concentrations of the biomarkers and their concentrations are available by request from the corresponding author.

Competing interests

The authors report no conflict of interests.

Funding

Our previous work was supported by a grant to EPD from the Canadian Brain Foundation and the Canadian Cancer Society Research Institute under a SPARK Grant (Grant Number: CCS707057). The Northwestern Nervous System Tumor Bank is supported by the P50CA221747 SPORE for Translational Approaches to Brain Cancer. The current work did not use any funding.

Authors’ contributions

MKC drafted and edited the manuscript, and prepared figure 2. LMA performed the statistical analysis, prepared figures 1 and 2 and edited the manuscript. EPD conceptualized, drafted and edited the manuscript.

Acknowledgements

Not applicable.

Shen L, Zhang Z, Wu P, Yang J, Cai Y, Chen K, et al. Mechanistic insight into glioma through spatially multidimensional proteomics. Sci Adv. 2024;10(7):eadk1721.
Fiala C, Diamandis EP. A multi-cancer detection test: focus on the positive predictive value. Ann Oncol. 2020;31(9):1267–8.
Diamandis EP, Li M. The side effects of translational omics: overtesting, overdiagnosis, overtreatment. Clin Chem Lab Med. 2016;54(3):389–96.
Fiala C, Taher J, Diamandis EP. P4 Medicine or O4 Medicine? Hippocrates Provides the Answer. J Appl Lab Med. 2019;4(1):108–19.
Chen M, Ren AH, Prassas I, Soosaipillai A, Lim B, Fraser DD, et al. Plasma Protein Profiling by Proximity Extension Assay Technology Reveals Novel Biomarkers of Traumatic Brain Injury-A Pilot Study. J Appl Lab Med. 2021;6(5):1165–78.
Ren AH, Diamandis EP, Kulasingam V. Uncovering the Depths of the Human Proteome: Antibody-based Technologies for Ultrasensitive Multiplexed Protein Detection and Quantification. Mol Cell Proteom MCP. 2021;20:100155.
Ren A, Prassas I, Sugumar V, Soosaipillai A, Bernardini M, Diamandis EP, et al. Comparison of two multiplexed technologies for profiling > 1,000 serum proteins that may associate with tumor burden. F1000Research. 2021;10:509.
Ghorbani A, Avery LM, Sohaei D, Soosaipillai A, Richer M, Horbinski C, et al. Discovery of novel glioma serum biomarkers by proximity extension assay. Clin Proteom. 2023;20(1):12.
Sturgeon CM, Duffy MJ, Stenman UH, Lilja H, Brünner N, Chan DW, et al. National Academy of Clinical Biochemistry laboratory medicine practice guidelines for use of tumor markers in testicular, prostate, colorectal, breast, and ovarian cancers. Clin Chem. 2008;54(12):e11–79.
Diamandis EP. The failure of protein cancer biomarkers to reach the clinic: why, and what can be done to address the problem? BMC Med. 2012;10(1):87.
Diamandis EP. Cancer Biomarkers: Can We Turn Recent Failures into Success? JNCI J Natl Cancer Inst. 2010;102(19):1462–7.
Diamandis EP. Analysis of serum proteomic patterns for early cancer diagnosis: drawing attention to potential problems. J Natl Cancer Inst. 2004;96(5):353–6.
Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer. 2005;5(2):142–9.
Ioannidis JPA, Discussion. Why An estimate of the science-wise false discovery rate and application to the top medical literature is false. Biostat Oxf Engl. 2014;15(1):28–36. discussion 39–45.
Ioannidis JPA. Molecular bias. Eur J Epidemiol. 2005;20(9):739–45.
Prassas I, Brinc D, Farkona S, Leung F, Dimitromanolakis A, Chrystoja CC, et al. False biomarker discovery due to reactivity of a commercial ELISA for CUZD1 with cancer antigen CA125. Clin Chem. 2014;60(2):381–8.
Petricoin EF, Ardekani AM, Hitt BA, Levine PJ, Fusaro VA, Steinberg SM, et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet Lond Engl. 2002;359(9306):572–7.
Schully SD, Carrick DM, Mechanic LE, Srivastava S, Anderson GL, Baron JA, et al. Leveraging biospecimen resources for discovery or validation of markers for early cancer detection. J Natl Cancer Inst. 2015;107(4):djv012.
Levett JJ, Elkaim LM, Alotaibi NM, Weber MH, Dea N, Abd-El-Barr MM. Publication retraction in spine surgery: a systematic review. Eur Spine J Off Publ Eur Spine Soc Eur Spinal Deform Soc Eur Sect Cerv Spine Res Soc. 2023;32(11):3704–12.
Ball P. Is AI leading to a reproducibility crisis in science? Nature. 2023;624(7990):22–5.
Fiala C, Diamandis EP. How to reduce scientific irreproducibility: the 5-year reflection. Clin Chem Lab Med. 2017;55(12):1845–8.

No competing interests reported.

Download PDF

Reviewers invited by journal
31 Oct, 2024
Editor assigned by journal
23 Oct, 2024
Submission checks completed at journal
23 Oct, 2024
First submitted to journal
22 Oct, 2024

You are reading this latest preprint version

Validation of new, circulating biomarkers for gliomas

Status:

Version 1

Abstract

Figures

Introduction

Materials and Methods

Results

Re-analysis of the data for SERPINA6

Discussion

Declarations

References

Additional Declarations

Status:

Version 1