HPN: A Multimodal Neural Network Model for Non-invasive HER2 Status Assessment in Breast Cancer Patients

doi:10.21203/rs.3.rs-5011153/v1

Background

HER2-positive breast cancer is known for its aggressive behavior and poorer prognosis in the absence of anti-HER2 therapy. Current assessments of HER2+ highlight the need for non-invasive diagnostic tools. This study introduces a multimodal approach called the HER2 Prediction Network (HPN) to noninvasively predict HER2 status, thereby supporting the precise administration of HER2-targeted therapies.

Methods

A cohort of 482 breast cancer patients were enrolled from Peking Union Medical College Hospital. HPN was developed by ResNet and Transformer, utilizing clinicopathological and ultrasound data collected from breast cancer patients. After training, this model could differentiate HER2-zero, HER2-low and HER2-positive breast cancer patient and detect HER2 status in different peritumoral regions.

Findings

The HPN demonstrated robust performance in HER2 expression identification of breast cancer patients. It achieved an Accuracy of 0.76 and an Area Under the Curve(AUC) of 0.86. Detections for different peritumoral regions have all shown favorable results(AUC_1.2x =0.85, AUC_1.4x=0.85 AUC_1.6x =0.86).

Conclusion

The HPN provided a non-invasive method for assessing HER2 expression, thereby facilitating decision-making regarding the intervention of HER2-targeted therapy.

Breast cancer

Human epidermal growth factor receptor 2

Neural Network Modeling，Artificial Intelligence-aided Diagnosis

Breast cancer (BC) is the second most common cancer worldwide after lung cancer, with approximately 2.3 million new cases in 2022, accounting for 11.6% of all cancer diagnoses(1). Human epidermal growth factor receptor 2 positive BC (HER2-positive BC) constitutes 20–25% of all BC cases. Patients with HER2 + BC who have not received anti-HER2 therapy typically face more aggressive tumor behavior and poorer prognosis(2).

According to the 2018 American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) guidelines, HER2-positive BC is defined as HER2 IHC 3 + or HER2 IHC 2 + with positive ISH results. HER2-low BC is defined as HER2 IHC 1 + or HER2 IHC 2 + with negative ISH results, while HER2-zero BC is characterized by HER2 IHC 0(3). With enhanced understanding of the biological characteristics and signaling pathways of HER2-positive BC, HER2-targeted drugs such as trastuzumab has led to a new era of targeted therapy(4). Subsequent clinical trials, including TBCRC022 and CLEOPATRA, have demonstrated the efficacy of combining tyrosine kinase inhibitors (TKIs) with dual-target therapies(5)^,(6). More inspiringly, for the common occurrence of brain metastases in advanced HER2-positive BC(2)^,(7), antibody-drug conjugates (ADCs) have been proven to reduce disease progression, particularly benefiting those with brain metastases(8).

Despite these advancements in targeting HER2-positive BC, HER2-low BC, comprising nearly half of all cases, is gaining importance. This subtype differed in molecular characteristics and treatment strategies compared to HER2-positive BC. The recent DESTINY-Breast 04 trial demonstrated the efficacy and safety of T-DXd in HER2-low expression BC with metastases, addressing a significant treatment gap(9).

Current HER2 diagnosis relies on surgical biopsy and immunohistochemistry(ISH). However, ISH is graded according to the degree of coloration on the cell membrane. Such clinical assays may lead to inaccuracies due to sampling bias(10). Additionally, changes in HER2 expression have been observed in 20–40% of patients undergoing neoadjuvant chemotherapy(NAC) in current clinical practice(11). Due to the intratumoral heterogeneity of HER2, real-time assessment of HER2 expression through multiple preoperative biopsies poses challenges and increases the additional time and economic costs for HER2-positive BC patients treatment. Therefore, developing a non-invasive method to assist in HER2 diagnosis is critically important to ensure all eligible patients receive appropriate treatment.

Deep learning models could analyze and process medical images to achieve the identification, thereby helping clinical practitioners conduct qualitative and even quantitative analysis of potential diseases(12). Deep learning algorithms like Residual Network(ResNet), Squeeze-and-Excitation Networks (SENet), and Graph Neural Networks (GNN) enhanced HER2 status prediction by integrating detailed image features and clinical data, resulting in more accurate and holistic diagnostic tools(13)^, (14). Ultrasonography (US) provides real-time, multi-angle, non-invasive, radiation-free, and repeatable evaluations, making it suitable for diagnosing breast diseases, especially in dense glandular tissues. However, in the experience of building artificial intelligence-aided diagnostic models using breast US, US images have been characterized by blurred boundaries, low contrast, ultrasound artifacts, and dispersive noise, making it difficult for prediction models to classify tumors in more detail from US images. Chen et al proposed Adaptive Attention U-net (AAU-net) to address the challenge of lesion segmentation in breast US images, especially for irregularly shaped malignant tumors(15). Although AAU-net could accurately segment breast tumors in US images, it could not classify tumors in more detailed molecular type.

Therefore, multiple modalities were tested to enhance the diagnostic performance of US radiomic models. First, the detection of breast tumors was expanded. Studies suggested that tumor microenvironment in peritumoral region is important in tumor molecular typing, biological features such as progression and metastasis(16). It has been practiced in MRI radiomics that a dual-modality MRI intratumoral and peritumoral radiomic characterization model has achieved higher diagnostic efficacy (AUC = 0.86) in predicting HER-2 status, superior to models of intratumoral or peritumoral imaging features alone(17). Furthermore, multimodal modeling incorporates parameters from multiple modalities in addition to US images to improve the diagnostic efficiency of the models, and most of studies have input patient clinical data into the models(18)^,(19), simulating real clinical diagnostic scenarios. Transformers have seen widespread application in medical image recognition, particularly in lesion detection(20). Their architecture, based on self-attention mechanisms, captures global information, overcoming the limitations of traditional convolutional neural networks (CNNs) in handling long-range dependencies.

Accordingly, this study has developed a multimodal US radiomic model called HER2 Prediction Network(HPN) for predicting HER2 expression in BC patients at various stages. By analyzing clinicopathological and breast US information, this multimodal model demonstrated favorable performance in HER2 status identification. The innovative approach can serve as a non-invasive and economic biomarker, aimed at identifying patients who would benefit from HER2-targeted therapies, thus providing a basis for clinical treatment decision-making of HER2-positive and HER2-low BC.

Patients

The Ethical Board of Peking Union Medical College Hospital (PUMCH) approved this retrospective study. Written informed consents from patients were not required owing to the retrospective study design, non-invasive nature of the intervention (Approval Number: K24C1567). Patients’ recruitment workflow was shown in Fig. 1.

Study Design

Initially, the precise status of HER2 was confirmed through post-operative pathological assessments. Subsequently, feature was extracted from US image and pathological data to construct HPN, thereby stratifying patients into three groups, HER2_zero, HER2_low and HER2_positive. In a third step, based on the HPN prediction results, the prognostic efficiency of HPN in predicting HER2 status was calculated and presented.

Clinicopathological Parameters Dataset

The postoperative pathology and ISH reports conformed to the standards set by the 8th edition of the International Union Against Cancer (UICC) and the American Joint Committee on Cancer (AJCC) in 2018(21). Detailed clinicopathological parameters were provided in Table 1. Among all the indicators, HER2 status was analysed as the dependent variable, with other indicators serving as independent variables.

Ultrasound Image Dataset

An EPIQ7 unit (Philips - Advanced Technology Laboratories, Bothell, WA, USA) equipped with a 12 − 5 MHz high-frequency linear array transducer was utilized for routine ultrasonographic evaluation of the breast tumor. Then, two sonographers with over five years of experience independently assessed the affected breast. BI-RADS criteria were referred to as criterion for grading breast tumors(22). In case of discrepancies between the sonographers, they negotiated until consensus was reached.

Standard-format US images were downloaded from the Picture Archiving and Communication System(PACS), and the slice with the largest cross-section of the breast lesion was selected as the input image for the HPN. The tumor regions in the original images were annotated by LabelMe software. Finally, the HER2 status prediction dataset(HER2_SP) was constructed, containing a total of 1168 US images.

Multimodal Modelling of Clinicopathological and Ultrasonic Image Data

For clinicopathological information, the text class information were structurally coded, and the subcategories of different indicators were represented by values such as 0, 1, 2, and then Pearson's correlation coefficient analysis were used to screen clinical features.

For US images, feature extraction of the Region of Interest (ROI) was performed using the ResNet18 network to obtain 1×512 dimensional features. To reduce the computational load and inference difficulty, Principal Component Analysis (PCA) was applied to reduce the image features from 512 dimensions to 32 dimensions.

For the multimodal model, a Transformer was utilized to build the HPN classifier. Initially, the dimensionality-reduced image features and filtered clinical features were concatenated. These input features were then processed by multilayer Transformer Encoders, and the resulting sequence features were fed into a linear classifier for classification. The Transformer Encoder comprises a multi-head self-attention mechanism and a feed-forward network, where inputs from each layer pass through sub-layers to produce outputs that are fed into the next layer. Each sub-layer includes Residual Connection and Layer Normalization. The structure is illustrated in Fig. 2, the source code is available on https://github.com/JinlinYY/HER2-pre. The development environment and tools used were: Ubuntu 20.04 operating system, Intel(R) Xeon(R) Gold 6330 processor, and GeForce RTX 3090 GPU. The dataset was divided into a training set and a test set in a 7:3 ratio. During the training phase, the CrossEntropy loss function and Adam optimizer were utilized. The classification performance of HPN was evaluated using metrics such as Accuracy, Precision, Recall, F1-score, ROC curves, and AUC. Additionally, HPN was compared with unimodal algorithms, including target detection algorithms and other machine learning algorithms.

Statistical Analyses

Continuous variables were represented as mean ± standard deviation, while categorical variables were expressed as numbers and percentages (%). The Kruskal-Wallis’s test, t-test and ANOVA were used for continuous variables, and the chi-square test was used for categorical variables to compare differences between groups. The diagnostic value of conventional US and the HPN for HER2 status was assessed by calculating accuracy (ACC), specificity (SPE) and false negative rate (FNR). Diagnostic performance was evaluated through receiver operating characteristic (ROC) curves and area under the curve (AUC), with differences between ROC curves of different groups analyzed using the DeLong test. Statistical analyses were conducted using SPSS, Python (version 3.8) and PyTorch (version 1.10.0), and a bilateral P value less than 0.05 was considered significant. Adobe Illustrator 2021 and https://www.processon.com were used for illustration rendering.

Clinicopathological Characteristics of the Patients

Between Jan 2021 and July 2023, 792 primary BC patients with was studied and finally 482 women with 1168 malignant breast lesions were enrolled for analysis. According to the ISH results, 288 was diagnosed with HER2-positive BC, 100 was diagnosed with HER2 -low BC, and 94 was diagnosed with HER2-zero BC. Other detailed clinicopathological features were presented in Table 1.

Table 1

Clinicopathological Characteristics of Participants in Training and Testing Set
Variables	Training Set(N = 337)	Testing Set(N = 145)	P-value
Age (Mean ± SD)	52.56 ± 11.57	51.21 ± 10.50	0.095
Clinical Stage			0.962
T1(≤ 20mm), no.(%)	215(64.80%)	87(60.00%)
T2 (21mm-50mm), no.(%)	115(34.12%)	54(37.24%)
T3(>50mm), no.(%)	7(2.07%)	4(2.76%)
Differentiation			0.130
Low Differentiation, no.(%)	128(37.98%)	64(44.14%)
Middle Differentiation, no.(%)	191(56.68%)	74(51.03%)
High Differentiation, no.(%)	18(5.34%)	7(4.83%)
Tumor infiltration			0.664
Carcinoma in situ	13(3.86%)	5(3.45%)
Invasive carcinoma	324(96.14%)	140(96.55%)
Ki-67			0.950
<20%	48(14.24%)	9(6.21%)
≥ 20%	289(85.76%)	136(93.79%)
EGFR			0.419
Positive, no.(%)	96(28.49%)	44(30.34%)
Negative, no.(%)	241(71.51%)	101(69.66%)
LVI			0.518
Positive, no.(%)	86(25.52%)	35(24.14%)
Negative, no.(%)	251(74.48%)	110(75.86%)
Nipple Involvement			0.121
Positive, no.(%)	17(5.04%)	5(3.45%)
Negative, no.(%)	320(94.96%)	140(96.55%)
Note: P* represents the difference between each clinicopathological variable between the training and testing sets. The chi-square test was used to compare the difference in categorical variables, while a Student's t-test was used to compare the difference in age, clinical stage and Ki-67.

Abbreviations: LVI, lymphovascular invasion; EGFR, epidermal growth factor receptor; Ki-67, Tumor proliferation marker-67

Prediction Performance of the Multimodal Model

HPN was cross experimented in the case of different backbone networks and different classifiers, the results are shown in Table 3. The generated confusion matrix, ROC curve, and evaluation metrics (Table 2, Fig 3a, 3b) indicated that the fusion model ResNet50-Transformer demonstrated the favorable performance. This classifier achieved an accuracy of 0.76 and an AUC of 0.86 for HER2 status prediction.

Table 2 The Performance Summary of the Intratumoral Models and Combined-region Models in Predicting HER2 Status of BC

Dataset	Method		ACC	PRE	Recall	F1-score	AUC	SPE	FNR
Intra	BP	ResNet18	0.69	0.67	0.69	0.65	0.81	0.79	0.47
		ResNet50	0.72	0.7	0.71	0.7	0.84	0.83	0.39
		ResNet101	0.7	0.69	0.7	0.66	0.83	0.81	0.44
	LSTM	ResNet18	0.7	0.7	0.7	0.67	0.83	0.81	0.44
		ResNet50	0.72	0.7	0.72	0.69	0.84	0.82	0.43
		ResNet101	0.7	0.67	0.7	0.66	0.83	0.8	0.46
	Trans former	ResNet18	0.73	0.72	0.73	0.70	0.85	0.83	0.40
		ResNet50	0.76	0.76	0.76	0.75	0.86	0.86	0.33
		ResNet101	0.74	0.73	0.74	0.72	0.86	0.83	0.39
1.2×cmb	Trans former	ResNet18	0.74	0.72	0.74	0.71	0.85	0.83	0.40
		ResNet50	0.74	0.73	0.74	0.72	0.85	0.84	0.38
		ResNet101	0.72	0.71	0.72	0.70	0.84	0.83	0.39
1.4×cmb	Trans former	ResNet18	0.74	0.73	0.74	0.71	0.85	0.80	0.39
		ResNet50	0.72	0.69	0.72	0.69	0.84	0.83	0.40
		ResNet101	0.72	0.68	0.72	0.68	0.82	0.80	0.44
1.6×cmb	Trans former	ResNet18	0.74	0.74	0.74	0.72	0.86	0.83	0.37
		ResNet50	0.73	0.70	0.73	0.71	0.86	0.84	0.38
		ResNet101	0.72	0.70	0.72	0.68	0.84	0.80	0.43

Intra: intratumoral region; cmb: combined-region model (including intratumoral and peritumoral regions with different magnification times, 1.2×=The magnification is 1.2 times; 1.4×=The magnification is 1.4 times; 1.6×=The magnification is 1.6 times;); ACC: accuracy; Pre: precision; SPE: Specificity; FNR: false negative rate

Comparison Between Multimodal and Unimodal Models

In this study, Random Forest, XGBoost, GaussianNB, Logistic Regression, Decision Tree, and KNN algorithms were applied to model clinicopathological data. The confusion matrices and ROC curves for these six machine learning algorithms were illustrated in Figures 4a and 4b, respectively. The Random Forest and XGBoost models demonstrated superior performance on the HER2 clinicopathological data. Specifically, the Random Forest model achieved the highest accuracy of 66.9%, while the XGBoost model obtained the highest F1-score of 0.58.

For US image data, ConvNeXt, Cascade RCNN, MS-RCNN, and Mask-RCNN algorithms were employed for HER2 status detection(Table 3). For HER2_positive images, all four models correctly classified the images and exhibited satisfactory segmentation performance. For HER2_low images, the ConvNeXt model correctly classified the images but slightly exceeded the actual segmentation boundaries, whereas the other three models failed in classification and exhibited large segmentation deviations. For HER2_zero images, the segmentation areas for all four models were incomplete, neglecting the cancerous region at the bottom. Only the ConvNeXt model correctly identified the image category.

In summary, the detection results of HPN surpassed those of unimodal models, with an accuracy of 0.76 and an AUC of 0.86. The overall performance of HPN demonstrated a superior diagnostic efficiency to predict the HER2 status of BC.

Table 3 Results of Multimodal Model and Unimodal Models

Method		Input Data	Acc	F1-score	Pre	Recall	Specificity	FPR	AUC
Uni modal	RF	Clinical Data	0.67	0.57	0.59	0.57	0.83	0.167	0.79
	XGBoost		0.65	0.58	0.60	0.57	0.82	0.172	0.79
	GaussianNB		0.63	0.48	0.54	0.48	0.82	0.183	0.73
	LR		61.4	0.42	0.42	0.45	0.81	0.193	0.68
	DT		60.0	0.50	0.51	0.50	0.80	0.200	0.65
	KNN		0.55	0.44	0.46	0.44	0.772	0.23	0.62
	ConvNeXt	US Image	0.32	0.46	0.38	0.61	0.558	0.442	0.44
	Cascade RCNN		0.31	0.44	0.36	0.58	0.517	0.483	0.45
	MS-RCNN		0.31	0.42	0.33	0.58	0.500	0.500	0.49
	Mask-RCNN		0.31	0.41	0.31	0.58	0.500	0.500	0.45
Multi modal	HPN	US Image and Clinical Data	0.76	0.75	0.76	0.76	0.86	0.14	0.86

To assist in the targeted adjuvant therapy for HER2-positive and HER2-low BC, this study developed a multimodal method based on breast US images with clinical parameters to assist in the HER2 status prediction in BC patients. The HPN demonstrated better diagnostic performance in distinguishing HER2 status compared to any unimodal method. Encouragingly, our model exhibited favorable discrimination ability between HER2-low and HER2-positive patients (AUC_{HER2-positive}=0.87, AUC_HER2-low=0.81). Previous studies on predicting HER2 status have reported AUC values ranging from 0.81 to 0.87(23)^,(24)^,(14). Compared with these studies, in HPN, Transformer-ResNet18 fusion modal has achieved a favorable diagnostic performance (AUC_overall=0.86) for HER2 status prediction. HPN has the potential to serve as a non-invasive imaging biomarker to complement postoperative pathology and ISH, improving the noninvasive diagnostic accuracy of HER2 status.

Most of the current studies on HER2 expression prediction are based on the intra-tumoral imaging features of BC primary foci, neglecting the peri-tumoral imaging features that may contain information about the tumor microenvironment (25). Evidence showed that peritumor imaging histological features could characterize the BC tumor microenvironment and that biological changes in the tissue surrounding a breast mass may be potential predictive or prognostic markers(26). This study is the first to develop a multimodal model for predicting HER2 status based on the peritumor region of breast US. Intratumoral, 1.2 x, 1.4 x, and 1.6 x peritumoral range were compared, and the results confirmed that expanding the detection of a 1.6 x peritumoral range improved the predictive efficacy of HER2-low BC (AUC _HER2-low from 0.77 to 0.81), whereas there was no significant increase in the overall predictive efficacy for HER2 expression. This may since US contains less sequence and image information compared to MRI(27). However, the results also suggested that the HPN can accommodate the range of diverse ROIs and maintain the stability of HER2 expression prediction. In comparison with other studies, it has been found that peritumoral texture features on DCE-MRI 9 to 12 mm from the outermost edge of the primary lesion better identified patients with HER-2-positive BC than intratumoral texture features, with an AUC of 0.85 (26). Further studies found that DCE-MRI-based peritumor (4 mm outward from the tumor margin) imaging was valuable in predicting HER-2 expression, and that models combining intratumor and peritumor imaging had higher diagnostic efficacy than intratumor or peritumor imaging alone(27).

There are still some problems in combining intratumoral and peritumoral BC radiomics, such as unstable features due to various ROI outlining modes, and large differences in model prediction performance due to different modeling methods. In terms of the criteria and extent of peritumoral augmentation, many studies have now utilized the basic method to center the breast mass and expand the corresponding diameter as the peritumoral area(28). However, while too small a peritumoral extent may miss some of the tissue information containing the tumor microenvironment of BC, too large a peritumoral extent may incorporate too much normal tissue, thus affecting the quantification of the tumor microenvironment. In future studies, it is more important to determine the cut-off value of the amplification range based on the surgical margins of BC, thereby exploring the optimal peri-tumor range and guide clinical practice. The quantitative analysis of combined intratumoral and peritumoral radiomic features has potential application in the prediction of HER2 expression in BC and provides a useful aid for therapeutic decision-making.

Some studies reported that some histopathological data such as tumor grade, lymphovascular invasion(LVI), Ki-67 and hormone receptor status may correlated with HER2 status(23)^,(24)^,(14). However, due to the limitations of using isolated pathological or imaging features for a comprehensive characterization of breast tumors, this study further investigated a multimodal model combining radiomics and clinicopathological data. This study is the first to propose a multimodal neural network model(HPN) for HER2 status prediction by US images and clinical features. HPN extracted image features by using ResNet as an image encoder, and structured encoding of clinical data. The image features were downscaled by PCA, after which the two modal features were fused. Finally, Transformer was used to construct a classifier to predict the HER2 status. The results showed that the HPN model demonstrated significant advantages over algorithms like Mask R-CNN, Cascade R-CNN, MS-RCNN, and ConvNeXt(29). Firstly, the Transformer's self-attention mechanism effectively captures long-range dependencies across the entire image, which is crucial for identifying subtle and dispersed features in BC US images(30). Moreover, unlike traditional convolutional neural networks(CNN) that rely on convolution operations, the highly modular structure of Transformers offers exceptional flexibility, making them more effective in handling deformed, irregular, and complex structures(29). This is particularly suitable for analyzing irregular and indistinct boundaries in BC US images. Although Transformers typically require large datasets for training, their strengths in pre-training and transfer learning enable them to perform well even with limited data. Similar results were observed when using Transformers to predict complete response in BC patients who received NAC (AUC=0.910)(31).

ResNet18 is a deep learning model known for its residual structure, which mitigates the vanishing gradient problem in deep networks. Compared to other ResNet variants, ResNet18 offers a simpler structure with lower computational requirements, making it efficient and suitable for resource-limited applications(32). In BC radiomics, ResNet18 has been effectively applied to classify breast images, demonstrating its ability to accurately identify features and improve diagnostic accuracy. For example, in MG radiomics, the ResNet18 model has shown excellent performance in predicting malignant microcalcifications in BC patients (AUC = 0.88)(33). Previous studies have documented the diagnostic performance of various machine learning algorithms in the context of BC(23)^,(24)^,(14). Random Forest (RF) integrates multiple decision trees to automatically assess feature importance, aiding in feature selection and model interpretation. In BC radiomics diagnosis, RF has shown unique advantages, with studies indicating an AUC of up to 0.929 in classification of benign and malignant breast lesions. This approach mitigates the risk of overfitting and enhances prediction accuracy and reliability. Additionally, the independent growth of decision trees facilitates parallel processing, accelerating training speed. These attributes make Random Forest highly effective for tasks such as classification, regression, and feature selection(34).

Some limitations have to be addressed in this study. The imbalance of dataset and single-center data restricted the performance of HPN. GANs might be a potential method for image generation and conduct multi-center studies to improve generalization and diagnostic efficacy(35)(36). The US images in this study were sourced from one manufacturer (Philips). More multiple manufacturers should be included in future models. Advanced multimodal models combine with mammography, MRI, and pathological data, can enhance performance. Combining deep learning-based tumor detection with radiomics can develop comprehensive clinical diagnostic software.

In conclusion, combining clinicopathological parameters with conventional US, HPN has offered a non-invasive and practical method for preoperatively HER2 expression prediction. It has also held the potential to assess HER2-low expression BC patients, thereby determining appropriate ADC drug treatment regimens. Future modeling of larger samples and multimodal data is needed to improve the accuracy of predictions. Prospective multi-center validation is expected to provide high-level evidence for the subsequent clinical application.

HER2, Human epidermal growth factor receptor 2

HPN, HER2 Prediction Network

BC, breast cancer

ADCs, antibody-drug conjugates

US, ultrasound

TKIs, tyrosine kinase inhibitors

AUC, Area Under the Curve

ISH, immunohistochemistry

NAC, neoadjuvant chemotherapy

ResNet, Residual Network

SENet, Squeeze-and-Excitation Networks

GNN, Graph Neural Networks

AAU-net, Adaptive Attention U-net

CNNs, convolutional neural networks

PACS, Picture Archiving and Communication System

ROI, Region of Interest

PCA, Principal Component Analysis

ACC, accuracy

SPE, specificity

FNR, false negative rate

ROC, receiver operating characteristic curves

LVI, lymphovascular invasion

EGFR, epidermal growth factor receptor

Ki-67, Tumor proliferation marker-67

Ethics approval and consent to participate

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Ethics Committee of Peking Union Medical College Hospital(K24C1567). Written informed consents from patients were not required owing to the retrospective study design.

Consent for publication

Not Applicable.

Availability of data and materials

The datasets generated and analyzed during the current study are not publicly available due to the privacy of patients but are available from the corresponding author on reasonable request.

Competing interests

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding

This article was funded by National High Level Hospital Clinical Research Funding (No. 2022- PUMCH-B-038).

Authors' contributions

YL, ZH and MW contributed to the data collection. Construction of multimodal model and data analysis was performed by JY and YL. The first draft of the manuscript was written by YL, ZH and CW and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Bray F, Laversanne M, Sung H, Ferlay J, Siegel RL, Soerjomataram I, et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA A Cancer J Clinicians. 2024 Apr 4;caac.21834.
Schlam I, Swain SM. HER2-positive breast cancer and tyrosine kinase inhibitors: the time is now. npj Breast Cancer. 2021 May 20;7(1):56.
Wolff AC, Hammond MEH, Allison KH, Harvey BE, Mangu PB, Bartlett MS, et al. Human Epidermal Growth Factor Receptor 2 Testing in Breast Cancer: American Society of Clinical Oncology/College of American Pathologists Clinical Practice Guideline Focused Update. JOURNAL OF CLINICAL ONCOLOGY. 2013;
Perez EA, Romond EH, Suman VJ, Jeong JH, Sledge G, Geyer CE, et al. Trastuzumab Plus Adjuvant Chemotherapy for Human Epidermal Growth Factor Receptor 2–Positive Breast Cancer: Planned Joint Analysis of Overall Survival From NSABP B-31 and NCCTG N9831. JCO. 2014 Nov 20;32(33):3744–52.
Freedman RA, Gelman RS, Wefel JS, Melisko ME, Hess KR, Connolly RM, et al. Translational Breast Cancer Research Consortium (TBCRC) 022: A Phase II Trial of Neratinib for Patients With Human Epidermal Growth Factor Receptor 2–Positive Breast Cancer and Brain Metastases. JCO. 2016 Mar 20;34(9):945–52.
Swain SM, Miles D, Kim SB, Im YH, Im SA, Semiglazov V, et al. Pertuzumab, trastuzumab, and docetaxel for HER2-positive metastatic breast cancer (CLEOPATRA): end-of-study results from a double-blind, randomised, placebo-controlled, phase 3 study. The Lancet Oncology. 2020 Apr;21(4):519–30.
Guven DC, Kaya MB, Fedai B, Ozden M, Yildirim HC, Kosemehmetoglu K, et al. HER2-low breast cancer could be associated with an increased risk of brain metastasis. Int J Clin Oncol. 2022 Feb;27(2):332–9.
Hurvitz SA, Hegg R, Chung WP, Im SA, Jacot W, Ganju V, et al. Trastuzumab deruxtecan versus trastuzumab emtansine in patients with HER2-positive metastatic breast cancer: updated results from DESTINY-Breast03, a randomised, open-label, phase 3 trial. The Lancet. 2023 Jan;401(10371):105–17.
Modi S, Jacot W, Yamashita T, Sohn J, Vidal M, Tokunaga E, et al. Trastuzumab Deruxtecan in Previously Treated HER2-Low Advanced Breast Cancer. N Engl J Med. 2022 Jul 7;387(1):9–20.
Fernandez AI, Liu M, Bellizzi A, Brock J, Fadare O, Hanley K, et al. Examination of Low ERBB2 Protein Expression in Breast Cancer Tissue. JAMA Oncol. 2022 Apr 1;8(4):607.
Niikura N, Tomotaki A, Miyata H, Iwamoto T, Kawai M, Anan K, et al. Changes in tumor expression of HER2 and hormone receptors status after neoadjuvant chemotherapy in 21 755 patients from the Japanese breast cancer registry. Annals of Oncology. 2016 Mar;27(3):480–7.
Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, et al. A survey on deep learning in medical image analysis. Medical Image Analysis. 2017 Dec;42:60–88.
Lu W, Toss M, Dawood M, Rakha E, Rajpoot N, Minhas F. SlideGraph + : Whole slide image level graphs to predict HER2 status in breast cancer. Medical Image Analysis. 2022 Aug;80:102486.
Ramtohul T, Djerroudi L, Lissavalid E, Nhy C, Redon L, Ikni L, et al. Multiparametric MRI and Radiomics for the Prediction of HER2-Zero, -Low, and -Positive Breast Cancers. Radiology. 2023 Aug 1;308(2):e222646.
Chen G, Li L, Dai Y, Zhang J, Yap MH. AAU-Net: An Adaptive Attention U-Net for Breast Lesions Segmentation in Ultrasound Images. IEEE Trans Med Imaging. 2023 May;42(5):1289–300.
Tekpli X, Lien T, Røssevold AH, Nebdal D, Borgen E, Ohnstad HO, et al. An independent poor-prognosis subtype of breast cancer defined by a distinct tumor immune microenvironment. Nat Commun. 2019 Dec 3;10(1):5499.
Li C, Yin J. Radiomics Nomogram Based on Radiomics Score from Multiregional Diffusion-Weighted MRI and Clinical Factors for Evaluating HER-2 2+ Status of Breast Cancer. Diagnostics. 2021 Aug 18;11(8):1491.
Zheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, et al. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun. 2020 Mar 6;11(1):1236.
Jiang M, Li CL, Luo XM, Chuan ZR, Lv WZ, Li X, et al. Ultrasound-based deep learning radiomics in the assessment of pathological complete response to neoadjuvant chemotherapy in locally advanced breast cancer. European Journal of Cancer. 2021 Apr;147:95–105.
Dalmaz O, Yurt M, Cukur T. ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis. IEEE Trans Med Imaging. 2022 Oct;41(10):2598–614.
Giuliano AE, Connolly JL, Edge SB, Mittendorf EA, Rugo HS, Solin LJ, et al. Breast Cancer—Major changes in the American Joint Committee on Cancer eighth edition cancer staging manual. CA A Cancer J Clinicians. 2017 Jul 8;67(4):290–303.
Spak DA, Plaxco JS, Santiago L, Dryden MJ, Dogan BE. BI-RADS ® fifth edition: A summary of changes. Diagnostic and Interventional Imaging. 2017 Mar;98(3):179–90.
Xu Z, Yang Q, Li M, Gu J, Du C, Chen Y, et al. Predicting HER2 Status in Breast Cancer on Ultrasound Images Using Deep Learning Method. Front Oncol. 2022 Feb 16;12:829041.
Quan MY, Huang YX, Wang CY, Zhang Q, Chang C, Zhou SC. Deep learning radiomics model based on breast ultrasound video to predict HER2 expression status. Front Endocrinol. 2023 Apr 18;14:1144812.
Gerlinger M, Endesfelder D, Stewart A, Tarpey P, McDonald NQ, Santos CR, et al. Intratumor Heterogeneity and Branched Evolution Revealed by Multiregion Sequencing. n engl j med. 2012;
Braman N, Prasanna P, Whitney J, Singh S, Beig N, Etesami M, et al. Association of Peritumoral Radiomics With Tumor Biology and Pathologic Response to Preoperative Targeted Therapy for HER2 (ERBB2) –Positive Breast Cancer. JAMA Netw Open. 2019 Apr 19;2(4):e192561.
Li C, Song L, Yin J. Intratumoral and Peritumoral Radiomics Based on Functional Parametric Maps from Breast DCE‐MRI for Prediction of HER ‐2 and Ki‐67 Status. Magnetic Resonance Imaging. 2021 Sep;54(3):703–14.
Sun Q, Lin X, Zhao Y, Li L, Yan K, Liang D, et al. Deep Learning vs. Radiomics for Predicting Axillary Lymph Node Metastasis of Breast Cancer Using Ultrasound Images: Don’t Forget the Peritumoral Region. Front Oncol. 2020 Jan 31;10:53.
Cai Z, Vasconcelos N. Cascade R-CNN: High Quality Object Detection and Instance Segmentation. IEEE Trans Pattern Anal Mach Intell. 2021 May 1;43(5):1483–98.
Gao C, Cheng J, Yang Z, Chen Y, Zhu M. SCA-Former: transformer-like network based on stream-cross attention for medical image segmentation. Phys Med Biol. 2023 Dec 21;68(24):245008.
Tong T, Li D, Gu J, Chen G, Bai G, Yang X, et al. Dual-Input Transformer: An End-to-End Model for Preoperative Assessment of Pathological Complete Response to Neoadjuvant Chemotherapy in Breast Cancer Ultrasonography. IEEE J Biomed Health Inform. 2023 Jan;27(1):251–62.
Subaar C, Addai FT, Addison ECK, Christos O, Adom J, Owusu-Mensah M, et al. Investigating the detection of breast cancer with deep transfer learning using ResNet18 and ResNet34. Biomed Phys Eng Express. 2024 May 1;10(3):035029.
Pesapane F, Trentin C, Ferrari F, Signorelli G, Tantrige P, Montesano M, et al. Deep learning performance for detection and classification of microcalcifications on mammography. Eur Radiol Exp. 2023 Nov 7;7(1):69.
Chen M, Kong C, Lin G, Chen W, Guo X, Chen Y, et al. Development and validation of convolutional neural network-based model to predict the risk of sentinel or non-sentinel lymph node metastasis in patients with breast cancer: a machine learning study. eClinicalMedicine. 2023 Sep;63:102176.
Kazeminia S, Baur C, Kuijper A, Van Ginneken B, Navab N, Albarqouni S, et al. GANs for medical image analysis. Artificial Intelligence in Medicine. 2020 Sep;109:101938.
Yi Z, Chen Z, Cai H, Mao W, Gong M, Zhang H. BSD-GAN: Branched Generative Adversarial Network for Scale-Disentangled Representation Learning and Image Synthesis. IEEE Trans on Image Process. 2020;29:9073–83.

No competing interests reported.

HPN: A Multimodal Neural Network Model for Non-invasive HER2 Status Assessment in Breast Cancer Patients

Status:

Version 1

Abstract

Figures

Introduction

Methods

Patients

Study Design

Clinicopathological Parameters Dataset

Ultrasound Image Dataset

Multimodal Modelling of Clinicopathological and Ultrasonic Image Data

Statistical Analyses

Results

Clinicopathological Characteristics of the Patients

Prediction Performance of the Multimodal Model

Comparison Between Multimodal and Unimodal Models

Discussion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1