OMG-Net: A Deep Learning Framework Deploying Segment Anything to Detect Pan-Cancer Mitotic Figures from Haematoxylin and Eosin-Stained Slides

doi:10.21203/rs.3.rs-4848443/v1

Download PDF

Article

OMG-Net: A Deep Learning Framework Deploying Segment Anything to Detect Pan-Cancer Mitotic Figures from Haematoxylin and Eosin-Stained Slides

https://doi.org/10.21203/rs.3.rs-4848443/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Mitotic activity is an important feature for grading several cancer types. Counting mitotic figures (MFs) is a time-consuming, laborious task prone to inter-observer variation. Inaccurate recognition of MFs can lead to incorrect grading and hence potential suboptimal treatment. In this study, we propose an artificial intelligence (AI)-aided approach to detect MFs in digitised haematoxylin and eosin-stained whole slide images (WSIs). Advances in this area are hampered by the limited number and types of cancer datasets of MFs. Here we establish the largest pan-cancer dataset of mitotic figures by combining an in-house dataset of soft tissue tumours (STMF) with five open-source mitotic datasets comprising multiple human cancers and canine specimens (ICPR, TUPAC, CCMCT, CMC and MIDOG++). This new dataset identifies 74,620 MFs and 105,538 mitotic-like figures. We then employed a two-stage framework (the Optimised Mitoses Generator Network (OMG-Net) to classify MFs. The framework first deploys the Segment Anything Model (SAM) to automate the contouring of MFs and surrounding objects. An adapted ResNet18 is subsequently trained to classify MFs. OMG-Net reaches an F1-score of 0.84 on pan-cancer MF detection (breast carcinoma, neuroendocrine tumour and melanoma), largely outperforming the previous state-of-the-art MIDOG++ benchmark model on its hold-out testing set (e.g. +16% F1-score on breast cancer detection, p<0.001) thereby providing superior accuracy in detecting MFs on various types of tumours obtained with different scanners.

Health sciences/Diseases/Cancer

Physical sciences/Mathematics and computing/Computer science

Physical sciences/Mathematics and computing/Scientific data

Mitotic activity is a crucial indicator of cellular proliferation and plays a pivotal role in cancer diagnosis and guiding the clinical management [1]. Counting mitotic figures (MFs) from haematoxylin and eosin (H&E)-stained whole slide images (WSIs) is a fundamental task in pathology, required for the grading of some tumours. By convention, in clinical practice, mitotic counts are performed in the 10 most mitotically active high-power microscopic fields (HPFs) within a tumour [2]. As this is a time-consuming task, and subject to significant inter-observer variability [3] [4] [5], there has been considerable interest and effort in the development of automated MF detection models, e.g. ICPR [6] [7] and TUPAC [8] initiated the development of breast cancer MF datasets. Initially, mitotic detection models focused on learning handcrafted features [9] [10] [11], but recently transitioned to deep-learning-based methods that show promise [12] [13] [14] [15]. However, MF detection remains a challenging task [16], due to the different appearance of MF in the four phases of mitosis, the range of features exhibited by abnormal MFs, as well as structures that mimic MFs (mitotic-like figures, MLFs). The above challenges are compounded by the histological heterogeneity in normal tissues and tumour types, staining variation between labs and differences in digital scanners used to generate WSIs.

To improve the detection of MF, the MItosis DOmain Generalization (MIDOG) [17] [18] published an updated version of their multi-domain dataset, MIDOG++ [16]. This contains 503 annotated images across seven different cancer types, representing the largest currently available published dataset of MFs. The data utilised in the MIDOG studies contains the HPFs manually selected by pathologists to mimic clinical practice. However, the pathologist-led decisions may not be reproducible because of the recognised inter-observer variation [19] [20], and discrepancies can be caused by the selection of areas with the densest mitotic activity [21]. In contrast, the CMC [22] and CCMCT [23] datasets used AI-assisted annotations to generate large-scale WSI datasets for MFs using canine cancers. The former used 21 WSIs of canine mammary carcinomas whereas the CCMCT dataset included 32 WSIs of canine mast cell tumours. These studies demonstrated that annotating MFs on a WSI improves the robustness of classifiers by removing the HPF selection bias and leads to a significantly higher number of detected mitoses, helping to refine further training [23].

One preferred approach to take forward this field of MF detection would have been to increase the size of the existing datasets incorporating multiple scanner types, staining differences across multiple sites, and tumour types. However, the lack of standardisation in the annotation protocol across existing various datasets limit their integration. For example, in the ICPR, each pixel within MFs was labelled, whereas the TUPAC only encircled MFs. MIDOG++, CCMCT, and CMC utilised bounding boxes to denote the targets. We therefore took the approach to standardise the annotations by contouring nuclei of MFs.

Historically, targets in cellular object detection tasks are denoted using bounding boxes. However, several studies have reported that incorporating a target’s mask facilitates model training and improves the overall classification performance. For instance, the Mask-RCNN outperformed the Faster-RCNN in a variety of object detection tasks [24], including the MF detection [13]. The advantages of integrating nuclei contours for detection include enhancing the definition of nuclei boundaries, mitigating the morphological variability of the MFs [25] and reducing the impact of tumour histological heterogeneity. Given the constraints of a small dataset and the significant variability between mitotic cells, introducing a recognisable mitotic feature into the model aids in stabilising the training process and leads to a faster convergence.

The aim of this study was to improve the detection of MF across multiple tumour types. First, we established a large uniform database of pan-cancer MFs by deploying the Segment Anything Model (SAM) [26], a foundation object detection model, in five open-source datasets (ICPR, TUPAC, CCMCT, CMC, MIDOG++) using a single nuclei mask format. Manual revision of the masks was performed to maximise database quality. Then, we contributed an in-house dataset of human soft tissue tumours (STT) MFs (N=8,400) (Soft-Tissue Mitotic Figures, STMF). Although STT represents a rare tumour group, they comprise over 100 subtypes exhibiting a wide variety of histological appearances and mimic other tumours including common cancers such as melanoma, carcinoma and lymphoma. STT harbours a variable number of MFs and aids in reaching a diagnosis and predicting disease behaviour [27]. As of now, no publicly accessible data have been published for MFs in STT. The STMF was initiated by staining WSIs with an anti-phosphorylated histone H3 (pHH3) antibody to target MFs which was expanded and improved by AI-assisted annotations made by pathologists.

The second objective was to develop an improved MF detection framework, which we named Optimised Mitoses Generator Network (OMG-Net). The structure of OMG-Net is outlined in Figure 2. By integrating nuclei masks into the pre-trained classifier via a first-layer addition, we allow the model to focus on the morphological features of MFs. We demonstrate that OMG-Net is both more sensitive and specific at detecting objects, including MFs, throughout the input WSI, compared to previous models.

Developing a large-scale MF dataset

We established a large in-house dataset for MFs in STT and merged it with five open-source datasets for MFs from human and canine specimens (Table 1). The final dataset contains 74,620 MFs and 105,538 MLFs from 712 different images or WSIs with the SAM-delineated masks for nuclei. Masks of human MFs were reviewed and modified to ensure the quality of nuclei contours. Additionally, the dataset included a large number of SAM-segmented objects, comprising tumour cells, immune cells, red blood cells, artefacts and any objects at the cell scale, collected during the data curation.

Table 1

Number of different types of objects in the integrated dataset.
Dataset	Tumour Types	Number of Images	Number of MFs	Number of MLFs	Non-MF objects
ICPR	Breast carcinoma	100	654	0	10,696
TUPAC	Breast carcinoma	73	1,999	10,483	233,992
MIDOG++	Breast carcinoma Lung carcinoma* Lymphosarcoma* Neuroendocrine tumour* Mast cell tumour* Melanoma Soft tissue sarcoma*	392	9,470	11,433	559,827
CMC	Breast carcinoma*	21^†	13,907	36,379	2,428,456
CCMCT	Mast cell tumour *	32^†	40,190	42,208	1,082,776
STMF	Soft tissue tumour	103^[1]† + 226^[2]	8,400	5,035	395,670
Total		938	74,620	105,538	4,701,417

*Canine Specimens. ^†WSIs rather than selected regions.

[1] pHH3-immunohistochemistry was used for identifying MFs.

[2] Active learning was used for annotating MFs.

Performance of MF detection in various human tumours

We benchmarked our OMG-Net against the current state-of-the-art MIDOG + + dataset/model containing three types of human tumours (breast carcinoma, neuroendocrine tumour, and melanoma). Table 2 hows the mean precision, recall and F1 scores with the standard deviation of the proposed framework trained five times using different random seeds, along with the F1 score obtained by ensemble voting and the F1 scores quoted in the MIDOG + + paper.

Table 2

Precision, recall and F1 scores in MIDOG + + testing set of OMG-Net against the model presented by MIDOG++
Tumour Types	Precision	Recall	F1	Ensemble F1	F1(MIDOG++)
Breast carcinoma	0.82 ± 0.02	0.88 ± 0.02	0.85 ± 0.02	0.87	0.71 ± 0.02
Neuroendocrine tumour	0.64 ± 0.02	0.65 ± 0.03	0.64 ± 0.02	0.67	0.59 ± 0.02
Melanoma	0.83 ± 0.02	0.84 ± 0.03	0.83 ± 0.01	0.85	0.81 ± 0.02

The F1 score comparison is also displayed in Fig. 3a. The MF detection scores of OMG-Net are significantly higher (p < 0.001) in all three types of human tumours within the testing set of MIDOG++. Figure 3b shows the benefit of combining multi-centre data, as the increased number of MFs for training correlated with an increase in the holdout F1-score.

Reviewing the SAM masks enhanced the detection performance

As shown in Fig. 4a, the appearance of MFs is highly diverse, exemplified by atypical MFs. The segmented mask may not fully cover the MFs or may contain background noise. To refine the training process, we reviewed the masks in the human subset of MIDOG++ (4435 in breast carcinoma, 2075 in melanoma and 2400 in neuroendocrine tumour), and adjusted the SAM prompt when required. The impact of this manual curation was assessed by comparing the F1 scores of the models only with RGB images (RGB Classifier), the score of the model with zero-shot SAM mask input (RGB-M0 Classifier) as well as the score of the model with reviewed and refined masks (RGB-M1 Classifier), with results shown in Fig. 4b.

Compared to the model without masks (RGB), the RGB-M0 model yielded higher F1 scores for detecting MFs from breast carcinoma (p = 0.011) and melanoma (p = 0.001) but not for neuroendocrine tumours. Upon further analysis, we noted that the fraction of masks requiring a second adjustment was higher in neuroendocrine tumours (16%), compared to breast carcinoma (8%) and melanoma (5%). As predicted, the RGB-M1 Classifier showed the best performance and significantly outperformed the RGB Classifier for breast carcinoma (p < 0.001), melanoma (p < 0.001) and neuroendocrine tumours (p = 0.021). We conclude that the low-quality masks, which may include surrounding backgrounds or exclude part of the nuclei (Fig. 4b), can impact the performance of the RGB-M0. Further examples of failed prompts are shown in Supplementary Fig. 1.

Canine mitotic figures help to train the detection of human mitotic figures

The merged and uniform dataset contains a significant proportion of canine MFs with examples from both human and canine WSI displayed in Fig. 5a. The inclusion of the canine data significantly improved the detection of MFs in breast carcinoma (p = 0.007) and neuroendocrine tumours (p = 0.015) and the F1 score in melanoma was also marginally increased (p = 0.080) (Fig. 5b).

Including mitotic-like figures and non-mitotic objects is key to improving model precision

Besides MFs, MLFs were also labelled in the original dataset. MLFs represent morphological structures that resemble MFs including pyknotic nuclei, apoptotic bodies, and neutrophil polymorph amongst others, often misclassified as MFs. An example is displayed in Fig. 6a. Apart from MLFs and tumour cells, the SAM-curated dataset contains other cells, including immune cells, red blood cells and any objects at the cell level such as artefacts, segmented by the SAM during data curation to present the classifier with a heterogeneous set of data. These were used in the training step to augment the original dataset and provide the model with a diverse representation of segmented objects. Figure 6b shows that the model including non-MF objects (SAM-AUG) has significantly higher precision for all three types of tumours (p = 0.008) compared to the model trained only with MFs and MLFs (Original). As expected, the recall remains unchanged, and the overall F1 scores are improved (p = 0.007).

Nuclei contours represent a key feature for improving mitotic-figure detection

In this study, OMG-Net showed significantly improved MF detection performance in all three types of tumours compared with the testing set of the current state-of-the-art MIDOG++. This improvement was achieved both by using a larger, multi-source, SAM-enhanced dataset of MFs (Fig. 1) and by integrating within the network the mask of the MFs’ nuclei (Fig. 4b).

The accuracy of our models varied considerably across different tumour types, with neuroendocrine tumours exhibiting significantly lower performance, which was consistent with the results of the MIDOG + + algorithm. In parallel, we observed a higher proportion of low-quality masks in neuroendocrine tumours (16%) compared to breast carcinoma (8%) and melanoma (5%), suggesting that the quality of the training data may have contributed to the disparities in model performance across these cancer types. Even then, manual curation of the masks helped improve significantly the model detection performance.

To decide which foundational model to select as a nuclei detector, we evaluated published fine-tuned variants of SAM against the overall mask quality for cells in histology images. Specifically, we tested MedSAM [28], which was fine-tuned on multiple medical image modalities, and CellSAM [29], which was fine-tuned on microscopy images. By inspecting the number of cells detected and the layout of the masks produced by both algorithms, we concluded that the quality of masks was degraded in both models and included a higher proportion of omitted cells (Supplementary Fig. 2). We attribute the reduced performance of the fine-tuned SAM models to different architectures used compared to the original SAM model. OMG-Net uses the highest capacity SAM variant (ViT-H), whereas MedSAM and CellSAM are fine-tuned on the lighter ViT-B, which may lead to reduced performance due to limited model capacity. Based on this analysis, we elected to keep the original SAM as the cell detector in our study. Future work will include refining the SAM object-proposal method for H&E-stained specific cell types

Decoupling segmentation and classification helps improve the detection performance

Object detection models such as Faster R-CNN [30], RetinaNet [31] and YOLO [32] have been widely used for MF detection [12] [23]. These models integrate in a single model an object proposal network with a primary classifier. However, these models suffer from the imbalanced loss problem, as the cell segmentation and classification loss have inherently unequal magnitudes. The gradient updates that occur during backpropagation can be dominated by the loss function with the larger norm [33], leading to suboptimal training and convergence issues. This becomes even more prominent when dealing with small datasets or complex objects, as the imbalance in the loss functions' impact can significantly hinder the model's ability to learn effectively from the limited available data [34]. The use of integrated object detection models in histopathological studies has been shown to generate false positive results due to the complex and variable nature of cell morphology.

More recently, it has been demonstrated [35] [36] that integrating a secondary classifier, trained on MFs and other objects such as MLFs, to review and reject false positive cases improves a framework’s precision. This approach limits the imbalanced loss problem, as the segmentation loss is excluded in training the additional classifiers. However, these methods add unnecessary complexity to the network since two classifiers must be trained.

To mitigate the imbalanced loss problem, we elected to separate entirely the object detection and classification steps. This offers an innovative approach that differs from those previously published. Instead of training an object detection model for generating objects that are highly likely to be MFs, all the objects at cell scale are segmented by SAM from the ROIs and classified, improving the sensitivity of our model. Other objects, including immune cells, cells not in mitosis, and artefacts generated during the data preparation stage, can also be used to train the classifier, improving its capability to reject false positives.

Large-scale MF datasets provide a resource for the development of pan-cancer models

Large-scale datasets are crucial for developing AI models capable of detecting MFs effectively in a variety of cancer types and overcoming the challenges posed by the heterogeneity of staining and scanning protocols. Here, we propose a workflow for creating a reliable MF dataset:

H&E destaining and employing immunohistochemistry for enhanced detection: efficient generation of a large-scale image dataset with accurate labels by detecting a substantial number of MFs on WSIs.

Continuous Data Curation: improve data quality by employing Segment Anything (SAM) to delineate precisely mitotic figure (MF) nuclei, followed by meticulous manual refinement of the generated contours.

Active learning: iteratively train and refine the model using a pathologist-in-the-loop approach, enabling efficient review of detected mitotic figures (MFs), and incorporating Mitotic-Like Figures (MLFs) and non-mitotic objects into the database for enhanced model performance

These steps are required as it is not feasible for pathologists to annotate MFs in the numbers and the precision required by AI models, thereby affecting the diversity and size of the dataset and, consequently, the detection accuracy of the trained model. Nevertheless, each of these steps encounters limitations.

Performing immunohistochemistry following the destaining procedure of H&E-stained sections allows for the rapid and largely specific detection of MFs (specificity > 99%) [35]. Still, it is not a perfect process as cells in the G2 phase of the cell cycle can exhibit weak immunoreactivity [37] as well as being prone to false-negative immunoreactivity due to the age of the slide and fixation method [36]. This restaining procedure also does not detect MLFs, which is crucial to enhance the model specificity.

Active learning can help identify MLFs but a consensus view of MF/MLF cannot always be reached by pathologists. This study highlighted the acknowledged problem of interobserver variation of MF by pathologists[4][5] which is compounded when interpreting MFs on digitised slides as it is not possible to adjust the focus plane on cells of interest. During our revision process, a notable proportion (13.8%) of AI-detected cells were categorised as "equivocal" (Supplementary Fig. 3). A secondary review of these images performed by at least two experienced pathologists resolved some of these images but differences in opinion remained in 9.5% of AI-detected MFs.

Finally, despite the limitations discussed above, the integration of immunohistochemistry for MF detection following the destaining of H&E sections, data curation, active learning, and consensus-based review by experienced pathologists enabled us to mitigate the challenges in creating a large-scale database and developing an improved, pan-cancer MF detection model.

We have established a large-scale MF dataset by integrating five open-source datasets acquired from multiple centres including an in-house dataset of STT. Using the curated dataset, we employed a novel two-step framework, OMG-Net, where SAM served as the object detector followed by an adapted ResNet18 as the MF classifier. This approach improved the accuracy of MF detection from various human tumours including breast carcinoma, neuroendocrine tumours and melanoma compared to existing state-of-the-art models. Future steps includes a head-to-head prospective assessment of this model with pathologists’ scores for MFs before introduction into safe clinical practice.

Dataset

Ethical Approvals

The data involved in the STMF dataset are collected in the Royal National Orthopaedic Hospital (RNOH) NHS Trust under the Health Research Authority (HRA) and Health and Care Research Wales (HCRW) Approval. Integrated Research Application System (IRAS) project ID: 328987. Protocol number: EDGE 161548. Research Ethics Committee (REC) reference: 23/NI/0166. Informed consent was obtained from all human participants.

Open-source datasets

We integrated five open-source datasets (ICPR, TUPAC, CCMCT, CMC, MIDOG++), comprising 68,687 MFs from eight different scanners and eight types of human and canine tumours. The types of tumours studied and scanners are listed in Supplementary Table 1. All the images were scanned in 40× magnification with a pixel size of approximately 0.25 µm.

In-house dataset

We describe a workflow for utilising an anti-phosphorylated histone H3 (pHH3) antibody to specifically detect MFs and expand the dataset by active learning (Fig. 1). The number of MFs in each diagnosis of soft tissue tumours is listed in Supplementary Table 2.

pHH3-assisted MF detection: the pHH3 antibody employed specifically detects the core protein histone H3 only when phosphorylated at serine 10 (Ser10) or serine 28 (Ser28), thereby identifying mitotic cells within a tissue sample [37]. We selected 94 archived slides and tissue blocks from soft tissue tumours and prepared fresh H&E tissue sections which were then scanned for generating our dataset. These H&E-stained tissue sections were then de-stained after which immunohistochemistry was performed using a rabbit monoclonal (RM) hybridoma Ser10 pHH3 [BC37] [38] and then counterstained with eosin. The masks of the MFs were extracted from pHH3-immunolabelled WSIs by setting thresholds for the RGB values and transferred to the same location on the matching H&E-stained WSIs. Registration between the pHH3-immunolabelled and H&E-stained WSIs was achieved by random sample consensus (RANSAC) [39] on both a WSI-level and patch-level. The contours and positions of 7,952 MFs (STMF-V0), were identified and validated by pathologists reviewing the H&E and immunolabelled sections. However, not all mitoses were identified by pHH3-labelling indicating that the antibody was not entirely sensitive [40].
Active Learning: Although the identification of cells in mitosis by pHH3 can establish a dataset with a large number of MFs, it cannot identify MLFs, and models trained only with IHC suffer from limited precision. Active learning is required to augment the dataset with MLFs.

During the active learning process, pathologists corrected the image labels given by a machine learning model and fed them back to re-train the initial model, so that the model performance for the target task can be continuously improved during the iteration of machine-generating and human-labelling.

To expand the STMF-V0 dataset, we trained an initial Mask-RCNN model on it and applied the model to new WSIs for detecting MFs. The AI-detected MFs were randomly assigned to six pathologists to be independently labelled as ‘MF’, ‘not MF’ if the pathologist could confidently make a decision, or ‘uncertain’ when the morphological features were equivocal. These equivocal MF were reviewed to two senior pathologists. Other structures such as apoptotic bodies were also labelled to create the final dataset, STMF, with 8400 MFs and 5035 MLFs.

Data Curation

The MFs were annotated using bounding boxes in the CCMCT, CMC and MIDOG + + datasets. However, the size of the boxes varies due to the lack of standard annotation criteria. We hypothesised that the contours of nuclei could provide extra information for classifying MFs, as the model would be guided to focus on the most representative pixels of the nuclei rather than the surrounding environment.

We use the bounding boxes provided in the CCMCT, CMC and MIDOG + + datasets as prompts to generate the masks using SAM. To ensure the quality of the automatically generated masks, we inspected individual masks of the MFs from three types of human tumours and canine soft tissue sarcoma in MIDOG++. The percentage of masks amended following review is 8%, 5% and 16% out of 4435 masks in breast carcinoma, 2075 in melanoma and 2400 in neuroendocrine tumour, respectively. In total, only 8% of the masks required a second inference of SAM using adjusted bounding boxes. Since the cells can be distorted during the de-staining and pHH3 labelling process, we also applied the SAM to the STMF using the outside boxes of the pHH3-immunolabelled masks as prompts. The numbers of MFs and MLFs from human and canine samples are shown in Fig. 1. Quality assurance was done for masks of all the human samples, whereas the generation of masks in canine sections was fully automated.

OMG-Net: A Two-Stage Detection Framework

The proposed framework consists of two steps:

The SAM was applied to patches of 1024 × 1024 pixels from the tumour regions. 64 points were evenly sampled along each dimension, totalling 4,096 points used as prompts per patch. The quality of the masks was predicted by two factors, an AI-predicted Intersection over Union (AI-IoU) and a stability score. The AI-predicted IoU comes from an adjacent multi-layer perceptron in the mask decoder section of SAM. The stability score is the IoU between the binary masks obtained by thresholding the predicted mask logits at high and low values. Only the objects with AI-IoU scores and stability scores higher than 0.8 and areas between 2.25 µm² and 225 µm² were kept after filtering. The filtered masks were then ranked by their AI-IoU scores. Non-maximum suppression (NMS) [41]was used to remove duplicated masks.
The objects generated were then classified by the second model, a ResNet18 pre-trained on ImageNet, as MFs or other objects. In addition to taking a 3-channel RGB image, the mask of the object was encoded by a convolutional layer and summed to the first convolutional layer of the ResNet18. Via this process, we retained the ability to use pre-trained models while providing extra mask information to the model.

Model Development and Testing

The framework was implemented using Pytorch and Pytorch Lightning and was trained using a single NVIDIA GeForce RTX 3090 for 30 epochs with a batch size of 8,000. The learning rate was set up at 0.001, optimised by the AdamW algorithm[42] and cosine annealing scheduler [43].

Training and Validation

We trained the ResNet18 to classify MFs while the SAM mask generator was not retrained. The SAM was applied to all patches in the dataset after data curation, and the other objects surrounding the targets were also segmented and included in the training and validation data. The binary classifier is trained on two classes: (1) MFs and (2) labelled MLFs and other cells or objects segmented by SAM. In each training process, 90% of the data was used for training the model, while the remaining 10% was used for validation. The training was repeated five times using different random seeds to get five models with different data split.

Data Augmentation

Colour and spatial augmentation were applied to the training data to reduce the impact of the staining variation and increase the robustness of the model. To achieve colour augmentation, RGB images are deconvolved into H&E stains using the stain vectors proposed by Ruifrok and Johnston [44]. The stain concentration perturbation scheme introduced by Tellez et al. [45] was used with a uniform sampling and $\:\sigma\:=0.14$ on the deconvolved H&E channels prior to reconstructing RGB images. Random horizontal flips ($\:p=0.4$) were also used.

Test set and performance metrics

We used the same testing set provided by MIDOG++, which contains 674 MFs from 56 sections of three types of human tumours. Precision, recall and F1 score were used to evaluate the performance of our mitotic detection framework. They were calculated by

$$\:Precision=\:\frac{{N}_{TP}}{{N}_{TP}+{N}_{FP}}$$

$$\:Recall=\:\frac{{N}_{TP}}{{N}_{TP}+{N}_{FN}}$$

$$\:F1=\:2\bullet\:\frac{Precision\bullet\:\:Recall\:}{Precision+\:Recall}$$

where $\:{N}_{TP}$, $\:{N}_{FP}$ and $\:{N}_{FN}$ represent the number of true positives, false positives and false negatives, respectively. Mann–Whitney U test [46] was used for comparing the unpaired scores of different models.

Acknowledgements

This project is supported by the UKRI Future Leaders Fellowship (MR/T040785/1), EPSRC Research Grant NR1 (EP/Y020030/1), the Radiation Research Unit at the Cancer Research UK City of London Centre Award (C7893/A28990), as well Sarcoma UK (Award SUK18.2021). AMF and KT are supported by the National Institute for Health Research, UCLH Biomedical Research Centre, and the CRUK Experimental Cancer Centre as well as the Royal National Orthopaedic Hospital R& D Department. The Royal National Orthopaedic Hospital, Stanmore, Middlesex HA7 4LP. SH was funded by the Children’s Cancer Foundation Basel (grant: C23-2021-21). TB is a PhD Clinical Fellow, funded by the Jean Shanks Foundation and the Pathological Society of Great Britain and Ireland. EK was funded by the Royal National Orthopaedic Hospital.

Author information

Authors and Affiliations

Department of Medical Physics and Biomedical Engineering, University College London, United Kingdom

Zhuoyan Shen, Mikael Simard, Douglas Brand, Gary Royle, Maria A. Hawkins & Charles-Antoine Collins Fekete

Department of Radiotherapy, University College London Hospitals NHS Foundation Trust, United Kingdom

Douglas Brand & Maria A. Hawkins

Research Department of Pathology, University College London Cancer Institute, United Kingdom

Vanghelita Andrei, Ali Al-Khader, Katherine Trevers, Thomas Butters, Simon Haefliger, Fernanda Amary, Roberto Tirabosco & Adrienne M. Flanagan

Cellular and Molecular Pathology, Royal National Orthopaedic Hospital NHS Foundation Trust, United Kingdom

Vanghelita Andrei, Ali Al-Khader, Fatine Oumlil, Katherine Trevers, Fernanda Amary, Roberto Tirabosco & Adrienne M. Flanagan,

Institute of Medical Genetics and Pathology, University Hospital Basel, University of Basel, Basel, CH, Switzerland

Simon Haefliger

Department of Neurology, Rutgers Biomedical and Health Sciences, Rutgers University, New Jersey, United States

Eleanna Kara

Department of Orthopaedics, The Robert Jones and Agnes Hunt Orthopaedic Hospital, United Kingdom

Paul Cool

School of Medicine, Keele University, United Kingdom

Paul Cool

Contributions

The study was designed by ZS, CACF and AMF. ZS, CACF and MS contributed to the code for the models. AMF, VA, AAK, SH, EK, and TB provided the WSIs and annotations. FO, KT oversaw slide scanning and anonymisation. ZS and CACF contributed to the data generation and curation. ZS wrote the manuscript of the paper. CACF, MS, DB, MH and AMF contributed to writing and improving the paper. PC contributed to the statistical analysis. All authors contributed to the critical revision of the paper. The authors acknowledge Rohan Tapabrata Chakraborty for fruitful discussions. CACF and AMF contributed equally to this work.

Corresponding author

Correspondence to Zhuoyan Shen.

Ethics declarations

Competing interests

The authors declare no competing interests.

Data availability

All MF images and their SAM-dilated contours are available without restriction via the figshare repository in accordance with the UKRI Common principles on research data.

Code availability

All code to reproduce the results, when coupled with the dataset available on the figshare repository, is provided free of use at https://github.com/cacof1/DigitalPathologyAI.

Williams, G.H. & Stoeber, K. The cell cycle and cancer. J. Pathol., 226, 352-364 (2012).
Cree, I.A., Tan, P.H., Travis, W.D. et al. Counting mitoses: SI(ze) matters!. Mod Pathol 34, 1651–1657 (2021).
Malon, C. et al. Mitotic figure recognition: agreement among pathologists and computerized detector. Anal Cell Pathol (Amst). 35(2), 97-100 (2012).
Veta, M., van Diest, P.J., Jiwa, M., Al-Janabi, S. & Pluim, J.P. Mitosis Counting in Breast Cancer: Object-Level Interobserver Agreement and Comparison to an Automatic Method. PLoS One. 11(8), e0161286; 10.1371/journal.pone.0161286 (2016).
Robbins, P. et al. Histological grading of breast carcinomas: a study of interobserver agreement. Hum Pathol. 26(8), 873-9 (1995).
Roux, L. et al. Mitosis detection in breast cancer histological images An ICPR 2012 contest. J Pathol Inform. 30, 4:8 (2013).
MITOS-ATYPIA-14-Dataset. https://mitos-atypia-14.grand-challenge.org/ (2014).
Veta, M. et al. Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge. Med Image Anal. 54, 111-121 (2019).
Irshad, H. Automated mitosis detection in histopathology using morphological and multi-channel statistics features. J Pathol Inform. 4, 10 (2013).
Tashk, A., Helfroush, M. S., Danyali, H. & Akbarzadeh, M. An automatic mitosis detection method for breast cancer histopathology slide images based on objective and pixel-wise textural features classification, The 5th Conference on Information and Knowledge Technology, Shiraz, Iran. 406-410 (2013).
Paul, A., Dey, A., Mukherjee, D.P., Sivaswamy, J. & Tourani, V. Regenerative Random Forest with Automatic Feature Selection to Detect Mitosis in Histopathological Breast Cancer Images. Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015. 94–102 (2015).
Mahmood, T., Arsalan, M., Owais, M., Lee, M.B. & Park, K.R. Artificial Intelligence-Based Mitosis Detection in Breast Cancer Histopathology Images Using Faster R-CNN and Deep CNNs. J Clin Med. 9(3), 749 (2020).
Sebai, M., Wang, X. & Wang, T. MaskMitosis: a deep learning framework for fully supervised, weakly supervised, and unsupervised mitosis detection in histopathology images. Med Biol Eng Comput. 58(7),1603-1623 (2020).
Li, C., Wang, X., Liu, W. & Latecki, L.J. DeepMitosis: Mitosis detection via deep detection, verification and segmentation networks. Med Image Anal. 45, 121-133 (2018).
Cai, D., Sun, X., Zhou, N., Han, X. & Yao, J.Efficient Mitosis Detection in Breast Cancer Histology Images by RCNN. 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy. 919-922 (2019).
Aubreville, M. et al. A comprehensive multi-domain dataset for mitotic figure detection. Sci Data. 10, 484 (2023).
Aubreville, M., Bertram, C. & Veta, M. MItosis DOmain Generalization Challenge. https://doi.org/10.5281/zenodo.4573978 (2021).
Aubreville, M., Bertram, C. & Breininger, K. MItosis DOmain Generalization Challenge 2022. https://doi.org/10.5281/zenodo.6362337 (2022).
Meyer, J. et al. Breast carcinoma malignancy grading by Bloom–Richardson system vs proliferation index: reproducibility of grade and advantages of proliferation index. Mod. Pathol. 18, 1067–1078 (2005).
Bertram, C. A. et al. Computerized Calculation of Mitotic Count Distribution in Canine Cutaneous Mast Cell Tumor Sections: Mitotic Count Is Area Dependent. Veterinary Pathology. 57(2), 214-226 (2020).
van Diest, P.J. et al. Reproducibility of mitosis counting in 2,469 breast cancer specimens: results from the Multicenter Morphometric Mammary Carcinoma Project. Hum Pathol. 23(6), 603-7 (1992).
Aubreville, M. et al. A completely annotated whole slide image dataset of canine breast cancer to aid human breast cancer research. Sci. Data. 7, 417 (2020).
Bertram, C.A. et al. A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci. Data. 6, 274 (2019).
He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. arXiv, 10.48550/arXiv.1703.06870 (2018).
Lou, W. et al. Structure Embedded Nucleus Classification for Histopathology Images. arXiv. 10.48550/arXiv.2302.11416 (2023).
Kirillov, A. et al. Segment Anything. arXiv. 10.48550/arXiv.2304.02643 (2023).
Coindre, J.M. Grading of soft tissue sarcomas: review and update. Arch Pathol Lab Med. 130(10), 1448-53 (2006).
Ma, J. et al. Segment anything in medical images. Nat. Commun. 15, 654 (2024).
Israel, U. et al. A Foundation Model for Cell Segmentation. arXiv. 10.48550/arXiv.2311.11004 (2023).
Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv. 10.48550/arXiv.1506.01497 (2016).
Lin, T., Goyal, P., Girshick, R., He, K. & Dollár, P. Focal Loss for Dense Object Detection. arXiv. 10.48550/arXiv.1708.02002 (2018).
Redmon, J., Divvala, S., Girshick R., & Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. arXiv. 10.48550/arXiv.1506.02640 (2016).
Chen, Z., Badrinarayanan, V., Lee, C. & Rabinovich, A. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. arXiv. arXiv.1711.02257 (2018).
Argyriou, A., Evgeniou, T. & Pontil, M. Multi-Task Feature Learning. Advances in Neural Information Processing Systems 19 (NIPS 2006). (2006).
Çayır, S. et al. MITNET: a novel dataset and a two-stage deep learning approach for mitosis recognition in whole slide images of breast cancer tissue. Neural Comput. & Applic. 34, 17837–17851 (2022).
Sohail, A. et al. A multi-phase deep CNN based mitosis detection framework for breast cancer histopathological images. Sci. Rep. 11, 6215 (2021).
Elmaci, İ., Altinoz, M.A., Sari, R. & Bolukbasi, F.H. Phosphorylated Histone H3 (PHH3) as a Novel Cell Proliferation Marker and Prognosticator for Meningeal Tumors: A Short Review. Appl Immunohistochem Mol Morphol. 26(9), 627-631 (2018).
Tacha, D. E. A New Rabbit Monoclonal phospho-histone H3 (pHH3) Hybridoma: An Immunohistochemical Comparison Study with a Rabbit Polyclonal pHH3. https://api.semanticscholar.org/CorpusID:54518201 (2015).
Fischler, M. A. & Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography, Readings in Computer Vision. 726-740 (1981).
Ribalta, T., McCutcheon, I.E., Aldape, K.D., Bruner, J.M. & Fuller, G.N. The mitosis-specific antibody anti-phosphohistone-H3 (PHH3) facilitates rapid reliable grading of meningiomas according to WHO 2000 criteria. Am J Surg Pathol. 28(11),1532-6 (2004).
Hosang, J., Benenson, R. & Schiele, B. Learning non-maximum suppression, arXiv. 10.48550/arXiv.1705.02950 (2017).
Loshchilov, I.& Hutter, F. Decoupled Weight Decay Regularization. arXiv. 10.48550/arXiv.1711.05101 (2019).
Loshchilov, I.& Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv. 10.48550/arXiv.1608.03983 (2017).
Ruifrok AC, Johnston DA. Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol. 23(4), 291-9 (2001).
Tellez, D. et al. Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks. IEEE Trans. Med. Imaging. 37(9), 2126-2136 (2018).
Nachar, N. The Mann-Whitney U: A Test for Assessing Whether Two Independent Samples Come from the Same Distribution. Tutorials in Quantitative Methods for Psychology. 4 (2008).
Hendzel, M.J. et al. Mitosis-specific phosphorylation of histone H3 initiates primarily within pericentromeric heterochromatin during G2 and spreads in an ordered fashion coincident with mitotic chromosome condensation. Chromosoma. 106(6), 348-60 (1997).
Kim, J.Y. et al. The value of phosphohistone H3 as a proliferation marker for evaluating invasive breast cancers: A comparative study with Ki67. Oncotarget. 8(39), 65064-65076 (2017).

There is NO Competing Interest.

supplementaryinformation.pdf

Download PDF

Version 1

posted

You are reading this latest preprint version

OMG-Net: A Deep Learning Framework Deploying Segment Anything to Detect Pan-Cancer Mitotic Figures from Haematoxylin and Eosin-Stained Slides

Status:

Version 1

Abstract

Figures

Introduction

Results

Developing a large-scale MF dataset

Performance of MF detection in various human tumours

Reviewing the SAM masks enhanced the detection performance

Canine mitotic figures help to train the detection of human mitotic figures

Including mitotic-like figures and non-mitotic objects is key to improving model precision

Discussion

Nuclei contours represent a key feature for improving mitotic-figure detection

Decoupling segmentation and classification helps improve the detection performance

Large-scale MF datasets provide a resource for the development of pan-cancer models

Conclusion

Methods

Dataset

Ethical Approvals

Open-source datasets

In-house dataset

Data Curation

OMG-Net: A Two-Stage Detection Framework

Model Development and Testing

Training and Validation

Data Augmentation

Test set and performance metrics

Declarations

Data availability

Code availability

References

Additional Declarations

Supplementary Files

Status:

Version 1