
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39232028
71250
10.1038/s41598-024-71250-4
Article
Brain tumor image segmentation method using hybrid attention module and improved mask RCNN
Yuan Jinglin p2316169@mpu.edu.mo

https://ror.org/02sf5td35 grid.445017.3 0000 0004 1794 7946 School of Applied Science, Macao Polytechnic University, Macau, 999078 China
4 9 2024
4 9 2024
2024
14 2061520 3 2024
26 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
To meet the needs of automated medical analysis of brain tumor magnetic resonance imaging, this study introduces an enhanced instance segmentation method built upon mask region-based convolutional neural network. By incorporating squeeze-and-excitation networks, a channel attention mechanism, and concatenated attention neural network, a spatial attention mechanism, the model can more adeptly focus on the critical regions and finer details of brain tumors. Residual network-50 combined attention module and feature pyramid network as the backbone network to effectively capture multi-scale characteristics of brain tumors. At the same time, the region proposal network and region of interest align technology were used to ensure that the segmentation area matched the actual tumor morphology. The originality of the research lies in the deep residual network that combines attention mechanism with feature pyramid network to replace the backbone based on mask region convolutional neural network, achieving an improvement in the efficiency of brain tumor feature extraction. After a series of experiments, the precision of the model is 90.72%, which is 0.76% higher than that of the original model. Recall was 91.68%, an increase of 0.95%; Mean Intersection over Union was 94.56%, an increase of 1.39%. This method achieves precise segmentation of brain tumor magnetic resonance imaging, and doctors can easily and accurately locate the tumor area through the segmentation results, thereby quickly measuring the diameter, area, and other information of the tumor, providing doctors with more comprehensive diagnostic information.

Keywords

Convolutional neural network
Brain tumors
Diagnostic information
Feature pyramid network
Magnetic resonance imaging
Subject terms

Biological techniques
Computational biology and bioinformatics
Medical research
Mathematics and computing
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Brain tumor is one of the common malignant tumors in the central nervous system, and its clinical diagnosis and treatment have received extensive attention1. Due to the special location and function of the brain, brain tumors pose a serious threat to patients' quality of life and survival. Therefore, early and accurate diagnosis and treatment is the key to improve the survival rate of patients. Magnetic Resonance Imaging (MRI) is the most commonly used non-invasive diagnostic modality for brain tumors2, It offers high-resolution, multi-parametric images of the brain, aiding physicians in assessing the size, location, morphology, and relation of the tumor to surrounding tissues3. However, the interpretation and analysis of MRI images typically constitute a complex and time-consuming process, necessitating expertise from highly specialized radiologists. Given this context, segmenting brain tumor MRI images has emerged as an imperative challenge, aiming to enhance the accuracy and efficiency of diagnoses.

Image segmentation is the process of identifying and isolating regions or targets of interest from medical images, such as tumor tissues4. In brain tumor MRI images, accurate tumor segmentation aids in evaluating the tumor's size, growth rate, and extent of invasion, thus informing surgical resection plans, radiation therapy, or chemotherapy5. Over the past several decades, researchers have proposed numerous automatic or semi-automatic algorithms for brain tumor segmentation. For example, Thayumanavan et al. introduced an optimized median filtering method for MRI skull stripping, extracting abnormal brain tissue under low contrast and precisely delineating the affected tissue's boundaries, analyzing the texture, shape, and other characteristics of the brain tumor6. Alam et al. employed a model using a template-based K-means and an enhanced Fuzzy C-Means (TKFCM) algorithm for detecting human brain tumors in MRI images7. Anand et al. proposed a machine learning and medical-assisted multimodal approach to segment and classify brain tumors, using geometric mean filters to remove image noise, and the Fuzzy C-Means algorithm to partition the image into smaller segments, achieving more accurate brain tumor region identification8. Zhan et al. combined semi-supervised learning theory with the image space and clinical prior knowledge of brain tumors, introducing a novel brain tumor segmentation method that, with limited labeled data, enhances segmentation accuracy through CoTraining with multiple classifiers9. In order to compare the performance of brain tumor segmentation algorithms, Kumar A and other researchers started with magnetic resonance images and used the brain tumor image segmentation benchmark dataset -2012 for simulation analysis. The results showed that the prediction accuracies of Otsu, Watershed, Level Set, k-means, and Discrete Wavelet Transform algorithms were 73.90%, 78.12%, 81.90%, 84.75%, and 88.12%, respectively10. These methods include traditional threshold segmentation, region growth, edge detection, etc. Although they have achieved certain results, they still face challenges such as poor accuracy and low efficiency in practical clinical applications.

In recent years, the rise of deep learning technologies has ushered in new breakthroughs for brain tumor segmentation. Convolutional Neural Networks (CNNs) have demonstrated exemplary performance in the domain of image processing and have been extensively employed for brain tumor segmentation tasks. Through deep learning, feature representations can be learned from vast MRI image datasets, thereby achieving more precise brain tumor segmentation. For example, Ma et al. introduced a novel brain tumor segmentation approach that models spatial relationships and channel relationships among different regions within images, enhancing visual detection and quantitative metrics for brain tumor segmentation tasks11. Micallef et al. presented a new U-Net++ architecture that, by improving the loss function and the number of convolutional blocks, realizes lightweight yet high-precision brain tumor segmentation12. Ma et al. proposed a 2D end-to-end attention R2U-Net model with Multi-Task Deep Supervision (MTDS), which extracts semantic information from images and integrates multi-scale attention mechanisms to precisely localize brain tumor areas13. Latif et al. introduced a method to enhance the automatic segmentation process by incorporating size variability within CNNs, achieving pixel-wise classification of tumor slices14. In order to compare the performance of several image segmentation algorithms used for brain tumor diagnosis, expert Kumar A selected Otsu, watershed, level set, K-means, HAAR discrete wavelet transform, and CNN, and simulated them on a specialized brain tumor image segmentation benchmark dataset 2018. The results showed that the measurement accuracies of the six methods were 71.42%, 78.26%, 80.45%, 84.34%, 86.95%, and 91.39%, respectively, and the response time of CNN was 2.519 s15.

However, brain tumor segmentation tasks still face numerous challenges. Firstly, different types of brain tumors exhibit vast variability in morphology, size, and location, which amplifies the complexity of segmentation16. Secondly, MRI images might contain issues such as noise, artifacts, and image camouflage, all of which could potentially degrade the performance of segmentation algorithms. Additionally, training deep learning models requires a substantial amount of annotated data, and acquiring accurate annotated data in the medical domain is a challenging endeavor17. Addressing these issues, this paper proposes an advanced instance segmentation method for the detection and segmentation of brain tumor MRI images. This approach is not only specifically optimized for brain tumor MRI images, ensuring its efficiency and accuracy during detection and segmentation, but also encompasses additional practical functionalities. For instance, this method can conduct a detailed symptomatic analysis of brain tumor MRI images, accurately measure the diameter of the brain tumor, and compute its area, providing physicians with a more comprehensive diagnostic overview.

Materials and methods

Data sets collection and processing

The dataset used in this study is the BraTs2020 public dataset. The dataset comprises 1066 brain tumor images exhibiting various symptoms. These images are sourced from medical imaging centers worldwide and encompass a myriad of brain tumor types, including benign, malignant, primary, and secondary tumors. All images are stored in JPG format, boasting high clarity and resolution18. We annotated the brain tumor MRI images using the labelme image annotation tool. Three experts, with extensive experience in the field of brain tumor research, were responsible for this task, ensuring that each brain tumor case was represented by irregular closed curves. After the initial annotation, we conducted a second review of all the annotations to eliminate any risk of missed or incorrect markings. All the annotation information is saved in a json file, as shown in Fig. 1, which includes the name, size of the image, and details of each pixel in the disease area.Fig. 1 Brain tumor MRI image dataset. A represents the original brain tumor MRI images. B denotes the annotated brain tumor MRI images. C is the json file containing both the brain tumor MRI images and their corresponding annotations.

Image augmentation

In order to solve the problem of small data set and lack of image diversity, the model accuracy may be poor after training. In this study, data enhancement was used to expand the dataset19. By applying spatial transformations such as rotation and mirroring20, the symptomatic regions of the brain tumor MRI images were adjusted, enhancing the diversity of the brain tumor images. In addition, to avoid image segmentation errors caused by insufficient or excessive brightness of the image, brightness adjustment is performed on the image. To further enhance the model's generalization capabilities, we introduced noise into the images. After data augmentation, the number of brain tumor MRI images increased to 1726. These images were divided into training, validation, and test sets at a ratio of 7:2:1, respectively. Images from different scenarios are illustrated in Fig. 2.Fig. 2 Original image and data augmented image.

MRI brain tumor instance segmentation based on improved mask RCNN

Mask RCNN, an advanced instance segmentation technique, integrates a mask generation mechanism based on Faster R-CNN, enhancing its object detection and instance segmentation capabilities21,22. Due to the limited application of this algorithm in brain tumor image segmentation tasks by previous researchers, it is necessary to formalize this algorithm for the convenience of medical experts in this field to read and understand this paper. In this study, we made specific modifications to Mask RCNN, proposing an instance segmentation network tailored for MRI brain tumor images. Unlike the conventional architecture, we employed a ResNet that amalgamates the attention mechanism23 with the Feature Pyramid Network (FPN) to replace the original Mask RCNN backbone, enabling more efficient extraction of brain tumor features from MRI images. Brain tumor image recognition requires high professional skills from doctors, and recognition errors pose significant risks to doctors and patients. Therefore, the efficiency of traditional manual recognition of brain tumor images is low. Therefore, the motivation of this study is to design an automated segmentation method for brain tumor images, which plays a role in identifying tumor regions in the image and improving the recognition efficiency of doctors for this type of image. This innovative combination approach can fully leverage the instance segmentation function, attention mechanism, and feature processing and abstraction capabilities of Mask RCNN, making the improved algorithm have higher potential for image segmentation applications compared to a single algorithm. This is also the technical contribution of this paper. Another contribution of this study is the innovative application of the improved Mask RCNN algorithm to medical tumor image segmentation tasks, providing more automated processing methods for such tasks, significantly reducing valuable medical manpower and improving image processing efficiency while achieving processing objectives. Compared with traditional manual methods for identifying brain tumor regions, the difference of this automated method is that the segmentation and recognition efficiency of brain tumor regions is extremely high, and the recognition results do not rely on the subjective judgment and experience of doctors. Subsequently, these extracted features are fed into the Region Proposal Network (RPN) to generate region proposals for potential tumor areas. To ensure precision, the RoIAlign technique24 is utilized to extract features from each proposal, aligning them meticulously with the original input image. Finally, the network undertakes two parallel operations: on one hand, potential tumor regions are classified via a fully connected layer (FC) and their bounding boxes are regressed; on the other hand, a Fully Convolutional Network (FCN) generates a highly accurate segmentation mask, pinpointing the precise location of the tumor. To offer a more intuitive portrayal of this innovative approach, Fig. 3 provides a detailed depiction of the MRI brain tumor image segmentation process based on this enhanced Mask RCNN structure. Further technical details and experimental validations will be elaborated upon in the subsequent sections. The tumor types involved in image segmentation research include glioma, meningioma, astrocytoma, and neuroblastoma.Fig. 3 Structure of MRI brain tumor segmentation method based on improved Mask RCNN.

Attention module

The inspiration for the attention mechanism is drawn from natural human attentive behaviors. Within our visual, auditory, or other sensory systems, we often assign higher priority to certain pieces of information, thereby overlooking other less relevant details. This phenomenon has been emulated in the domain of deep learning, enabling models to focus more intently on the pivotal information within the input data, resulting in more accurate task execution. In this study, we selected the channel attention SENet25 and spatial attention CANet26 for a series of ablation experiments. We delved into the roles and effects of different attention mechanisms specifically within the context of MRI brain tumor image segmentation.

SENet primarily focuses on capturing the deep semantic features of images, enabling the network to discern the main content of images more accurately. Its strength lies in intuitively modeling the inter-channel dependencies in the feature maps, subsequently allowing for dynamic adjustment and optimization of these channels. As depicted in Fig. 4, the SENet starts by employing global average pooling to consolidate each channel, generating a vector C that characterizes the properties of every channel. To compute channel weights, SENet introduces a two-layered fully connected network: the first layer compresses the vector's dimensions to 1/r of its original size and activates it through the ReLU function; the second layer then expands its dimensions back to the original size, using the Sigmoid function to output weights for the c channels. This dual-layer design not only captures the intricate relationships between channels deeply but also reduces computational overhead to some extent. Ultimately, SENet optimizes and recalibrates each channel in the feature map by multiplying these weights with the original input features.Fig. 4 Structure of SENet model.

CANet aims to precisely identify key locations within an image, thereby focusing on significant feature regions. Its central idea revolves around capturing and enhancing the image's local features through specialized spatial dimension processing. As illustrated in Fig. 5, for a given input feature, average pooling is first applied separately along the image's horizontal and vertical dimensions, producing two sets of feature maps closely associated with spatial positions. These two resulting feature maps are then concatenated along the spatial axis, yielding a feature representation that incorporates information from both directions. Following this, a 1 × 1 convolution operation combined with subsequent activation reshapes these features, reinforcing essential information while attenuating irrelevant noise. After the initial fusion and reshaping, the features are processed in segments, dividing them into two parts, each undergoing convolution and activation to derive two distinct feature representations. These two representations are then weighted and combined at their corresponding spatial positions, allowing the model to dynamically adjust the importance of each position based on information from both directions. In Fig. 5, the dimensions of the input module and output module are the same, both being H × W × C. The size of two-dimensional convolutions is 3 × 3 × 3.Fig. 5 Structure of SENet model.

Backbone network

The backbone network typically serves as a feature extraction mechanism, drawing out features from images to lay the foundation for subsequent classification, localization, and segmentation. In this study, we employed a ResNet50, infused with attention modules and FPN, as the backbone for Mask RCNN to extract features from MRI brain tumor images27. ResNet predominantly consists of residual blocks, each comprising several convolution layers and endowed with a skip connection to preserve input information. Such a configuration effectively mitigates the gradient vanishing problem encountered in deep neural networks. A ResNet integrated with attention modules can hone in more on the pivotal regions within images, which is particularly vital given the multi-scale characteristics of brain tumors28. The model structure is detailed in Fig. 6, which delineates the specifics, including the convolution kernels, strides, and channel counts across various layers, while also highlighting the precise locations and roles of the attention modules. Through this backbone network, we can extract the intricate details and texture features of MRI brain tumor images more efficiently, supporting subsequent tumor detection and segmentation.Fig. 6 Structure of ResNet50 network improved with attention module.

In ResNet50, there are four key points involved from stage 0 to stage 4, namely input preprocessing, Bottleneck block operation, introduction of residual connections, and dimension transformation processing. Firstly, ResNet50 needs to preprocess the input data in stage 0, such as resizing. Secondly, in Stage 1, it is necessary to use 3 Bottlenecks to reduce the number of parameters and maintain the dimensionality of the feature mapping. Then, in stages 2, 3, and 4, cross layer feature transfer and reuse need to be achieved through Bottleneck blocks. Finally, output the final result. In addition, all Bottleneck blocks use residual connections, and dimension transformation is achieved by adding linear transformations on shortcut connections. Tumors within brain MRI images exhibit various sizes and morphologies, implying that features of different spatial resolutions and scales are crucial for accurately detecting and localizing these tumors. The combination of ResNet50, attention modules, and FPN offers us a robust mechanism to capture these attributes. When processing brain tumor MRI images, the ResNet50 is divided into five stages. At the conclusion of each stage, the feature map's size is halved while the number of channels doubles. As a result, five feature maps of differing scales are generated, denoted as [C1, C2, C3, C4, C5]. We commence with the deepest layer of ResNet50, C5. This layer has the smallest spatial resolution but contains the most semantic information, essential for recognizing the overall tumor structure. To harness more local details and morphological information, we employ FPN to construct a feature pyramid starting from C5. Utilizing 1 × 1 convolutions and upsampling, we amalgamate low-level detail information (such as tumor edges or small tumors) with high-level semantic information (like the overall tumor shape or large tumors), eventually producing a series of feature maps: [P1, P2, P3, P4, P5]. Since computing the feature map corresponding to C1 is time-intensive, P1 is not used in subsequent steps. The structure of FPN is shown in Fig. 7. These feature maps, with their varying spatial resolutions, encapsulate deeper semantic information, suitable for detecting tumors of diverse scales and morphologies.Fig. 7 The structure of FPN.

Generation of region of interest (RoI) and RoIAlign

The enhanced backbone feature extraction network produces feature maps of different scales that are fed into the RPN network. The primary objective of the RPN is to propose regions on the feature map where a target might be located, and these proposed regions of interest are what we refer to as RoIs. By processing feature maps across multiple scales, the RPN ensures effective tumor detection across varying image scales. However, the RoIs generated by the RPN typically have a fixed size and shape, which might not fully align with the actual morphology of brain tumors. To address this, we employ the RoIAlign technique. The aim of RoIAlign is to map these irregular RoIs onto a feature map of fixed dimensions for subsequent classification and regression tasks. Specifically, when extracting a feature map of fixed size from an RoI, RoIAlign uniformly partitions this RoI into a grid of fixed dimensions, sampling four points within each grid cell. It then employs bilinear interpolation to obtain the final feature values, as illustrated in Fig. 8. This allows for the precise position of that point on the original feature map to be calculated and the corresponding feature value to be obtained. This ensures that, regardless of how the RoI coordinates change, the extracted features are accurate and consistent. Thus, when dealing with the complex and diverse morphologies of brain tumors, the model can still yield satisfactory results.Fig. 8 RoIAlign implementation process.

Brain tumor instance segmentation and loss function

The feature maps outputted by RoIAlign are fed into fully connected layers and convolutional layers for subsequent processing. The fully connected layers primarily handle the feature maps from RoIAlign for classification and bounding box regression tasks. The convolutional layers are employed for the segmentation of brain tumor MRI images, providing a precise contour for each brain tumor instance in the MRI images. To effectively perform instance segmentation of brain tumors, we utilize the following loss function to measure the disparity between the predicted and actual results.1 Lcls=-∑ipilogp^i+1-pilog1-p^i

2 Lreg=∑ismoothL1ti-t^i

3 Lmask=-∑jmjlogm^j+1-mjlog1-m^j

4 L=λclsLcls+λregLreg+λmaskLmask

Lcls represents the classification loss, aimed at discerning whether the region of interest encompasses a brain tumor and subsequently assigns the appropriate category. Within this equation, pi denotes the genuine class label, while p^i indicates the model's predicted probability. Lreg outlines the bounding box regression loss, ensuring that the predicted bounding box closely encapsulates the brain tumor instance. Here, ti is the actual bounding box coordinates, and t^i signifies the predicted box coordinates by the model, smoothL1· is the smoothing calculation function under L1 regularization condition Lmask introduces the instance segmentation loss, gauging the difference between the predicted contour of the brain tumor and its actual contour. In this context, mj corresponds to the real label of each pixel, and m^j symbolizes the pixel probability as predicted by the model. Lmask consolidates the model's overall loss function, which establishes the brain tumor's identification, localization, and segmentation efficacy. The coefficients λcls, λreg and λmask are weight parameters, adjustable based on the specific objective.

Experimental platform and parameter setting

The construction, training and testing of the instance segmentation network are all based on the Pytorch deep learning framework, written in Python language, and trained with GPU acceleration. The experiment was carried out under windows10 operating system, and the computer hardware was configured as Intel(R) Core(TM) i7-12700F CPU, 32 GB running memory, and NVIDIA GeForce RTX 3060Ti GPU. Parameters of model training are shown in Table 1. A total of 200 rounds are trained, and the model achieves maximum convergence.Table 1 Training parameters.

Parameter	Value	
Optimizer	Adam	
Learning rate	0.001	
Momentum	0.93	
Batch size	16	
Label smoothing	0.005	
RPN anchor scales	[16, 32, 64, 128, 256]	

Evaluation metrics

In order to verify the performance of the model, this study used accuracy Precision, Recall, F1 Score, and average segmentation intersection ratio (MIoU) to evaluate the model performance. The calculation formula is as follows:5 Precision=TPTP+FP

6 Recall=TPTP+FN

7 F1Score=2×Precision×RecallPrecision+Recall

8 MIoU=1k+1∑i=0kTPFN+FP+TP

where is the number of brain tumors correctly detected, the number of brain tumors incorrectly detected, and the number of brain tumors not correctly detected. Precision represents the probability that the detection area contains a brain tumor target, and Recall represents how many brain tumor targets are detected in the detection image. F1 Score is used to evaluate the overall performance of the model. The closer the value is to 1, the better the model performance is. Used to assess the degree of overlap between the predicted segment and the real segment. Where k represents a class, and k + 1 represents a background class.

In the specific brain tumor MRI image segmentation task, the paper used images, time complexity, and dice scores for analysis. In addition, Grad CAM was used for visualization in the performance analysis of attention mechanism.

Results and Analysis

Segmentation results of brain tumor MRI images

In order to verify the performance of the improved model in brain tumor MRI image instance segmentation, we selected 72 images that were not involved in model training for result testing. These images cover many types of brain tumors, including glioma, meningioma, astrocytoma and neuroblastoma, ensuring the comprehensiveness and objectivity of the test. The experimental results are shown in Fig. 9. By observing the experimental results, we can see that the model shows a high accuracy in the extraction of brain tumor MRI images. Even under the condition of low image resolution, more noise or less obvious contrast, the model can effectively distinguish brain tumor features, showing the robustness and efficiency of the model.Fig. 9 Segmentation results of the improved model.

The time complexity analysis of the improved model and the dice score results of segmented images are shown in Table 2.Table 2 Time complexity analysis of improved models and dice scores for segmented images.

Types of brain tumors	Time consuming/s	Dice score	
Glioma	1.37	1.29	1.55	1.49	0.945	0.921	0.933	0.924	
Meningioma	1.65	1.54	1.21	1.36	0.952	0.966	0.935	0.972	
Astrocytoma	1.03	1.17	1.29	1.45	0.919	0.951	0.927	0.983	
Neuroblastoma	0.95	1.34	1.12	1.53	0.993	0.973	0.941	0.928	

From Table 2, it can be seen that in terms of time complexity, the average time consumption of the improved model designed in the paper on gliomas, meningiomas, astrocytomas, and neuroblastoma is 1.425 s, 1.440 s, 1.235 s, and 1.235 s, respectively. It can be seen that the improved model has less time consumption and better performance. In addition, in terms of dice scores, the average values of the improved model on gliomas, meningiomas, astrocytomas, and neuroblastoma were 0.931, 0.956, 0.945, and 0.959, respectively. The closer the dice score is to 1, the more perfect the image segmentation matching is. From this, it can be seen that the improved model designed in the paper has good accuracy and completeness in brain tumor image segmentation.

Analysis of segmentation results of brain tumor MRI images

After example segmentation of brain tumor MRI images, the model successfully generates accurate detection frames and masks for brain tumor regions. Specifically, the detection box mainly provides the coordinate information of the brain tumor in the image, which provides a preliminary positioning reference for doctors or other relevant personnel. Further, using the maximum length and width of the detection box, we were able to calculate the diameter of the brain tumor lesion area, providing an important quantitative indicator for subsequent treatment or surgery. The mask provides more detailed information about the shape of the brain tumor. It is a binary graph of the same size as the original image, with the brain tumor part labeled 1 and the non-brain tumor part labeled 0. This allows us to calculate the area of the brain tumor in detail and then make a judgment about the size of the brain tumor and the likely growth trend. Figure 10 shows the analytical results after segmentation of brain tumor MRI images.Fig. 10 Analysis of the segmentation results of brain tumor MRI images.

Ablation experiment

In order to investigate the effects of different attention mechanisms on the performance of the model, we designed ablation experiments to verify the performance of different attention modules. For the correctness of the ablation experiment, the setting of model hyperparameters and the operating environment are the same, and the experimental results are shown in Table 3. It can be seen from the table that compared with the original Mask RCNN model, the model after introducing the attention mechanism has improved in Precision, Recall, F1 Score and MIoU. Among them, the CA attention mechanism is the best in all indicators, achieving 90.72% accuracy, 91.68% recall rate, 91.85% F1 score and 94.56% MIoU. This indicates that the CA attention mechanism has the most obvious effect on the segmentation of brain tumor MRI images in this experiment. Both of these two attention mechanisms effectively improve the performance of the model, which also validates the importance of attention mechanisms in the segmentation task of brain tumor MRI image instances. Attention can help the model focus on the important information in the image, so as to improve the segmentation accuracy and recall rate of the model.Table 3 Different attention mechanisms to improve the performance of the model.

Method algorithms	Precision (%)	Recall (%)	F1 score (%)	MIoU (%)	
Mask RCNN	89.96	90.73	91.24	93.17	
Mask RCNN + SE	90.31	91.62	91.69	94.22	
Mask RCNN + CA	90.72	91.68	91.85	94.56	

Visualization of attention mechanism performance

In order to more intuitively demonstrate the effect of attention mechanisms on brain tumor MRI images, we selected Grad-CAM for visualization. Gred-CAM is a class activation graph visualization technique for deep learning models that highlights areas of the input image that are critical to the model's decision making. To do this, we compared the original model with two models that incorporate different attention mechanisms. It can be observed from Fig. 11a that the original model in some cases pays too much attention to the background region and does not capture enough key information of the whole image. Figure 11b,c show that after incorporating the attention mechanism, the model gives higher attention to the brain tumor region in the MRI image. These visualizations highlight the power of the attention module, which allows the model to more intensively target key regions where brain tumors are located, thereby improving its ability to distinguish between different brain tumor instances. However, it is also worth noting that, despite the addition of the attention mechanism, SENet and CANet still exhibit small biases in positioning in certain scenarios. This slight deviation may lead to errors in the detection of brain tumors in specific cases, and affect its segmentation accuracy. Among these models, the model with CANet showed more focused and precise attention, especially on small tumor targets, which was significantly better than the other models.Fig. 11 Visual comparison between the original model and the attention model.

Comparison of different models

In order to further verify the superiority of the improved Mask RCNN model in case segmentation of brain tumor MRI images, two case segmentation models, YOLACT and SOLO, were selected for comparison. YOLACT is a real-time instance segmentation algorithm that obtains instance masks from the current image mask and bbox instance mask coefficients and uses them for image segmentation29. SOLO is a direct end-to-end instance segmentation algorithm that takes an image as input and outputs the probability of the corresponding class of the image's instance mask, with low computational complexity30. In order to ensure fairness, we set the training parameters of each model to the same conditions, including batch size, learning rate, weight attenuation, etc., and each model achieved maximum convergence. Specific performance indicators are shown in Table 4. In recent years, advanced methods from reference31 and reference32 have also been selected as comparative models. From the table data, it can be seen that the improved Mask RCNN model outperforms YOLACT, SOLO, and the advanced image segmentation algorithms TransUNet, Swin Transformer, Semantic Segmentation (SETR), reference31 method and reference32 method that have emerged in recent years in all four metrics. In particular, for MIoU, the improved Mask RCNN reaches 94.56%, which is significantly better than the other seven models. The image segmentation method designed in reference31 requires high sample richness. Due to the difficulty in meeting the requirements of the dataset size and the number of tumor features contained in the samples, the performance of this method is not good. In the method designed in reference32, the histogram mean method is used for preprocessing and image filtering, which leads to the loss of some of the true information in the image, thereby reducing the overall computational performance of the model. This means that the performance of the improved Mask RCNN model in the brain tumor MRI image instance segmentation task is obviously superior, and further proves the positive benefits of our improved strategy for the instance segmentation task.Table 4 Different attention mechanisms to improve the performance of the model.

Method Algorithms	Precision (%)	Recall (%)	F1 score (%)	MIoU (%)	
YOLACT	90.22	91.15	90.89	92.08	
SOLO	89.65	90.72	90.17	91.26	
Improved Mask RCNN	90.72	91.68	91.85	94.56	
TransUNet	85.41	86.24	86.33	87.19	
Swin Transformer	87.73	87.52	86.36	87.40	
SETR	86.58	84.27	85.19	86.24	
Reference31 method	87.92	88.79	86.94	87.02	
Reference32 method	88.51	88.26	87.53	88.28	

Finally, to further ensure the reliability of the test results, brain MRI image tumor segmentation tests were conducted on a patient basis. In this test. Considering that medical test results are closely related to the patient's medical behavior, medical costs, and quality of life, the rules for determining test results should be set as strictly as possible. Therefore, if and only if the algorithm can correctly segment all images of a patient, it is considered that the algorithm has correctly processed the image of that patient. The test results show that the improved Mask RCNN model designed this time correctly identified 86.2% of patients, which is higher than all other comparison models. The second highest recognition accuracy is the method in reference32, which is 83.5%. In order to better validate the performance of the improved Mask RCNN model designed in the paper, the Receiver Operating Characteristic (ROC) curve and Area Under Curve (AUC) area of different models were compared and studied. The ROC curve can evaluate the performance of classification models and compare the advantages and disadvantages of different models. AUC is the area under the ROC curve, which is a quantified value used to evaluate the performance of a model. The closer its value is to 1, the better the performance of the model. The comparison results of ROC curves for different models are shown in Fig. 12.Fig. 12 Comparison results of ROC curves for different models.

From Fig. 12, it can be seen that in the comparison of ROC curves, the AUC values of the improved Mask RCNN model, YOLACT, SOLO, Reference32 method, Reference31 method, Swin Transformer, TransUNet, and SETR are 0.973, 0.957, 0.943, 0.924, 0.917, 0.905, 0.903, and 0.874, respectively. It can be seen that the AUC value of the improved Mask RCNN model is closer to 1, which also indicates that the model has better performance and can better segment brain tumor MRI images.

Limitations of case segmentation of brain tumor MRI images

In the in-depth study of brain tumor MRI image segmentation, the improved model shows remarkable performance. It can accurately detect and segment tumor lesion areas in MRI images, and further analyze key indicators such as tumor area and diameter. However, there are some limitations to this study. The first problem is the treatment of tumor classification. There are many types of brain tumors, but in this study, for all tumors, we only set them into a single category for example segmentation, without further subdividing their specific types. Although the model can accurately capture the lesion area, for different types of tumors, medical professionals still need to rely on their rich expertise to make specific judgments and classifications. This means that while the model provides a valuable initial analysis for doctors, the intervention of medical experts is still needed in the exact diagnosis and treatment decisions. In future work, further expanding the model to identify and classify different types of brain tumors will be a challenging and necessary direction.

Conclusion

In this study, we proposed a high-precision segmentation method for brain tumors in MRI images. In view of the limitations of the traditional Mask RCNN model in processing this task, we combined the channel attention mechanism of SENet and the spatial attention mechanism of CANet to enhance the model's ability to recognize and extract key features, ensuring that the model can capture and enhance tumor features in MRI images more accurately. Combined with the attention mechanism, the model can more accurately capture and enhance tumor features in MRI images. In addition, by using the Feature Pyramid Network (FPN) and RoIAlign technology, we further ensure that the model can give accurate segmentation results when dealing with brain tumors of various sizes and shapes. Compared with the original Mask RCNN model, our improved method has achieved significant improvement in many performance indicators, among which the precision reaches 90.72%, which is 0.76% higher than that of the original model. Recall reached 91.68%, which was 0.95% higher than the original model. MIoU reaches 94.56%, which is 1.39% higher than the original model. Through visual analysis of the performance of different attention mechanisms, we found that CANet was particularly good at focusing on brain tumor lesion areas. In order to confirm the superiority of our method, we compared it with other case segmentation models, and the results show that the improved Mask RCNN has the most outstanding performance in case segmentation of brain tumor MRI images. Finally, based on the results of case segmentation, we further analyzed and extracted the key medical indicators of brain tumors, such as area and diameter, to provide valuable reference information for clinical medical research.

Author contributions

The author confirms being the sole contributor of this work and has approved it for publication.

Data availability

All data generated or analysed during this study are included in this article.

Competing interests

The author declares no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Svart K Jensen RH Høgedal L Vukovic-Cvetkovic V Beier D Korsbæk JJ Phenotyping non-idiopathic pseudotumor cerebri syndrome—A prospective cohort study Cephalalgia 2022 42 14 1510 1520 10.1177/03331024221120073 35983777
Svart, K. et al. Phenotyping non-idiopathic pseudotumor cerebri syndrome—A prospective cohort study. Cephalalgia 42(14), 1510–1520 (2022).35983777 10.1177/03331024221120073
2. Çinar A Yildirim M Detection of tumors on brain MRI images using the hybrid convolutional neural network architecture Med. Hypotheses 2020 139 109684 10.1016/j.mehy.2020.109684 32240877
Çinar, A. & Yildirim, M. Detection of tumors on brain MRI images using the hybrid convolutional neural network architecture. Med. Hypotheses 139, 109684 (2020).32240877 10.1016/j.mehy.2020.109684
3. Pandiselvi T Maheswaran R Efficient framework for identifying, locating, detecting and classifying MRI brain tumor in mri images J Med Syst. 2019 43 7 189 10.1007/s10916-019-1253-1 31111265
Pandiselvi, T. & Maheswaran, R. Efficient framework for identifying, locating, detecting and classifying MRI brain tumor in mri images. J Med Syst. 43(7), 189 (2019).31111265 10.1007/s10916-019-1253-1
4. Hashemzehi R Mahdavi SJS Kheirabadi M Kamel SR Detection of brain tumors from MRI images base on deep learning using hybrid model CNN and nade Biocybern. Biomed Eng. 2020 40 3 1225 1232 10.1016/j.bbe.2020.06.001
Hashemzehi, R., Mahdavi, S. J. S., Kheirabadi, M. & Kamel, S. R. Detection of brain tumors from MRI images base on deep learning using hybrid model CNN and nade. Biocybern. Biomed Eng. 40(3), 1225–1232 (2020).10.1016/j.bbe.2020.06.001
5. Sharma SR Alshathri S Singh B Kaur M Mostafa RR El-Shafai W Hybrid multilevel thresholding image segmentation approach for brain MRI Diagnostics. 2023 13 5 925 10.3390/diagnostics13050925 36900074
Sharma, S. R. et al. Hybrid multilevel thresholding image segmentation approach for brain MRI. Diagnostics. 13(5), 925 (2023).36900074 10.3390/diagnostics13050925
6. Thayumanavan M Ramasamy A An efficient approach for brain tumor detection and segmentation in MR brain images using random forest classifier Concurr. Eng. 2021 29 3 266 274 10.1177/1063293X211010542
Thayumanavan, M. & Ramasamy, A. An efficient approach for brain tumor detection and segmentation in MR brain images using random forest classifier. Concurr. Eng. 29(3), 266–274 (2021).10.1177/1063293X211010542
7. Alam MS Rahman MM Hossain MA Islam MK Ahmed KM Ahmed KT Singh BC Miah MS Automatic human brain tumor detection in MRI image using template-based k means and improved fuzzy C means clustering algorithm Big Data Cognit. Comput. 2019 3 2 27 10.3390/bdcc3020027
Alam, M. S. et al. Automatic human brain tumor detection in MRI image using template-based k means and improved fuzzy C means clustering algorithm. Big Data Cognit. Comput. 3(2), 27 (2019).10.3390/bdcc3020027
8. Anand L Rane KP Bewoor LA Bangare JL Surve J Raghunath MP Sankaran KS Osei B Development of machine learning and medical enabled multimodal for segmentation and classification of brain tumor using MRI images Comput. Intell. Neurosci. 2022 2022 1 8 10.1155/2022/7797094
Anand, L. et al. Development of machine learning and medical enabled multimodal for segmentation and classification of brain tumor using MRI images. Comput. Intell. Neurosci. 2022, 1–8 (2022).10.1155/2022/7797094
9. Zhan T Shen F Hong X Wang X Chen Y Lu Z Yang G A glioma segmentation method using cotraining and superpixel-based spatial and clinical constraints IEEE Access 2018 6 57113 57122 10.1109/ACCESS.2018.2873674
Zhan, T. et al. A glioma segmentation method using cotraining and superpixel-based spatial and clinical constraints. IEEE Access 6, 57113–57122 (2018).10.1109/ACCESS.2018.2873674
10. Kumar A Chauda P Devrari A Machine learning approach for brain tumor detection and segmentation Int. J. Org. Collect. Intell. 2021 11 3 68 84 10.4018/IJOCI.2021070105
Kumar, A., Chauda, P. & Devrari, A. Machine learning approach for brain tumor detection and segmentation. Int. J. Org. Collect. Intell. 11(3), 68–84 (2021).10.4018/IJOCI.2021070105
11. Ma Q Zhou S Li C Liu F Liu Y Hou M Zhang Y Dgrunit: Dual graph reasoning unit for brain tumor segmentation Comput. Biol. Med. 2022 149 106079 10.1016/j.compbiomed.2022.106079 36108413
Ma, Q. et al. Dgrunit: Dual graph reasoning unit for brain tumor segmentation. Comput. Biol. Med. 149, 106079 (2022).36108413 10.1016/j.compbiomed.2022.106079
12. Micallef N Seychell D Bajada CJ Exploring the U-net++ model for automatic brain tumor segmentation IEEE Access. 2021 9 125523 125539 10.1109/ACCESS.2021.3111131
Micallef, N., Seychell, D. & Bajada, C. J. Exploring the U-net++ model for automatic brain tumor segmentation. IEEE Access. 9, 125523–125539 (2021).10.1109/ACCESS.2021.3111131
13. Ma S Tang J Guo F Multi-task deep supervision on attention R2U-net for brain tumor segmentation Front Oncol. 2021 17 11 704850 10.3389/fonc.2021.704850
Ma, S., Tang, J. & Guo, F. Multi-task deep supervision on attention R2U-net for brain tumor segmentation. Front Oncol. 17(11), 704850 (2021).10.3389/fonc.2021.704850
14. Latif U Shahid AR Raza B Ziauddin S Khan MA Anend-to-end brain tumor segmentation system usingmulti-inception-unet Int. J. Image Syst. Technol. 2021 31 4 1803 1816 10.1002/ima.22585
Latif, U., Shahid, A. R., Raza, B., Ziauddin, S. & Khan, M. A. Anend-to-end brain tumor segmentation system usingmulti-inception-unet. Int. J. Image Syst. Technol. 31(4), 1803–1816 (2021).10.1002/ima.22585
15. Kumar A Study and analysis of different segmentation methods for brain tumor MRI application Multimedia Tools Appl. 2023 82 5 7117 7139 10.1007/s11042-022-13636-y
Kumar, A. Study and analysis of different segmentation methods for brain tumor MRI application. Multimedia Tools Appl. 82(5), 7117–7139 (2023).10.1007/s11042-022-13636-y
16. Rajan PG Sundar C Brain tumor detection and segmentation by intensity adjustment J. Med. Syst. 2019 43 8 282 10.1007/s10916-019-1368-4 31300899
Rajan, P. G. & Sundar, C. Brain tumor detection and segmentation by intensity adjustment. J. Med. Syst. 43(8), 282 (2019).31300899 10.1007/s10916-019-1368-4
17. Sun J Chen W Peng S Liu B DRRNet: Dense residual refine networks for automatic brain tumor segmentation J. Med. Syst. 2019 43 7 221 10.1007/s10916-019-1358-6 31177346
Sun, J., Chen, W., Peng, S. & Liu, B. DRRNet: Dense residual refine networks for automatic brain tumor segmentation. J. Med. Syst. 43(7), 221 (2019).31177346 10.1007/s10916-019-1358-6
18. Vijila Rani K Joseph Jawhar S Palanikumar S Nanoscale imaging technique for accurate identification of brain tumor contour using nbds method J. Ambient Intell. Human Comput. 2021 12 7 7667 7684 10.1007/s12652-020-02485-y
Vijila Rani, K., Joseph Jawhar, S. & Palanikumar, S. Nanoscale imaging technique for accurate identification of brain tumor contour using nbds method. J. Ambient Intell. Human Comput. 12(7), 7667–7684 (2021).10.1007/s12652-020-02485-y
19. Xu M Yoon S Fuentes A Park DS A comprehensive survey of image augmentation techniques for deep learning Pattern Recogn. 2023 137 109347 10.1016/j.patcog.2023.109347
Xu, M., Yoon, S., Fuentes, A. & Park, D. S. A comprehensive survey of image augmentation techniques for deep learning. Pattern Recogn. 137, 109347 (2023).10.1016/j.patcog.2023.109347
20. Wang T Yan D Liu Z Xiao L Liang C Xin H Feng M Zhao Z Wang Y Diagnosis of cervical lymph node metastasis with thyroid carcinoma by deep learning application to ct images Front Oncol. 2023 13 1099104 10.3389/fonc.2023.1099104 36776294
Wang, T. et al. Diagnosis of cervical lymph node metastasis with thyroid carcinoma by deep learning application to ct images. Front Oncol. 13, 1099104 (2023).36776294 10.3389/fonc.2023.1099104
21. Tao X Zhang D Wang Z Liu X Zhang H Xu D Detection of power line insulator defects using aerial images analyzed with convolutional neural networks IEEE Trans. Syst. Man Cybernet. B 2020 50 4 1486 1498 10.1109/TSMC.2018.2871750
Tao, X. et al. Detection of power line insulator defects using aerial images analyzed with convolutional neural networks. IEEE Trans. Syst. Man Cybernet. B 50(4), 1486–1498 (2020).10.1109/TSMC.2018.2871750
22. Zhao X Xu T Peng L Li S Zhao Y Liu H He J Liang S Recognition and segmentation of teeth and mandibular nerve canals in panoramic dental x-rays by mask RCNN Displays 2023 78 102447 10.1016/j.displa.2023.102447
Zhao, X. et al. Recognition and segmentation of teeth and mandibular nerve canals in panoramic dental x-rays by mask RCNN. Displays 78, 102447 (2023).10.1016/j.displa.2023.102447
23. Niu Z Zhong G Yu H A review on the attention mechanism of deep learning Neurocomputing. 2021 452 48 62 10.1016/j.neucom.2021.03.091
Niu, Z., Zhong, G. & Yu, H. A review on the attention mechanism of deep learning. Neurocomputing. 452, 48–62 (2021).10.1016/j.neucom.2021.03.091
24. Wang S Sun G Zheng B Du Y A crop image segmentation and extraction algorithm based on mask RCNN Entropy-Switzerland 2021 23 9 1160 10.3390/e23091160
Wang, S., Sun, G., Zheng, B. & Du, Y. A crop image segmentation and extraction algorithm based on mask RCNN. Entropy-Switzerland 23(9), 1160 (2021).10.3390/e23091160
25. Zhou P Cao Y Li M Ma Y Chen C Gan X Wu J Lv X Chen C Hccanet: Histopathological image grading of colorectal cancer using cnn based on multichannel fusion attention mechanism Sci. Rep.-UK 2022 12 1 15103 10.1038/s41598-022-18879-1
Zhou, P. et al. Hccanet: Histopathological image grading of colorectal cancer using cnn based on multichannel fusion attention mechanism. Sci. Rep.-UK 12(1), 15103 (2022).10.1038/s41598-022-18879-1
26. Ma J Shi G Li Y Zhao Z Maff-net: Multi-attention guided feature fusion network for change detection in remote sensing images Sensors-Basel 2022 22 3 888 10.3390/s22030888 35161634
Ma, J., Shi, G., Li, Y. & Zhao, Z. Maff-net: Multi-attention guided feature fusion network for change detection in remote sensing images. Sensors-Basel 22(3), 888 (2022).35161634 10.3390/s22030888
27. Zhang J Ma P Jiang T Zhao X Tan W Zhang J Zou S Huang X Grzegorzek M Li C Sem-RCNN: A squeeze-and-excitation-based mask region convolutional neural network for multi-class environmental microorganism detection Appl. Sci. 2022 12 19 9902 10.3390/app12199902
Zhang, J. et al. Sem-RCNN: A squeeze-and-excitation-based mask region convolutional neural network for multi-class environmental microorganism detection. Appl. Sci. 12(19), 9902 (2022).10.3390/app12199902
28. Yuan L Qiu Z Mask-RCNN with spatial attention for pedestrian segmentation in cyber–physical systems Comput. Commun. 2021 180 109 114 10.1016/j.comcom.2021.09.002
Yuan, L. & Qiu, Z. Mask-RCNN with spatial attention for pedestrian segmentation in cyber–physical systems. Comput. Commun. 180, 109–114 (2021).10.1016/j.comcom.2021.09.002
29. Zhang Z Huang S Liu X Zhang B Dong D Adversarial attacks on YOLACT instance segmentation Comput. Secur. 2022 116 102682 102683 10.1016/j.cose.2022.102682
Zhang, Z., Huang, S., Liu, X., Zhang, B. & Dong, D. Adversarial attacks on YOLACT instance segmentation. Comput. Secur. 116, 102682–102683 (2022).10.1016/j.cose.2022.102682
30. Jamali A Mahdianpari M Brisco B Granger J Salehi B Comparing solo versus ensemble convolutional neural networks for wetland classification using multi-spectral satellite imagery Remote Sens. 2021 13 11 2046 10.3390/rs13112046
Jamali, A., Mahdianpari, M., Brisco, B., Granger, J. & Salehi, B. Comparing solo versus ensemble convolutional neural networks for wetland classification using multi-spectral satellite imagery. Remote Sens. 13(11), 2046 (2021).10.3390/rs13112046
31. Yu B Zhou L Wang L Yang W Yang M Bourgeat P Fripp J SA-LuT-Nets: Learning sample-adaptive intensity lookup tables for brain tumor segmentation IEEE Trans. Med. Imaging 2021 40 5 1417 1427 10.1109/TMI.2021.3056678 33534704
Yu, B. et al. SA-LuT-Nets: Learning sample-adaptive intensity lookup tables for brain tumor segmentation. IEEE Trans. Med. Imaging 40(5), 1417–1427 (2021).33534704 10.1109/TMI.2021.3056678
32. Raghavendra S Harshavardhan A Neelakandan S Partheepan R Walia R Chandra S Multilayer stacked probabilistic belief network-based brain tumor segmentation and classification Int. J. Found. Comput. Sci. 2022 33 6/7 559 582 10.1142/S0129054122420047
Raghavendra, S. et al. Multilayer stacked probabilistic belief network-based brain tumor segmentation and classification. Int. J. Found. Comput. Sci. 33(6/7), 559–582 (2022).10.1142/S0129054122420047
