
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

70750
10.1038/s41598-024-70750-7
Article
Semi-supervised recognition for artificial intelligence assisted pathology image diagnosis
Pan Yao 1
Gou Fangfang gff8221@163.com

2
Xiao Chunwen 3
Liu Jun markliu8899@163.com

3
Zhou Jing laleave@163.com

4
1 https://ror.org/03jy32q83 grid.411868.2 0000 0004 1798 0690 School of Computer Science, Jiangxi University of Traditional Chinese Medicine, Nanchang, 330004 China
2 https://ror.org/02wmsc916 grid.443382.a 0000 0004 1804 268X State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025 China
3 https://ror.org/02sqxcg48 grid.470132.3 The Second People’s Hospital of Huaihua, Huaihua, 418000 China
4 grid.67293.39 Hunan University of Medicine General Hospital, Huaihua, 418000 China
20 9 2024
20 9 2024
2024
14 2198428 4 2024
20 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
The analysis and interpretation of cytopathological images are crucial in modern medical diagnostics. However, manually locating and identifying relevant cells from the vast amount of image data can be a daunting task. This challenge is particularly pronounced in developing countries where there may be a shortage of medical expertise to handle such tasks. The challenge of acquiring large amounts of high-quality labelled data remains, many researchers have begun to use semi-supervised learning methods to learn from unlabeled data. Although current semi-supervised learning models partially solve the issue of limited labelled data, they are inefficient in exploiting unlabeled samples. To address this, we introduce a new AI-assisted semi-supervised scheme, the Reliable-Unlabeled Semi-Supervised Segmentation (RU3S) model. This model integrates the ResUNet-SE-ASPP-Attention (RSAA) model, which includes the Squeeze-and-Excitation (SE) network, Atrous Spatial Pyramid Pooling (ASPP) structure, Attention module, and ResUNet architecture. Our model leverages unlabeled data effectively, improving accuracy significantly. A novel confidence filtering strategy is introduced to make better use of unlabeled samples, addressing the scarcity of labelled data. Experimental results show a 2.0% improvement in mIoU accuracy over the current state-of-the-art semi-supervised segmentation model ST, demonstrating our approach’s effectiveness in solving this medical problem.

Subject terms

Bone cancer
Image processing
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

The analysis and interpretation of cytopathological images are critical tasks in modern medical diagnosis. However, due to the vast amount of image data, it is extremely difficult to locate and identify relevant cells manually, particularly in developing countries where there is a shortage of professional medical personnel. Therefore, improving the efficiency and accuracy of image analysis has become a significant challenge in the medical field. Computer vision is a core research field of artificial intelligence that has gained significant attention due to its applications in image processing. By analyzing image or video content, computer vision enables AI systems to comprehend environmental information. Its research areas include image classification, object detection, and semantic segmentation, providing computers with the ability to understand images from classification to pixel level1. Computer vision has great potential, particularly in image semantic segmentation. This technique explores the semantic information of each object in an image by labelling each pixel point, providing computers with a more detailed and accurate under-standing of the image. This technological advancement offers new possibilities for solving problems in medical image analysis and improving the efficiency and accuracy of medical diagnosis.

However, conventional semantic segmentation methods typically require a significant amount of labeled data, which can be challenging to obtain in many practical applications2. As a result, semi-supervised semantic segmentation has emerged as an alter-native approach. This method employs a large amount of unlabeled data and a small amount of labeled data for training, which enhances the model’s generalization ability and accuracy3. Semi-supervised semantic segmentation not only allows for the utilization of underutilized data resources but also provides new ideas for dealing with the problem of scarce labeled data. Its emergence has expanded the applications of computer vision in fields such as automatic driving, intelligent security, medical imaging, and re-mote sensing imaging. The use of semi-supervised semantic segmentation will accelerate and deepen the development of these fields, highlighting its significance in future computer vision research. Although semi-supervised semantic segmentation methods can use unlabeled data, effectively doing so remains a challenge. Current methods, such as self-training or Generative Adversarial Networks (GANs), rely on the quality and distribution of unlabeled data and require complex training strategies. Furthermore, while these methods can decrease dependence on labeled data, they typically do not achieve the same level of segmentation accuracy and speed as fully supervised semantic segmentation methods. This is primarily due to the added computational complexity required to handle the uncertainty of unlabeled data.

Challenges in medical image processing are particularly prominent. Deep neural networks have led to an increase in the use of image segmentation techniques in the medical field. These techniques are used to extract regions of interest, such as lesions, organs, or tissue structures, from medical images4. This technology is utilized in biopsy images to detect cancer, identify cancerous cells, and evaluate the size and spread of the lesion. However, acquiring precise medical image annotations is a costly and labor-intensive process, particularly for cytopathology images, which typically require annotation by pathologists or clinical professionals. Moreover, the intricate contextual structure, high-density distribution, cellular adhesion, and overlap of fully scanned section images present further challenges for manual labeling. Therefore, the effective utilization of unlabeled medical image data and the design of semi-supervised semantic segmentation methods capable of handling this data complexity are important topics of current re-search.

Developing countries encounter significant challenges in diagnosing and treating cancer. The use of deep learning models for pathology image segmentation often demands substantial computational resources and time, which is a significant obstacle for developing countries with limited resources5. Additionally, obtaining sufficient pathology image data and accurately labeling it can be very challenging in developing countries, primarily due to the scarcity of medical resources. The application of semi-supervised models for pathology image data requires a large amount of unlabeled data and a small amount of labeled data. To achieve satisfactory results, the model must be able to fully utilize the unlabeled data. The complexity and diversity of pathology image data make this a challenging task6,7. Furthermore, conventional fully supervised segmentation methods struggle to extract the intricate features present in pathology images. On the other hand, deep learning-based techniques heavily rely on high-quality labeled datasets. Thus, the effective utilization of unlabeled data to enhance the efficiency of cancer diagnosis is a crucial issue that must be addressed when implementing deep learning for cancer diagnosis in developing countries.

To tackle the challenges of segmenting cellular pathology images, we propose and implement an improved network called RSAA(ResUNet-SE-ASPP-Attention), based on ResUNet. RSAA significantly enhances segmentation accuracy by integrating Squeeze and Excitation (SE), Atrous Spatial Pyramid Pooling (ASPP), and Attention modules, while also addressing the issue of category imbalance8. In this paper, we propose a semi-supervised image semantic segmentation method called RU3S. This method effectively utilizes unlabeled data and reduces the dependence on labeled data. The proposed method can alleviate the pressure faced by developing countries in cancer diagnosis due to the shortage of medical resources. It provides a more economical and effective option for the segmentation and diagnosis of cytopathology images. The main contributions of this paper are: We propose a novel ResUNet architecture, named RSAA, integrating SE, ASPP, and Attention modules. This method is specialized for the problem of osteosarcoma image segmentation and achieves a significant improvement in segmentation accuracy compared to existing methods;

We design a semi-supervised learning method called RU3S (Reliable-Unlabeled Semi-Supervised Segmentation), which utilizes a confidence screening strategy for unlabeled samples. This addresses the issue of unlabeled data being underutilized. RU3S is particularly suitable for developing countries with limited medical resources as it only requires a small number of labeled samples to achieve efficient pathology image segmentation;

We further extend the RSAA deep learning architecture by applying it to the automatic segmentation of osteosarcoma for CRF. This has resulted in improved performance. The effectiveness of applying CRF to our model is supported by both quantitative and qualitative results;

The thesis is structured as follows: Part II describes the relevant studies and provides an in-depth explanation of the motivation for the proposed methodology; Part III demonstrates the proposed research methodology in detail; and Part IV provides the experimental dataset and results to demonstrate the effectiveness of the methodology. Finally, Part V summarizes the main findings of this study and draws corresponding conclusions.

Related work

Pathology image analysis

Digital pathology images, particularly whole-mount images, are obtained by sectioning a tissue sample and scanning it with a high-resolution scanner9,10. These images have extremely high resolution and provide a wealth of information about cell and tissue structure, and therefore have a wide range of applications in pathological diagnosis, biological research, and drug discovery. In particular, these images play a key role in the diagnosis of osteosarcoma, a common malignant bone tumor11. When faced with high-resolution digital pathology images, physicians are challenged by the amount of detailed information to process.

The application of deep learning and artificial intelligence technologies has enabled the successful detection and prediction of various types of cancer. These models can automatically detect and segment cells, recognize cell types, and predict the disease prognosis. Computer-aided image analysis, particularly deep learning algorithms, can enhance efficiency and improve diagnostic accuracy.

MRI image segmentation

In pathology image analysis, the objective of semantic segmentation tasks is to identify and locate cells, tissue structures, and disease-related regions at the pixel level12. Traditional image segmentation methods, such as threshold segmentation, edge detection, and region growing, do not perform well when dealing with complex pathology images. This is because they rely on hand-designed features and rules, which require a lot of manual intervention and parameter tuning. Deep learning methods, can automatically extract features without manual design. Therefore, they have become the mainstream methods in this field13–15. VGGNet16 and ResNet17 extract features automatically through deep convolutional networks, removing the limitation of traditional methods that require manual feature design. U-Net18 and U-Net++19 improve segmentation accuracy by down-sampling and up-sampling the image, allowing the model to capture con-textual information of the image. SE-Net20 introduces an attention mechanism to dynamically adjust the importance of different features, enabling the model to focus on more useful information. PSPNet21 and UPerNet22 perform feature extraction at different scales, allowing the model to adapt to the diversity of images. These methods have significantly improved the accuracy and efficiency of segmentation in the field of pathology image analysis.

Despite significant progress in the semantic segmentation of pathology images using deep learning, some challenges remain. One such challenge is the issue of processing category imbalance, which is particularly prominent in cytopathology images. In these images, the target of interest, such as diseased cells, typically occupies only a small portion of the image, while the background, including healthy cells and other non-cellular structures, may occupy a large portion of the image. This imbalance in categories may result in a bias towards predicting the background, leading to the neglect of important target categories. To address these issues, we propose an optimized RSAA model that incorporates the SE module, the ASPP module, and the Attention module. To address these issues, we propose an optimized RSAA model that incorporates the SE module, the ASPP module, and the Attention module. To address these issues, we propose an optimized RSAA model that incorporates the SE module, the ASPP module, and the Attention module. This model enables more accurate recognition and distinction of various categories in an image, particularly when the target categories are sparse. This study aims to enhance the precision and efficiency of pathology image segmentation using an optimized model, particularly when addressing the issue of category imbalance.

Semi-supervised segmentation model

In pathology image analysis, fully supervised learning models require accurate pixel-level annotation, which can be time-consuming and require expert knowledge23,24. Furthermore, there may be discrepancies in labeling between annotators, which can impact model training and evaluation. These limitations make it challenging to implement fully supervised models on a large scale in practical applications. Semi-supervised learning models have become increasingly common as they use a small amount of labeled data and a large amount of unlabeled data. Research on semi-supervised semantic segmentation models is particularly active in the field of medical images. SGAN25 allows generated fake samples to approximate the distribution of real samples through adversarial training. Mean Teacher26 and MixMatch27 improve the model’s generalization ability by constraining the prediction results of different transformed versions of the same sample. Self-Training28 and Student–Teacher29 effectively use unlabeled data by employing the model’s own predictions for training. SimCLR30 and MoCo31 improve the model’s feature representation by enabling it to differentiate between various samples and transformations of the same sample.

Existing semi-supervised learning models often treat each unlabeled sample equally and utilize them in the same way, without considering the reliability and difficulty of each sample32. This can lead to incorrect predictions for complex samples, which can negatively impact the training process. To tackle this issue, we present RU3S, an enhanced self-training algorithm for semi-supervised segmentation. The main concept behind RU3S is to improve the utilization of information in unlabeled samples by computing their reliability through correlation. We employ a pre-trained model to predict the unlabeled samples and assess their reliability using a prediction confidence strategy. The model is initially trained using high-reliability and labeled samples to create accurate predictive models at an early stage. The gradual addition of low-reliability samples to the training set occurs only after the model accurately predicts the high-reliability samples to avoid noise interference caused by training on difficult samples in the early stages.

Post-processing module

In semantic segmentation models, post-processing modules are crucial for optimizing and correcting the initial predictions of the model. This is achieved by eliminating small and isolated regions in the predictions, filling in small voids, or improving the boundary accuracy. These functions are usually performed through morphological operations, such as expansion and erosion, as well as open and close operations to improve the shape and continuity of the prediction results. However, current semi-supervised learning models primarily focus on utilizing both labeled and unlabeled data to improve performance, without giving sufficient consideration to the spatial relationships between pixels. Recent research33 has shown that incorporating spatial relationships between pixels can significantly enhance the accuracy of image semantic understanding.

Therefore, to better utilize the relationships between pixels, we added a CRF post-processing module to our semi-supervised learning model. The CRF module effectively captures the spatial dependencies between pixels and enhances the model’s accuracy in recognizing and segmenting region boundaries. The CRF module enables the model to consider spatial location information between pixels by establishing structured relationships between them. This leads to a better understanding of the global consistency of the image34–36. Additionally, CRF strengthens the model’s ability to handle boundary regions, resulting in more accurate and coherent segmentation results by optimizing the label assignment of each pixel. These enhancements allow our semi-supervised learning model to more effectively utilize the connections between pixels, resulting in improved segmentation accuracy.

System model design

Technological advances are improving medical treatment. Gene sequencing, medical imaging, and artificial intelligence have significantly enhanced medical diagnosis, allowing for early detection and precise treatment. Technological advances have expanded medical diagnostic possibilities. However, in many developing countries, doctors face challenges in diagnosis due to limited medical resources and high cancer diagnosis workloads. Early detection and precise diagnosis are crucial for cancer-related diseases, making efficient diagnoses challenging for doctors. Therefore, we suggest using computer vision techniques to screen important information, reducing physicians’ workload and improving diagnostic efficiency. This has significant implications for automated cancer diagnosis in developing countries. Figure 1 illustrates the architecture of our proposed image segmentation model.Fig. 1 Overall program architecture diagram.

First, we optimize the cytopathology images using various image enhancement techniques. Then, we apply the RSAA model for semantic segmentation to generate predicted classifications for each pixel. This improves segmentation accuracy and solves the category imbalance problem. Next, we give priority to using reliable unlabeled data alongside labeled data for training. This is based on the reliability screening strategy for unlabeled data in the RU3S model, which helps to reduce the interference of complex images during early training while also saving image annotation resources. Finally, we use CRF for post-processing to further improve segmentation accuracy and consistency.

Image enhancement module

Advanced image processing and machine learning techniques can extract useful features from pathology images to assist doctors in making more accurate diagnoses and save valuable medical resources. In developing countries where resources are scarce, automated image processing technology can effectively reduce doctors’ workload, allowing them to focus on more complex diagnostic tasks. However, many developing countries lack the technical and financial support required to collect and process large amounts of medical image data37–39. Additionally, due to a shortage of medical facilities and specialized personnel, it may not be possible to carry out large-scale pathology image acquisition and annotation in these countries.

In this study, we employed a series of preprocessing and data augmentation techniques to expand the dataset and enhance the model’s robustness. Figure 2 displays the flowchart of our data augmentation process. Initially, we simulated various angular and size differences, as well as orientation changes, that are possible in pathology sections under the microscope by rotating, translating, cropping, and flipping the images. These operations increase both the amount and diversity of data, improving the model’s generalization ability. Additionally, we adjusted the brightness and saturation of the images to simulate variations in pathology sections caused by different lighting and staining conditions. This approach enhances color distortion and expands our dataset, enabling the model to adapt to the various color and contrast variations that may be encountered in real-world applications. Furthermore, we employed the Generative Adversarial Network (GAN) for image enhancement to the augmented images, thereby enhancing the quality of the images. In this process, the generator receives low-quality images and then generates high-quality images. The discriminator is then required to assess whether the generated image is an accurate representation of the original, high-quality image. Consequently, the generator is able to discern how to enhance the quality of the image in the context of a confrontation with the discriminator. These preprocessing and data enhancement strategies address the challenge of acquiring accurately labeled pathology images and avoid the over-fitting problem due to insufficient data. This improves the performance and robustness of our model in processing high-resolution pathology images.Fig. 2 Data enhancement flowchart.

Image enhancement allows for the generation of numerous new images from a limited set of originals, thereby expanding the dataset. This not only improves the model’s generalization ability by increasing its training data but also simulates various changes in real-world environments, enabling the model to better handle diverse situations in practical applications. This approach effectively addresses the challenge of accessing large, high-quality annotated datasets in developing countries due to resource and capacity constraints. It also enables the use of deep learning techniques in the task of semantic segmentation of cancer cytopathology images.

Image segmentation module

A significant challenge in working with pathology images, particularly in image segmentation, is the issue of category imbalance. Conventional models tend to be biased toward predicting background categories during training due to the higher number of background pixels compared to target pixels (such as cancer cells). This results in the target categories being ignored. This is particularly relevant when the target categories are infrequent compared to the background categories, which presents a significant obstacle to enhancing the accuracy and resilience of segmentation models.

To tackle this issue, we introduce a novel deep neural network architecture named RSAA. The aim is to create a model that can more precisely identify and distinguish between different categories in an image, particularly when the target categories are less frequent than the background categories. The RSAA model is based on ResUNet ,SE, ASPP, and Attention modules, which are combined to form a strong network structure. This design enables our RSAA model to extract and utilize multi-scale and global information from the image, while also adaptively focusing on important parts of the image. This effectively solves the problem of category imbalance in cytopathology images and improves the segmentation accuracy of the model.Fig. 3 RSAA framework diagram.

(1) Encoder

Figure 3 shows that in our approach, the input pathology image undergoes feature extraction through two 3×3 convolutional layers. The current output is then added to the original input to form a residual join. This approach improves the model’s segmentation performance by introducing an inductive bias, as opposed to using a single 4x4 convolution to achieve a quarter-sized output. To enhance the model’s efficiency, we propose the SEBlock module while improving segmentation performance.

The BN technique is commonly used to accelerate neural network training40, optimize weights, and provide slight regularization effects. Equation (1) is the basic equation of the BN method.1 x^pi=xpi-μBpσBp2+ϵ

where p represents the pathology image and i represents the corresponding pixel index within the image. x^pi is the normalized output, xpi is the input, μBp is the mean of the batch, σBp2 is the variance of the batch, and ϵ is a small constant that prevents division by zero.

The obtained output is put into the ReLU activation function and is used to introduce non-linearity in the neural network to speed up the computation. The equation of the ReLU activation function is shown in Eq. (2).2 f(xpi)=xpiifxpi≥0,0ifxpi<0.

where xpi is the output of the BN and f(xpi) is the result after the ReLU function.

In cytopathology images, osteosarcoma cells typically occupy a small portion of the image, while the majority of the area is taken up by normal cells and background. This class imbalance can cause the model to be biased towards predicting the majority class during training, thereby ignoring the minority class of osteosarcoma cells. To address this issue, SEBlock incorporates the SE (Squeeze-and-Excitation) module before the 3×3 convolution described above. The SE module adapts by learning the weights of feature channels to extract useful features more effectively. This strategy improves the model’s ability to recognize certain categories, such as osteosarcoma cells, by focusing on the most significant feature channels. As a result, the model outperforms the strategy without the SE module in terms of performance. After three SEBlock down samplings, the model will obtain feature maps of h2×w2,h4×w4,h8×w8 respectively.

The Squeeze-and-Excitation (SE) module is a mechanism that adaptively recalibrates the channel weights of convolutional features41,42. Its purpose is to enhance the model’s representation by explicitly modeling the inter-channel dependencies of input features. The SE module consists of three steps: Squeeze, Excitation, and Scale. The SE module compresses the spatial dimensions of the input features using the Global Average Pooling (GAP) operation, as shown in Eq. (3).3 zcp=1H×W∑i=1H∑j=1Wxcpij

where xcpij is the value of the input feature map at channel c and spatial location (i,j), and H and W are the height and width of the input feature map.

The global average pooling operation captures the global spatial information, and the compressed feature zcp represents the global response of channel c. The SE module then nonlinearly transforms the compressed features through two fully connected layers to obtain the weights of each channel, a step called Excitation, as shown in Eq. (4).4 scp=σ(g(zc,W))=σ(W2δ(W1zc))

where scp is the computed channel weights, σ(·) is the sigmoid activation function, δ(·) is the ReLU activation function, and W1 and W2 are the weights of the two fully connected layers. the sigmoid activation function is used to restrict the weights to the range [0,1], which allows each channel to be tuned independently. Finally, the SE module performs the recalibration of the features through a multiplication operation between the channels, this step operates as shown in Eq. (5):5 ycpij=sc×xcpij

where ycpij is the value of the output feature map in channel c and spatial position (i,j), this step implements adaptive recalibration of the input feature map.

The innovation of the SE module is that it enhances the representation of the model by learning the dependencies between channels and adaptively adjusting the weights of each channel. This mechanism can be easily inserted into any convolutional network to enhance its performance with little additional computational and parametric overhead.

(2) Bridge

At the intersection of the encoder and decoder, we introduce the Atrous Spatial Pyramid Pooling (ASPP) module as a connecting bridge. The ASPP module uses null convolution at different sampling rates to integrate multi-scale contextual information effectively while maintaining the original image size. This design enables the model to understand the complex structures and patterns in the image more deeply, thus improving the recognition accuracy of details and global context. This mechanism is highly effective when dealing with targets of varying scales and shapes, which further enhances the performance of our network in semantic segmentation tasks. This is especially true for the recognition of a few categories.

The ASPP module comprises three parallel null convolution blocks, each comprising a null convolution layer, a ReLU activation function, and a bulk normalization (BN) layer. Equation (6) is used to perform null convolution for each block.6 Yijk(t,O)=BNRCrtXijk

where Yijk(t,O) is an element of the output feature map of the t-th null convolution block, Crt denotes a null convolution operation with a null rate of rt, R(·) is the ReLU activation function, BN(·) is the batch normalization operation, and Xijk is the input feature map.

By varying the null rates, contextual information can be captured at different scales while maintaining a constant feature map size. The output feature maps of all null convolution blocks are then concatenated in the channel dimension, resulting in Eq. (7) as the total merge result of the three null convolution blocks.7 Yijk=MYijk(1,O),Yijk(2,O),Yijk(3,O)

where (Yijk(1,O),Yijk(2,O),Yijk(3,O) are the output feature maps of the three null convolutional blocks, and M(·) is the merge operation on the channel dimension.

Finally, a 1x1 convolutional layer is applied to the connected feature maps to combine the features with different void rates. Equation (8) is then used to mix the contextual information at different scales, improving the model’s recognition accuracy.8 Zijk=C1Yijk

Where Zijk is an element of the output feature map, C1 denotes a 1×1 convolution operation, and Yijk is the connected feature map.

The advantage of utilizing ASPP is that it efficiently incorporates multi-scale contextual information through null convolution with varying sampling rates, thereby enhancing the accuracy of the model in identifying both intricate and broad contexts

(3) Decoder

An Attention module was integrated into the decoder section of the ResUNet model. The module’s primary function is to focus on significant regions in the image, particularly those that contribute significantly to the classification task, thereby improving the model’s accuracy in recognizing target objects, especially for a few categories43,44. By introducing the Attention module, our model can effectively focus on critical regions of the feature map, improving overall segmentation performance.

The Attention module comprises of two parallel convolutional blocks, one for the encoder output and the other for the decoder output. The outputs of these blocks are then summed, and the attention weights are generated by a 1x1 convolutional layer. Equation (9) is used to operate the two parallel convolutional blocks.9 YijkO=PRC3BNXencO+RC3BNXdecO

where YijkO is the sum of the output feature maps and XencO and XdecO are the input feature maps of the encoder and decoder, P(·), R(·), BN(·) stand for pooling, ReLU activation function and batch normalization operations, and C3 denotes a 3×3 convolution operation.

This step integrates the information from the encoder and decoder to capture richer contextual information. Next, we apply a 1×1 convolutional layer on the summed feature maps to generate the attention weights as shown in Eq. (10).10 AijkO=RC1BNYijkO

where AijkO is the attention weight matrix. This step generates the attention weights for each location to indicate the region of attention of the model.

Finally, we weigh the input feature map of the decoder with the attention weights and Eq. (11) yields the final output. This operation implements an adaptive weighting of the decoder’s input feature map to enhance the model’s attention to critical regions.11 ZijkO=AijkO·XdecO

where ZijkO is the corresponding feature values on the feature map, an operation that implements an adaptive weighting of the decoder’s input feature map to enhance the model’s focus on key regions.

Although the RSAA segmentation model is derived from the ResUNet model, each of the additional modules possesses distinctive advantages, and their integration can further enhance the performance of the model. The SE module emphasizes crucial features and suppresses irrelevant ones by weighting the feature map with learned weights. This mechanism enhances the model’s ability to capture critical information. The ASPP module captures information at multiple scales through multi-scale null convolution, which is important for image segmentation tasks where the scales of objects and scenes can vary greatly. The Attention module helps the model to better focus on the important parts of the image, thus improving the model’s performance. The integration of these three modules confers upon the model enhanced performance and robustness in the context of complex image segmentation tasks.

Semi-supervised learning image segmentation module

Fully supervised semantic segmentation models achieve semantic label assignment at the pixel level by learning from a large number of densely labeled images. However, the main limitation of this model is the difficulty and cost of acquiring high-quality labeled datasets. The labeling process is not only time- and labor-intensive but also often requires specialized knowledge and skills, which may not be feasible in certain scenarios, such as medical image segmentation. To tackle this issue, we opted for a semi-supervised semantic segmentation model. This model has the advantage of requiring only a portion of labeled data, while most of the data can be unlabeled. This approach significantly reduces the workload and complexity of the data preparation phase, while maintaining the model’s performance and reducing its dependence on labeled data. As a result, the model be-comes more useful and scalable.

However, the primary challenge in semi-supervised learning is how to effectively utilize unlabeled images. Classical self-training frameworks attempt to use all unlabeled images simultaneously, but this approach is problematic. Specifically, different unlabeled images may vary in difficulty, and thus the reliability of the generated pseudo-labels can vary, leading to serious confirmation bias45,46. Pseudo-labels that are incorrect can accumulate during iterations, causing the model to overfit to the wrong supervisory signals. This can significantly degrade the model’s performance.

To address the aforementioned issues, we suggest a novel semi-supervised learning model called RU3S. This model incorporates a confidence screening approach for unlabeled images, building upon the traditional semi-supervised learning model. Figure 4 displays the specific framework diagram; we give priority to unlabeled samples with high confidence for training by screening the confidence of unlabeled images. This strategy enables more efficient utilization of unlabeled samples while avoiding degradation of model performance caused by noise interference from complex samples. It has been demonstrated that the similarity between the pseudo-masks generated during model training can be employed to assess the stability of unlabeled samples. Samples with good stability are capable of optimizing model training. Consequently, mIoU is employed as a metric for gauging the reliability of unlabeled samples and the stability of model training. Furthermore, the utilization of mIoU as a screening criterion offers the additional advantage of enabling the dynamic reflection of the performance changes of the model throughout the training process. As the model is optimized, changes in the mIoU value can inform the selection strategy for samples, guiding the model towards the handling of increasingly complex data and further improvement in overall performance.Fig. 4 RU3S framework diagram.

In our RU3S model training process, we follow the following three stages: For the initial training, we use all unlabeled samples to train the model. Once the training is complete, we apply our proposed confidence screening strategy to unlabeled images to determine their confidence levels. Based on the results, we classify the unlabeled samples as either high or low confidence. In the second stage, we use the labeled samples along with the filtered high-confidence unlabeled samples to further train the model. This step aims to improve the model’s generalization ability using high-confidence unlabeled samples. In the final stage, we use all labeled and unlabeled samples, including both high and low-confidence samples, for the final training of the model. This phase aims to enhance the model’s performance, enabling it to handle a wider range of sample types.

In our proposed semi-supervised learning model, a confidence policy for unlabeled samples is designed. Given n unlabeled samples, the model operates at k different time points during training. For each unlabeled sample ui denoted by i(i=1,2,...,n), the model saves the generated data at each time point tj (j=1,2,...,K) saves the generated pseudo-label, denoted as Pij.

At the end of the training, the final generated pseudo-label of each unlabeled sample is saved, denoted as PiK of the i-th unlabeled sample. subsequently, the pseudo-label of the unlabeled sample ui at the time point tj with the final time point mIoU(i,j) is calculated as shown in Eq. (12), m(·) denotes the mIoU computed for the sample.12 mIoUij=mPij,PiK

These differences are then aggregated at all time points to obtain the total difference mIoUi for the i-th unlabeled sample, as shown in Eq. (13).13 mIoUi=∑j=1kmIoUij

Finally, the model ranks the unlabeled samples according to their mIoUi values. The first r unlabeled samples with high mIoUi values are regarded as high confidence samples, and the rest are classified as low confidence samples. By prioritizing the learning of these high-confidence samples, the model is able to circumvent the problem of label error accumulation caused by difficult samples, thereby extracting useful information with greater efficiency. This strategy not only improves the learning efficiency of the model, but also enhances the model’s ability to generalize to unlabeled data. The process of semi-supervised learning has been described in detail. In contrast, pseudo-labels are generated for low-confidence samples, which are then employed in the subsequent retraining process.

Using low confidence samples can still be helpful in model training. We can improve the model by training it with high confidence samples and then relabeling the low confidence samples using the trained model. This not only increases the amount of training data and learning opportunities, but also allows the model to handle difficult samples, improving robustness and generalization to more challenging real-world situations. In addition, adding low confidence unlabeled samples also reduces the cost of labelling and reduces the need for additional labelled samples. The training framework is shown in Algorithm 1.Algorithm 1 RU3S semi-supervised modeling algorithm

The core concepts and main processes of the RU3S model have been elaborated. Following this, we will compare the computational complexity and memory usage of RU3S with traditional semi-supervised models, assuming that the input feature maps are presented in the form of h×w×d, where d denotes the total number of channels, h and w denote the height and width of the feature maps, and the number of labeled samples is M, and the number of unlabeled samples is N.

The time complexity of the RU3S model is mainly determined by the components of training the model, generating pseudo-labels, calculating the mIoU score, and sorting. Overall, it is O((M+N)hwd+Nhw+NlogN). Although this time complexity may be higher than that of some semi-supervised algorithms (e.g., some self-training semi-supervised algorithms may only require O((M+N)hwd), the advantage of the RU3S algorithm lies in its efficient strategy for selecting and utilizing unlabeled samples. The RU3S algorithm may outperform some traditional semi-supervised algorithms in terms of actual running time by selecting high-confidence samples for training.

Regarding memory usage, the RU3S model’s requirements primarily stem from storing the labeled and unlabeled sample sets, parameters of the teacher and student models, and generated pseudo-labels. The space complexity of the RU3S model is O((M+N)hwd+P), where P represents the number of model parameters. The general self-training semi-supervised algorithm typically has a space complexity of O((M+N)hwd+P), because it needs to store all labeled and unlabeled samples, as well as the model’s parameters. However, the RU3S model divides the unlabeled samples into high-confidence and low-confidence sets, allowing for more efficient use of memory resources. This approach may be more effective for large-scale datasets.

Post-processing module

In the field of image semantic segmentation, existing models do not fully utilize inter-pixel relationships. This is particularly noticeable in medical image analysis. For in-stance, in MRI scans of osteosarcoma, the grayscale values of the tissue edema area, muscle area, and tumor area are similar at the boundaries. This similarity may cause difficulties for physicians in judging the disease condition, especially for less experienced physicians. Furthermore, these indistinct boundaries also affect the efficiency of image segmentation.

Simultaneously, it is observed that the likelihood of a pixel being assigned to a particular class is closely linked to the distribution of information among the surrounding pixels47,48. However, neural networks do not consider this directly during image segmentation. As a result, we opted to use a Conditional Random Field (CRF) to post-process our segmentation model. The Conditional Random Fields (CRF) algorithm addresses the issue of inadequate spatial relationships between pixels by introducing inter-pixel dependencies. This results in more consistent labeling of neighboring pixels, reducing misclassified pixels, particularly at the boundaries of differently organized regions. As a result, our approach improves overall segmentation accuracy while maintaining continuity at the boundaries.

In the final output section of the model, a fully connected Conditional Random Field (CRF) module is introduced to enhance the segmentation performance of the model. This is particularly useful when dealing with details and edge regions in an image, as the CRF can consider the spatial relationship between pixels, resulting in more accurate pixel-level classification. By integrating the CRF module, our model can efficiently process image details and edge regions, resulting in improved segmentation performance.

The CRF module comprises two primary stages: defining and minimizing the energy function. First, we define an energy function, Eq. (14) for describing the consistency be-tween pixel labels.14 E(X)=∑iθi(xi)+∑i,jθij(xi,xj)

where E(X) is the energy function, X is the label of the pixel, xi is the label of pixel i, θi(xi) is the unitary potential function of pixel i, and θij(xi,xj) is the binary potential function between pixels i and j.

The purpose of this step is to define an energy function that describes the consistency between pixel labels. Then, we obtain the optimal pixel labels by minimizing the energy function, as shown in Eq. (15).15 X∗=argminXE(X)

where X∗ is the optimal pixel label. The purpose of this step is to find a pixel label that minimizes the energy function, i.e., maximizes the consistency between pixel labels.

By introducing the CRF module, we enable the model to take into account the spatial relationship between pixels for more accurate pixel-level classification.

Experiments and results

This section describes the experiments, including the datasets used, data prepossessing, evaluation metrics, implementation process, and experimental results. The performance of our RU3S model is compared with other semi-supervised learning models, demonstrating the accuracy and effectiveness of the model.

Authors should discuss the results and how they can be interpreted from the perspective of previous studies and of the working hypotheses. The findings and their implications should be discussed in the broadest context possible. Future research directions may also be highlighted.

Dataset

To evaluate the RU3S, this paper’s dataset comprises 1000 pathology images provided by Huaihua Second People’s Hospital. Screenshots of random areas were taken at 40 times magnification, and 10 sub-images were then intercepted for each pathology image, resulting in a total of 10,000 images. After filtering, 8040 images were deemed suitable for training, with 6440 images used as the training set and the remaining 1600 images as the test set.

Evaluation indicators

Although the segmentation of cell nuclei is an example segmentation task, the criteria of semantic segmentation are predominantly employed in the paper due to the fact that in the majority of cases it is a single-category segmentation and the image is highly specific. This is particularly evident in this paper, where it is a single-category segmentation of osteosarcoma cells. Consequently, the evaluation metrics employed in this study are mean intersection over union (mIoU), precision, recall, F-value and Dice coefficient. These metrics facilitate the quantification of the model’s performance from diverse vantages. The metric used to quantify the degree of overlap between the predicted tumor region (I1) and the actual tumor region (I2) is Intersection over Union (IoU). It is calculated by dividing the area of intersection between I1 and I2 by the area of their union. IoU has a value between 0 and 1, the closer the value is to 1, the more accurate the prediction. The formula is presented in equation Eq. (16):16 IoU=I1∩I2I1∪I2

To solve the multi-class segmentation problem, it is necessary to calculate the IoU for each class and then determine the average value, known as mIoU. The formula for mIoU is presented in Eq. (17):17 mIoUO=1N∑i=1NIoUiO

where N is the number of categories, IoUiO is the IoU of osteosarcoma cells of the ith category, and mIoUO is the mean value of the IoU of pathological cells of different categories.

mIoU can be interpreted as a measure of the degree of similarity between the predicted tumor region and the real tumor region in our tumor segmentation task. A high mIoU value means that our model can predict the location and shape of the tumor more accurately, which can better serve the subsequent clinical decisions.

The precision rate is defined as the proportion of the intersection between the predicted positive category regions and the actual positive category regions to the predicted positive category regions. The precision rate can be employed to assess the accuracy of the model prediction. The calculation formula is presented in Eq. (18):18 Precision=TPTP+FP

Recall is defined as the proportion of the intersection between the predicted positive category regions and the actual positive category regions to the actual positive category regions. The recall rate can be employed to assess the extent to which the model encompasses the positive categories. The calculation formula is presented in Eq. (19):19 Recall=TPTP+FN

The dice coefficient is defined as the ratio of the intersection of the predicted positive category region and the actual positive category region, expressed as a proportion of the sum of the predicted positive category region and the actual positive category region. The value of the dice coefficient can be used to assess the segmentation effectiveness of a model, with a larger value indicating a more effective segmentation. The calculation formula is presented in Eq. (20):20 DSC=2∗TPTP+FN+2∗TP

The F-value represents the reconciled average of precision and recall, thereby enabling a comprehensive reflection of the precision and coverage of the model. The calculation formula is presented in Eq. (21):21 F1=2∗Pre∗RePre+Re

Implementation details

For our experiments, we trained our model on a computer equipped with an NVIDIA 4090 GPU. The operating system used was Windows 11, and we employed PyTorch 2.2.1 with Python version 3.9 as our deep learning framework. To enhance the generalization ability of our models, we utilized various data augmentation techniques during the training process. The hyper parameters for both the teacher and student models were chosen to be the same. This includes a batch size of 16, an initial learning rate of 0.001, a momentum of 0.9, and a decay rate of 0.0001. Stochastic gradient descent (SGD) was used as the optimizer and the training process was set to 100 epochs. Additionally, the time node K was set to 3. To improve the accuracy and stability of the segmentation results, Conditional Random Fields (CRF) were used for post-processing on top of the model output. This post-processing method effectively eliminates noise in the segmentation results and enhances their accuracy.

Comparison experiment

Our study aims to demonstrate the effectiveness of the newly proposed RU3S method. We conducted a series of comparative experiments to provide a detailed analysis of the model’s performance.Fig. 5 Comparison of model final segmentation qualitative results.

Fig. 6 Comparison of final segmentation qualitative results of different models.

Fig. 7 Performance of different models on different scaled labeled data.

Firstly, we selected established semi-supervised learning models as benchmarks to compare with the RU3S method. This will provide a clearer understanding of RU3S’s performance in various tasks and identify its strengths and weaknesses relative to other methods. To ensure a fair comparison, we used the same dataset and experimental setup. Figure 5 shows the experimental segmentation results. Difference maps use blue, green, and red regions to represent true-positive splits, false-positive splits, and false-negative splits, respectively. These results demonstrate the exceptional performance of our model in handling challenging image tasks, particularly in accurately segmenting images with multiple cells and recognizing cells in images where they overlap. These tasks often require high levels of performance and accuracy from the model. These tasks typically require high levels of performance and accuracy from the model. Therefore, the exceptional performance of our model on these tasks demonstrates its strong utility and potential for a wide range of applications.

Figures 6 and 7 present a comparison of our model with other semi-supervised learning models from qualitative and quantitative perspectives. Figure 6 displays the model’s performance in real tasks. Our method provides comparable results to those of the prior art for simple segmentation tasks, as shown in Fig. 6a–d. Nevertheless, other models frequently encounter numerous difficulties when confronted with these intricate images. Conversely, they may be unable to accurately identify overlapping cell nuclei, which would result in a lower accuracy of the segmentation results. Conversely, they may also blur when identifying boundaries, thereby reducing the clarity of the segmentation results. In contrast, our model demonstrates significant advantages in dealing with these complex situations. The model is capable of accurately identifying and segmenting overlapping nuclei while maintaining the clarity of the boundaries. This is clearly demonstrated in Fig. 6e,f, where the blue region indicating true-positive segmentation occupies the majority of the area, while the green color indicating false-positive segmentation and the red color indicating false-negative segmentation are fewer compared to other models. This indicates that our model continues to demonstrate high stability and robustness when confronted with complex and variable image scenes. While Fig. 6 shows the scores of the model on various performance metrics. The data in Fig. 7 indicates that the RU3S model outperforms the other models in terms of mIoU scores for all label scales. The RU3S model demonstrates consistent superior performance across tasks of varying complexity, regardless of the labeling scales. This is a crucial property as practical applications may encounter label-sparse or label-rich situations. The figures clearly show that the RU3S model outperforms other existing semi-supervised learning models trained on the osteosarcoma dataset, both qualitatively and quantitatively. The experimental results indicate that our proposed model is more effective in handling complex cells than traditional methods and has certain advantages over existing models.

Table 1 lists the mIoU scores of various methods, including our RU3S model and its different variants, at different labeling ratios. To ensure a fair comparison, we selected several widely recognized semi-supervised learning models in this field, including AdvSemiSeg49, CAC50, ST51, DMT52, and ReCo53. Additionally, we conducted several comparative experiments based on our RU3S model, including the implementation of data augmentation (DA) techniques, the use of the RSAA model, and the inclusion of the CRF post-processing module. It is evident that at a labeling ratio of 1/100(65), the mIoU scores of the RU3S model are significantly higher than those of the other models even with a very small number of labels. This indicates that the model has a strong ability to learn valid information from limited labels. The mIoU scores of all models improved as the labeling ratio increased. However, the improvement of the RU3S model was not as significant as that of the other models. This could be because the RU3S model had already learned enough information with fewer labels, and thus additional labels did not result in a significant improvement. The RU3S model maintains its superior performance with sufficient labels, as demonstrated by its highest mIoU score at a labeling ratio of 1/8 (805).

Table 2 shows the results of the Pre, Re, Dice and F1 metrics under different methods. It can be seen that our RU3S model achieves excellent performance in the Pre and F1 metrics and compares favorably with the AdvSemiSeg, CAC, DMT and Reco models in all metrics. As for the ST model, although our model is slightly inferior in the DSC, Pre and F1 metrics, it still has an advantage in the Re metric. Table 1 Composition of cellular pathology images dataset.

Method	1/100(65)	1/50(129)	1/20(322)	1/8(805)	
AdvSemiSeg	68.97	73.13	74.54	74.85	
CAC	56.21	62.77	69.92	71.46	
ST	70.32	75.03	78.01	80.32	
DMT	68.76	69.59	70.87	71.24	
ReCo	63.25	70.32	73.67	74.28	
Our(RU3S)	60.39	71.86	78.28	79.59	
Our(RU3S+DA)	71.23	79.54	80.78	80.98	
Our(RU3S+DA+CRF)	72.64	81.46	81.98	82.05	
Our(RU3S+DA+CRF+RSAA)	73.59	81.74	82.15	82.33	

Table 2 Performance of different models on different metrics.

Method	Pre	Re	DSC	F1	
AdvSemiSeg	82.11	86.13	82.57	84.07	
CAC	77.82	84.26	80.54	80.91	
ST	86.24	89.45	89.34	87.82	
DMT	77.37	84.82	81.23	80.92	
ReCo	79.80	87.40	83.07	83.43	
Our(RU3S)	85.46	89.50	87.31	87.43	
Our(RU3S+DA)	86.71	89.80	88.18	88.23	
Our(RU3S+DA+CRF)	87.86	90.12	88.95	88.98	
Our(RU3S+DA+CRF+RSAA)	88.74	89.85	89.29	89.29	

In conclusion, based on the above quantitative and qualitative analyses, our RU3S model outperforms traditional models in terms of evaluation metrics, while our RU3S model still contains suboptimal parameters compared to existing novel semi-supervised segmentation models. Therefore, in the combined situation, our RU3S model can achieve a good application situation in relevant cytopathological image segmentation scenarios.

Ablation experiment

Our study’s primary contribution is the screening strategy for unlabeled samples. This innovation not only enhances our model’s performance but also offers new directions for researchers in the field of semi-supervised learning to explore. In this section, we provide a detailed examination of the practical effectiveness of the Data Augmentation (DA), RSAA, and CRF components. These components are crucial to our model and enhance its performance by providing additional training samples, improving the model structure, and implementing more accurate post-processing methods. We will demonstrate the effectiveness of these components in real-world applications through a series of experiments and analyses, and show to what extent they improve our model’s performance.Fig. 8 Comparison between the original model and segmentation results using different modules.

Fig. 9 Comparison between the original model and segmentation results using different modules.

Figures 8 and 9 demonstrate the performance improvement of our model after a series of data enhancement operations, both qualitatively and quantitatively. Our model shows a significant improvement in the key performance metric mIoU compared to the original results after these operations. The results indicate that performing these types of data enhancements can significantly improve the performance of segmentation models. Data augmentation is a simple yet effective method that can enhance a model’s generalization ability and its capacity to handle complex tasks by introducing more training samples and richer sample variations.

Additionally, our experimental results demonstrate that incorporating the CRF component further enhances the performance of our model. This could be attributed to the CRF component’s ability to consider a greater amount of pixel information from the surrounding area, allowing the model to better comprehend the contextual information of the image and ultimately improving the accuracy and stability of the segmentation. Finally, our experiments demonstrate that incorporating RSAA into our segmentation model can enhance its performance. The introduction of RSAA enables the model to utilize spatial information more efficiently, improving its sensitivity to image details and ultimately enhancing segmentation accuracy.As the proportion of labels increases, the mIoU scores of all models improve. It is suggested that additional labels can provide more information and improve the model’s understanding of the data. However, it is important to note that the improvement in mIoU gradually decreases as the proportion of labels increases. This may be due to the fact that the model has already learned most of the information from the existing labels, and additional labels do not provide significant additional benefits. Table 3 Quantitative analysis of modeled high-confidence sample proportions.

Proportion of highly reliable images	1/4	1/2	3/4	
mIoU (%)	82.04	82.33	82.41	

Fig. 10 Comparison between the original model and segmentation results using different modules.

Fig. 11 Comparison of results using low confidence samples or not.

Table 4 Quantitative analysis of high-confidence sample screening strategies.

Methods	1/100(65)	1/50(129)	1/20(322)	1/8(805)	
One-stage re-training	70.32	75.03	78.01	80.32	
Two-stage re-training	70.98	75.86	77.82	80.41	
Ours	73.59	81.74	82.15	82.33	

An ablation study was conducted using a high-confidence sample proportion. Table 3 shows that good results can be achieved by setting the high-confidence sample proportion to half. Although increasing the ratio to 3/4 would improve performance, it would also increase the model’s complexity. Therefore, we chose to balance performance and complexity by selecting half of the high-confidence sample proportion.

Figure 10 demonstrates the optimal performance of our model when handling high and low-confidence sample sets. The results indicate that the filtered high-confidence samples can significantly enhance the model’s accuracy and effectively utilize the unlabeled samples while avoiding the noise interference caused by complex samples.

Figure 11 shows the best results of our model when comparing whether or not to use a low confidence sample set. The results show that model performance enhancement can be achieved by relabeling unlabeled samples after the model has been trained with high confidence samples. This approach makes effective use of unlabeled samples while avoiding noise interference caused by complex samples. In this way, we can make better use of the low confidence samples, give full play to the unlabeled samples, and improve the performance and generalization ability of the model.

To further validate the effectiveness of the high-confidence sample screening strategy, we compared it with a multi-stage retraining method that does not apply the strategy. The relevant results are demonstrated in Table 4. The results indicate that the semi-supervised model does not benefit from regular multi-stage training, and sometimes its performance is even worse than single-stage training. This finding further confirms the usefulness of our high-confidence screening strategy for semi-supervised models.

In summary, the proposed methods, namely RU3S+DA, RU3S+DA+CRF, and RU3S+DA+CRF+RSAA, outperform RU3S. These results validate the effectiveness of DA, CRF, RSAA, and their combinations in enhancing the performance of RU3S. The proposed unlabeled sample screening strategy is a novel and effective approach to utilizing unlabeled data more efficiently. This strategy improves the learning efficiency and performance of the model by screening out the most valuable samples for model training. The efficacy of this approach has been confirmed not only in our experiments but also in numerous practical applications. The system reduces dependence on labeling resources and ensures high segmentation accuracy through three modules: data enhancement, semi-supervised semantic segmentation, and post-processing. The proposed unlabeled sample screening strategy is an effective approach to utilizing unlabeled data more efficiently. This strategy enhances the efficiency and performance of the learning model by filtering out the most valuable samples for training. Its effectiveness has been verified not only in our experiments but also in many practical applications.

However, it is important to acknowledge the limitations of the model. Specifically, the model may not accurately detect all cells, indicating that there is still room for improvement in segmentation accuracy. Additionally, while we have reduced our reliance on labeled data, there is still potential for enhancing the utilization of labeled tags. Future research will focus on improving both segmentation accuracy and label utilization efficiency. We will investigate the implementation of advanced deep learning techniques and optimization algorithms, as well as more efficient utilization of unlabeled data. Additionally, we will collaborate closely with clinicians to obtain their feedback and evaluate the model’s performance in real-world medical scenarios.

Conclusion

Our research proposes a cytopathology image segmentation system that can assist physicians in recognizing cancer lesions and improving diagnostic efficiency. The method demonstrates strong robustness in handling diverse and complex cytopathology images, giving it a competitive advantage in the field of cancer recognition. This advancement improves the efficiency of medical workflow and opens up new avenues for timely and accurate cancer diagnosis.

Our main objective for future research is to enhance the model’s segmentation accuracy by investigating the implementation of advanced deep learning techniques and optimization algorithms. Additionally, we aim to utilize other semi-supervised learning methods to more efficiently use unlabeled data, allowing us to leverage the vast amount of available medical image data without the need for tedious labeling. We will also seek deep collaboration with clinicians to obtain their feedback and understand how the model per-forms in real-world medical scenarios so that we can further optimize the model to meet clinical needs.

Acknowledgements

This paper is funded by the Jiangxi University of Chinese Medicine Science and Technology Innovation Team Development Program (No. CXTD22015). Feasibility analysis on development of an individualized prognostic model of osteosarcoma S2022JJSSLH0271. Correspondence should be addressed to Fanggfang Gou.

Author contributions

Writing—original draft, Y.P.; writing—review and editing J.L., C.X., J.Z., and F.G. All authors have read and agreed to the published version of the manuscript.

Data availability

Data used to support the findings of this study are currently under embargo while the research findings are commercialized. Requests for data, 12 months after the publication of this article, will be considered by the corresponding author.

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Isensee F Jaeger PF Kohl SA Petersen J Maier-Hein KH nnu-net: a self-configuring method for deep learning-based biomedical image segmentation Nat. Methods 2021 18 203 211 10.1038/s41592-020-01008-z 33288961
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J. & Maier-Hein, K. H. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021).33288961
2. Li X H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes IEEE Trans. Med. Imaging 2018 37 2663 2674 10.1109/TMI.2018.2845918 29994201
Li, X. et al. H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE Trans. Med. Imaging 37, 2663–2674 (2018).29994201
3. Neupane B Horanont T Aryal J Deep learning-based semantic segmentation of urban features in satellite images: A review and meta-analysis Remote Sensing 2021 13 808 10.3390/rs13040808
Neupane, B., Horanont, T. & Aryal, J. Deep learning-based semantic segmentation of urban features in satellite images: A review and meta-analysis. Remote Sensing 13, 808 (2021).
4. Xie X A survey on incorporating domain knowledge into deep learning for medical image analysis Med. Image Anal. 2021 69 101985 10.1016/j.media.2021.101985 33588117
Xie, X. et al. A survey on incorporating domain knowledge into deep learning for medical image analysis. Med. Image Anal. 69, 101985 (2021).33588117
5. Lv L Zhao B Kang J Li S Wu H Trend of disease burden and risk factors of breast cancer in developing countries and territories, from 1990 to 2019: Results from the global burden of disease study 2019 Front. Public Health 2023 10 1078191 10.3389/fpubh.2022.1078191 36726635
Lv, L., Zhao, B., Kang, J., Li, S. & Wu, H. Trend of disease burden and risk factors of breast cancer in developing countries and territories, from 1990 to 2019: Results from the global burden of disease study 2019. Front. Public Health 10, 1078191 (2023).36726635
6. Wu, L. et al. Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
7. Zhang B Semi-supervised deep learning via transformation consistency regularization for remote sensing image semantic segmentation IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2022 16 5782 5796 10.1109/JSTARS.2022.3203750
Zhang, B. et al. Semi-supervised deep learning via transformation consistency regularization for remote sensing image semantic segmentation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 16, 5782–5796 (2022).
8. Liang Y Weakly supervised deep nuclei segmentation with sparsely annotated bounding boxes for DNA image cytometry IEEE/ACM Trans. Comput. Biol. Bioinform. 2021 20 785 795 10.1109/TCBB.2021.3138189
Liang, Y. et al. Weakly supervised deep nuclei segmentation with sparsely annotated bounding boxes for DNA image cytometry. IEEE/ACM Trans. Comput. Biol. Bioinform. 20, 785–795 (2021).
9. Foucart A Debeir O Decaestecker C Shortcomings and areas for improvement in digital pathology image segmentation challenges Comput. Med. Imaging Graph. 2023 103 102155 10.1016/j.compmedimag.2022.102155 36525770
Foucart, A., Debeir, O. & Decaestecker, C. Shortcomings and areas for improvement in digital pathology image segmentation challenges. Comput. Med. Imaging Graph. 103, 102155 (2023).36525770
10. Zhou X Emerging applications of deep learning in bone tumors: current advances and challenges Front. Oncol. 2022 12 908873 10.3389/fonc.2022.908873 35928860
Zhou, X. et al. Emerging applications of deep learning in bone tumors: current advances and challenges. Front. Oncol. 12, 908873 (2022).35928860
11. Wu J An artificial intelligence multiprocessing scheme for the diagnosis of osteosarcoma MRI images IEEE J. Biomed. Health Inform. 2022 26 4656 4667 10.1109/JBHI.2022.3184930 35727772
Wu, J. et al. An artificial intelligence multiprocessing scheme for the diagnosis of osteosarcoma MRI images. IEEE J. Biomed. Health Inform. 26, 4656–4667 (2022).35727772
12. Li X A comprehensive review of computer-aided whole-slide image analysis: From datasets to feature extraction, segmentation, classification and detection approaches Artif. Intell. Rev. 2022 55 4809 4878 10.1007/s10462-021-10121-0
Li, X. et al. A comprehensive review of computer-aided whole-slide image analysis: From datasets to feature extraction, segmentation, classification and detection approaches. Artif. Intell. Rev. 55, 4809–4878 (2022).
13. Wu J Research on artificial-intelligence-assisted medicine: A survey on medical artificial intelligence Diagnostics 2022 12 e1439 10.1007/s40747-024-01471-7
Wu, J. et al. Research on artificial-intelligence-assisted medicine: A survey on medical artificial intelligence. Diagnostics 14, 1472. 10.1007/s40747-024-01471-7 (2024).
14. Van der Velden BH Kuijf HJ Gilhuijs KG Viergever MA Explainable artificial intelligence (xai) in deep learning-based medical image analysis Med. Image Anal. 2022 79 102470 10.1016/j.media.2022.102470 35576821
Van der Velden, B. H., Kuijf, H. J., Gilhuijs, K. G. & Viergever, M. A. Explainable artificial intelligence (xai) in deep learning-based medical image analysis. Med. Image Anal. 79, 102470 (2022).35576821
15. Chen X Recent advances and clinical applications of deep learning in medical image analysis Med. Image Anal. 2022 79 102444 10.1016/j.media.2022.102444 35472844
Chen, X. et al. Recent advances and clinical applications of deep learning in medical image analysis. Med. Image Anal. 79, 102444 (2022).35472844
16. Kora P Transfer learning techniques for medical image analysis: A review Biocybern. Biomed. Eng. 2022 42 79 107 10.1016/j.bbe.2021.11.004
Kora, P. et al. Transfer learning techniques for medical image analysis: A review. Biocybern. Biomed. Eng. 42, 79–107 (2022).
17. Xu W Fu Y-L Zhu D Resnet and its application to medical image processing: Research progress and challenges Comput. Methods Programs Biomed. 2023 240 107660 10.1016/j.cmpb.2023.107660 37320940
Xu, W., Fu, Y.-L. & Zhu, D. Resnet and its application to medical image processing: Research progress and challenges. Comput. Methods Programs Biomed. 240, 107660 (2023).37320940
18. Yu Z Yu L Zheng W Wang S Eiu-net: Enhanced feature extraction and improved skip connections in u-net for skin lesion segmentation Comput. Biol. Med. 2023 162 107081 10.1016/j.compbiomed.2023.107081 37301097
Yu, Z., Yu, L., Zheng, W. & Wang, S. Eiu-net: Enhanced feature extraction and improved skip connections in u-net for skin lesion segmentation. Comput. Biol. Med. 162, 107081 (2023).37301097
19. Jamali A Roy SK Li J Ghamisi P Transu-net++: Rethinking attention gated transu-net for deforestation mapping Int. J. Appl. Earth Obs. Geoinf. 2023 120 103332
Jamali, A., Roy, S. K., Li, J. & Ghamisi, P. Transu-net++: Rethinking attention gated transu-net for deforestation mapping. Int. J. Appl. Earth Obs. Geoinf. 120, 103332 (2023).
20. Lv H Attention mechanism in intelligent fault diagnosis of machinery: A review of technique and application Measurement 2022 199 111594 10.1016/j.measurement.2022.111594
Lv, H. et al. Attention mechanism in intelligent fault diagnosis of machinery: A review of technique and application. Measurement 199, 111594 (2022).
21. Barrientos-Espillco F Gascó E López-González CI Gómez-Silva MJ Pajares G Semantic segmentation based on deep learning for the detection of cyanobacterial harmful algal blooms (cyanohabs) using synthetic images Appl. Soft Comput. 2023 141 110315 10.1016/j.asoc.2023.110315
Barrientos-Espillco, F., Gascó, E., López-González, C. I., Gómez-Silva, M. J. & Pajares, G. Semantic segmentation based on deep learning for the detection of cyanobacterial harmful algal blooms (cyanohabs) using synthetic images. Appl. Soft Comput. 141, 110315 (2023).
22. Zhang X Cen C Li F Liu M Mu W Crformer: Cross-resolution transformer for segmentation of grape leaf diseases with context mining Expert Syst. Appl. 2023 229 120324 10.1016/j.eswa.2023.120324
Zhang, X., Cen, C., Li, F., Liu, M. & Mu, W. Crformer: Cross-resolution transformer for segmentation of grape leaf diseases with context mining. Expert Syst. Appl. 229, 120324 (2023).
23. Santos MS On the joint-effect of class imbalance and overlap: A critical review Artif. Intell. Rev. 2022 55 6207 6275 10.1007/s10462-022-10150-3
Santos, M. S. et al. On the joint-effect of class imbalance and overlap: A critical review. Artif. Intell. Rev. 55, 6207–6275 (2022).
24. Zou Z Chen K Shi Z Guo Y Ye J Object detection in 20 years: A survey Proc. IEEE 2023 111 257 276 10.1109/JPROC.2023.3238524
Zou, Z., Chen, K., Shi, Z., Guo, Y. & Ye, J. Object detection in 20 years: A survey. Proc. IEEE 111, 257–276 (2023).
25. AlAmir M AlGhamdi M The role of generative adversarial network in medical image analysis: An in-depth survey ACM Comput. Surv. 2022 55 1 36 10.1145/3527849
AlAmir, M. & AlGhamdi, M. The role of generative adversarial network in medical image analysis: An in-depth survey. ACM Comput. Surv. 55, 1–36 (2022).
26. Xu Z Ambiguity-selective consistency regularization for mean-teacher semi-supervised medical image segmentation Med. Image Anal. 2023 88 102880 10.1016/j.media.2023.102880 37413792
Xu, Z. et al. Ambiguity-selective consistency regularization for mean-teacher semi-supervised medical image segmentation. Med. Image Anal. 88, 102880 (2023).37413792
27. Peng X Adversarial domain adaptation network with mixmatch for incipient fault diagnosis of pmsm under multiple working conditions Knowl. Based Syst. 2024 284 111331 10.1016/j.knosys.2023.111331
Peng, X. et al. Adversarial domain adaptation network with mixmatch for incipient fault diagnosis of pmsm under multiple working conditions. Knowl. Based Syst. 284, 111331 (2024).
28. Zhu Y Improving semantic segmentation via efficient self-training IEEE Trans. Pattern Anal. Mach. Intell. 2021 46 1589 1602 10.1109/TPAMI.2021.3138337
Zhu, Y. et al. Improving semantic segmentation via efficient self-training. IEEE Trans. Pattern Anal. Mach. Intell. 46, 1589–1602 (2021).
29. Li Y Yao H Xu C Test: Triplet ensemble student-teacher model for unsupervised person re-identification IEEE Trans. Image Process. 2021 30 7952 7963 10.1109/TIP.2021.3112039 34534083
Li, Y., Yao, H. & Xu, C. Test: Triplet ensemble student-teacher model for unsupervised person re-identification. IEEE Trans. Image Process. 30, 7952–7963 (2021).34534083
30. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607 (PMLR, 2020).
31. He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729–9738 (2020).
32. Qi G-J Luo J Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods IEEE Trans. Pattern Anal. Mach. Intell. 2020 44 2168 2187 10.1109/TPAMI.2020.3031898
Qi, G.-J. & Luo, J. Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods. IEEE Trans. Pattern Anal. Mach. Intell. 44, 2168–2187 (2020).
33. Li Z Sun Y Zhang L Tang J Ctnet: Context-based tandem network for semantic segmentation IEEE Trans. Pattern Anal. Mach. Intell. 2021 44 9904 9917 10.1109/TPAMI.2021.3132068
Li, Z., Sun, Y., Zhang, L. & Tang, J. Ctnet: Context-based tandem network for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 44, 9904–9917 (2021).
34. Ono D The mammalian circadian pacemaker regulates wakefulness via crf neurons in the paraventricular nucleus of the hypothalamus Sci. Adv. 2020 6 eabd0384 10.1126/sciadv.abd0384 33158870
Tang, X et al. Artificial intelligence multiprocessing scheme for pathology images based on transformer for nuclei segmentation. Complex Intell. Syst. (2024). https://doi.org/10.1007/s40747-024-01471-7.33158870
35. Li Z Li Q Zou X Ren J Causality extraction based on self-attentive bilstm-crf with transferred embeddings Neurocomputing 2021 423 207 219 10.1016/j.neucom.2020.08.078
Li, Z., Li, Q., Zou, X. & Ren, J. Causality extraction based on self-attentive bilstm-crf with transferred embeddings. Neurocomputing 423, 207–219 (2021).
36. Chen S Gamechi ZS Dubost F van Tulder G de Bruijne M An end-to-end approach to segmentation in medical images with cnn and posterior-crf Med. Image Anal. 2022 76 102311 10.1016/j.media.2021.102311 34902793
Chen, S., Gamechi, Z. S., Dubost, F., van Tulder, G. & de Bruijne, M. An end-to-end approach to segmentation in medical images with cnn and posterior-crf. Med. Image Anal. 76, 102311 (2022).34902793
37. Li C Underwater image enhancement via medium transmission-guided multi-color space embedding IEEE Trans. Image Process. 2021 30 4985 5000 10.1109/TIP.2021.3076367 33961554
Li, C. et al. Underwater image enhancement via medium transmission-guided multi-color space embedding. IEEE Trans. Image Process. 30, 4985–5000 (2021).33961554
38. Rahman T Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images Comput. Biol. Med. 2021 132 104319 10.1016/j.compbiomed.2021.104319 33799220
Rahman, T. et al. Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Comput. Biol. Med. 132, 104319 (2021).33799220
39. Littlejohns TJ The UK biobank imaging enhancement of 100,000 participants: Rationale, data collection, management and future directions Nat. Commun. 2020 11 2624 10.1038/s41467-020-15948-9 32457287
Littlejohns, T. J. et al. The UK biobank imaging enhancement of 100,000 participants: Rationale, data collection, management and future directions. Nat. Commun. 11, 2624 (2020).32457287
40. Bronskill, J., Gordon, J., Requeima, J., Nowozin, S. & Turner, R. Tasknorm: Rethinking batch normalization for meta-learning. In International Conference on Machine Learning, 1153–1164 (PMLR, 2020).
41. Jin X Delving deep into spatial pooling for squeeze-and-excitation networks Pattern Recognit. 2022 121 108159 10.1016/j.patcog.2021.108159
Jin, X. et al. Delving deep into spatial pooling for squeeze-and-excitation networks. Pattern Recognit. 121, 108159 (2022).
42. Li Y Epileptic seizure detection in eeg signals using a unified temporal-spectral squeeze-and-excitation network IEEE Trans. Neural Syst. Rehabil. Eng. 2020 28 782 794 10.1109/TNSRE.2020.2973434 32078551
Li, Y. et al. Epileptic seizure detection in eeg signals using a unified temporal-spectral squeeze-and-excitation network. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 782–794 (2020).32078551
43. Galassi A Lippi M Torroni P Attention in natural language processing IEEE Trans. Neural Netw. Learn. Syst. 2020 32 4291 4308 10.1109/TNNLS.2020.3019893
Galassi, A., Lippi, M. & Torroni, P. Attention in natural language processing. IEEE Trans. Neural Netw. Learn. Syst. 32, 4291–4308 (2020).
44. Niu Z Zhong G Yu H A review on the attention mechanism of deep learning Neurocomputing 2021 452 48 62 10.1016/j.neucom.2021.03.091
Niu, Z., Zhong, G. & Yu, H. A review on the attention mechanism of deep learning. Neurocomputing 452, 48–62 (2021).
45. Chen, X., Yuan, Y., Zeng, G. & Wang, J. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2613–2622 (2021).
46. Ouali, Y., Hudelot, C. & Tami, M. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12674–12684 (2020).
47. Zhang S-X Zhu X Chen L Hou J-B Yin X-C Arbitrary shape text detection via segmentation with probability maps IEEE Trans. Pattern Anal. Mach. Intell. 2022 45 2736 2750
Zhang, S.-X., Zhu, X., Chen, L., Hou, J.-B. & Yin, X.-C. Arbitrary shape text detection via segmentation with probability maps. IEEE Trans. Pattern Anal. Mach. Intell. 45, 2736–2750 (2022).
48. Kim W Kanezaki A Tanaka M Unsupervised learning of image segmentation based on differentiable feature clustering IEEE Trans. Image Process. 2020 29 8055 8068 10.1109/TIP.2020.3011269
Kim, W., Kanezaki, A. & Tanaka, M. Unsupervised learning of image segmentation based on differentiable feature clustering. IEEE Trans. Image Process. 29, 8055–8068 (2020).
49. Hung, W.-C., Tsai, Y.-H., Liou, Y.-T., Lin, Y.-Y. & Yang, M.-H. Adversarial learning for semi-supervised semantic segmentation. arXiv preprint arXiv:1802.07934 (2018).
50. Lai, X. et al. Semi-supervised semantic segmentation with directional context-aware consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1205–1214 (2021).
51. Wang L Yoon K-J Semi-supervised student–teacher learning for single image super-resolution Pattern Recognit. 2022 121 108206 10.1016/j.patcog.2021.108206
Wang, L. & Yoon, K.-J. Semi-supervised student–teacher learning for single image super-resolution. Pattern Recognit. 121, 108206 (2022).
52. Feng Z Dmt: Dynamic mutual training for semi-supervised learning Pattern Recognit. 2022 130 108777 10.1016/j.patcog.2022.108777
Feng, Z. et al. Dmt: Dynamic mutual training for semi-supervised learning. Pattern Recognit. 130, 108777 (2022).
53. Liu, S., Zhi, S., Johns, E. & Davison, A. J. Bootstrapping semantic segmentation with regional contrast. arXiv preprint arXiv:2104.04465 (2021).
