
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39245695
71650
10.1038/s41598-024-71650-6
Article
Prior-guided attention fusion transformer for multi-lesion segmentation of diabetic retinopathy
Xu Chenfangqian 12
Guo Xiaoxin guoxx@jlu.edu.cn

12
Yang Guangqi 12
Cui Yihao 3
Su Longchen 12
Dong Hongliang 2
Hu Xiaoying 4
Che Songtian 5
1 https://ror.org/00js3aw79 grid.64924.3d 0000 0004 1760 5735 Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012 China
2 https://ror.org/00js3aw79 grid.64924.3d 0000 0004 1760 5735 College of Computer Science and Technology, Jilin University, Changchun, 130012 China
3 https://ror.org/00js3aw79 grid.64924.3d 0000 0004 1760 5735 College of Software, Jilin University, Changchun, 130012 China
4 https://ror.org/034haf133 grid.430605.4 0000 0004 1758 4110 Ophthalmology Department, Bethune First Hospital of Jilin University, Changchun, 130021 China
5 https://ror.org/01hht6e57 grid.452635.3 Ophthalmology Department, Bethune Second Hospital of Jilin University, Changchun, 130041 China
8 9 2024
8 9 2024
2024
14 2089230 3 2024
29 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
To solve the issue of diagnosis accuracy of diabetic retinopathy (DR) and reduce the workload of ophthalmologists, in this paper we propose a prior-guided attention fusion Transformer for multi-lesion segmentation of DR. An attention fusion module is proposed to improve the key generator to integrate self-attention and cross-attention and reduce the introduction of noise. The self-attention focuses on lesions themselves, capturing the correlation of lesions at a global scale, while the cross-attention, using pre-trained vessel masks as prior knowledge, utilizes the correlation between lesions and vessels to reduce the ambiguity of lesion detection caused by complex fundus structures. A shift block is introduced to expand association areas between lesions and vessels further and to enhance the sensitivity of the model to small-scale structures. To dynamically adjust the model’s perception of features at different scales, we propose the scale-adaptive attention to adaptively learn fusion weights of feature maps at different scales in the decoder, capturing features and details more effectively. The experimental results on two public datasets (DDR and IDRiD) demonstrate that our model outperforms other state-of-the-art models for multi-lesion segmentation.

Keywords

Diabetic retinopathy
Multi-class segmentation
Vessel segmentation
Transformer
Attention fusion
Subject terms

Retinal diseases
Computational science
http://dx.doi.org/10.13039/501100001809 National Natural Science Foundation of China 82071995 Xu Chenfangqian Key Research and Development Program of Jilin Province, China20220201141GX Xu Chenfangqian issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Among the complications caused by diabetes, diabetic retinopathy (DR) has a high incidence, which is an important cause of visual defects and permanent blindness1–3. Until now, the most recognized treatment is the early diagnosis and intervention, which can avoid 90% of vision loss4. Thus, DR screening on a regular basis is effective in detecting the disease earlier and reducing the rate of blindness caused by DR. The main pathological features of DR lesions in color fundus images often present as microaneurysms (MAs), hemorrhages (HEs), soft exudates (SEs), and hard exudates (EXs). Especially MAs are the earliest symptom of DR lesions. These four-category lesions are the basis of physicians’ diagnosis, so accurate segmentation is essential for the correct detection and diagnosis of DR lesions. However, as shown in Fig. 1a, the shape and the size of each lesion can vary widely, e.g., MAs are generally very small, ranging from one pixel to a few pixels, while some HEs can reach a maximum of thousands of pixels5. Different lesions may share similar characteristics, e.g., MAs and HEs are both red lesions, while SEs and EXs are both bright lesions. Therefore, in clinical practice, it is very cumbersome and time-consuming for ophthalmologists to manually identify these lesions from a large number of fundus images. The identifying results are all dependent on ophthalmologists’ experience, which may lead to missed and misdiagnosed cases. Above all, in order to reduce the workload of ophthalmologists and improve the accuracy of diagnosing DR, automatic segmentation technology for DR lesions has become a trend.Fig. 1 The distribution relationship of four categories of lesions, where segmented EX, HE, MA and SE are marked in red, green, blue and yellow. (a) is an example of DR lesions in fundus images from the DDR dataset. (b) shows the distribution relationships between different lesions that SEs usually surround HEs and EXs usually surround MAs. (c) shows the distribution relationships between vessles and different lesions that EXs are mostly located around the main trunks of vessels, whereas MAs are mostly located on both sides of the capillaries.

Compared with single-lesion segmentation, multi-lesion segmentation, which primarily involves segmenting MAs, HEs, SEs, and EXs, holds greater significance for the diagnosis of DR. However, multi-lesion segmentation faces more challenges due to significant differences in lesion size, shape, and intensity. Several effective methods have been proposed in this field, yet two issues remain inadequately addressed in previous research. Firstly, there is a lack of effective interaction between lesions and vessels, leading to the less sensitivity of the model to the areas around vessels and the loss of interdependence between lesions and vessels. Secondly, a unified structure makes it difficult to execute appropriate feature extraction strategies for different types of lesions with large-scale differences.

As shown in Fig. 1b, most EXs typically appear in a circular pattern around one or several MAs, while most SEs commonly appear at the periphery of HEs. This information, emphasized by the self-attention head, can be better used to differentiate multiple lesions. As shown in Fig. 1c, MAs are generally distributed at the margins of the capillaries, while SEs are near the trunk of upper and lower arteries. The cross-attention head effectively utilizes the vessel branch and lesion branches.

Much of the research has only achieved the segmentation of just one or two lesions, neglecting the dependencies between blood vessels and lesions, and the dependencies among multiple lesions. Huang et al.6 first utilized vascular information for four-category lesion segmentation simultaneously, and proposed the pathological connections between lesions and vessels and confirmed that these connection contributes to multi-lesion segmentation. However, first, the cross-attention head in the relation transformer block (RTB) they proposed introduced some noise in vessel features, and vessel features were only fused in single-scale features after up-sampling without effectively using multi-scale information of vessels. Therefore, although helpful in lesion segmentation, vascular information was not fully and effectively utilized, which caused negative impacts on the performance of multi-lesion segmentation. Second, they didn’t take full advantage of multi-scale lesion information, resulting in the difficulty that scale inconsistency between lesions poses to the multi-lesion segmentation: large lesions need to utilize long-range contexts in deep features, while small lesions need details in shallow features.

Motivated by the above observations, we propose in this work a novel attention fusion approach that more efficiently utilizes vascular information as prior knowledge without introducing new noise. Our proposed model uses a dual-branch UNet structure consisting of two encoder branches, which extract lesion and vessel features, respectively, and an attention fusion transformer encoder (AFTE), which is integrated between two branches. Specifically, both branches extract features by convolving input images, but since lesion features are more complex, a shift block is used in the lesion branch to extract local features further and enhance deep feature extraction.

An attention fusion block (AFB) is designed, which is a combination of a self-attention head and a cross-attention head in AFTE. Distinguishing from the traditional combination approach, i.e., a simple parallel, series, or summation of the self-attention head and the cross-attention head, vascular information is injected into the key in the attention mechanism with the aim of simultaneously capturing strong similarity relationships between lesions and vessels, and relationships among multiple lesions.

The self-attention head of AFB is employed to find the certain spatial arrangement for the majority of lesions in fundus images, which is consistent with the pathological causes of DR lesions. The cross-attention head, achieved by adding vascular information, i.e., the distribution of blood vessels in fundus images, to the key of the attention mechanisms, aids the localization of lesions. Furthermore, the selective introduction of vascular information only in the key can reduce the introduction of noise in vessel features. At the same time, vascular information at two different scales of features is fused in the encoder and the decoder to improve the integration of lesion features and vessel features.

Another improvement proposed in this paper is the optimization of the multi-scale feature fusion in the decoder. Existing works only convolve simply concatenated multi-scale features equally without the distinction between different scales. A scale-adaptive attention (SAA) is proposed to perform dynamic selection of different scales during the fusion to make multi-scale features fusion more flexible and improve the consistency of the fused multi-scale features.

To evaluate the efficiency of the proposed model, it is trained and tested on DDR and IDRiD datasets. Experimental results show that, compared with other state-of-the-art models, our model achieves outstanding results in the task of fundus lesion segmentation including segmenting EX, MAs, and SEs on two datasets. Furthermore, the ablation studies are conducted on the DDR dataset and the effectiveness of our improvements is validated in DR lesion segmentation.

In summary, our contributions are as follows: We propose a prior-guided attention fusion Transformer including a dual-branch encoder to extract lesion features and vessel features. In the proposed attention fusion transformer encoder (AFTE), vessel features extracted from the vessel branch as prior knowledge guide lesion segmentation. In the lesion branch, a shift block is added to expand the receptive field.

We propose an attention fusion block (AFB) that integrates two encoder branches using a new mechanism of attention at two different scales to more fully and effectively utilize vascular information while exploring the internal relationships between lesions.

In the decoder, we propose a scale-adaptive attention (SAA) to dynamically adjust the fusion of multi-scale features and solve the issue of conflicts in different scale features.

The rest of this paper is organized as follows. Section Related works briefly reviews some closely related works, including the studies of DR lesion segmentation and the shift mechanism. Section Methodology explains the details of our proposed model. Section Experiments includes the experimental settings and evaluation approaches, and illustrates the testing performance of each model by conducting different ablation studies. Section Conclusion discusses and summarises the experimental results.

Related works

Lesion segmentation in fundus images

Due to the development of DL models, various DL-based methods have been used for lesion segmentation.

For single lesion segmentation, Guo et al.7 proposed a multi-channel bin loss function to deal with the class imbalance and loss imbalance in EX segmentation, which focus on pixels that are difficult to classify. Mo et al.8 designed a fully convolutional residual network(FCRN) incorporating multi-level hierarchical information to segment the exudates. However, such models are only applicable to single lesion segmentation and are unable to be extended to other lesion segmentation.

For the same type of lesion segmentation, DR lesions are divided into red lesions (MAs and HEs) and bright lesions (SEs and EXs) according to the characteristics of lesions in fundus images. Playout et al.9 proposed a fully convolutional architecture that combines pixel-level training and image-level weakly supervised training with labels. They used two identical decoders, each of which specializes in segmenting red and bright lesion categories, and introduced an exchange layer that shares parameters softly between two decoders. However, their network lacks a sufficiently large receptive field to consider a wider context, resulting in segmenting some larger lesions into multiple smaller lesions rather than predicting complete boundaries.

For simultaneous segmentation of multiple lesions, Kumar et al.10 used iterative morphological approaches and watershed segmentation methods to detect and remove blood vessels and optic discs and obtain the structures of MAs and HEs used to extract seven features to train a classification model for automated detection of DR. Guo et al.11 developed L-Seg on the top of HED12 by adding a side extraction layer to each convolutional layer group of VGG-net13 to extract and fuse the features of different scales, and handling the class imbalance issue and segmenting DR lesions simultaneously. Liu et al.5 noticed the impact of different lesion scales and proposed a novel Many-to-Many Reassembly of Features(M2MRF) that maintains discriminative information about small lesions as much as possible and captures long-term spatial dependencies.

Transformer in fundus images

Transformer14, a convolution-free structure, has become one of the most widely used models in the field of natural language processing (NLP)15,16 since its introduction in 2017, for the capability of modeling long-range dependencies for data. It has an encoder-decoder structure, both of which are tandem by several similar blocks. Each block comprises components such as masked multi-head attention, layer normalization(LayerNorm), position-wise feed-forward network, etc. Additionally, there are cleverly inserted residual connections between these components.

Before introducing Transformer to the field of computer vision (CV) , there have been numerous attempts to combine convolution neural network (CNN) and attention17,18, but none of them yielded better results than CNN. Until 2020, Dosovitskiy et al.19, inspired by the successful experiences in NLP, proposed a pioneering model with the fewest modifications to the standard transformer structure: Vision Transformer (ViT), which lays a solid foundation for the adoption of transformer-based techniques in the field of CV. Swin Transformer20 improves on ViT and expands the applicability of Transformer, making it more suitable as a generic backbone for CV tasks. Nowadays, ViT variants have rapidly been applied to many CV tasks, such as image classification, object detection, semantic segmentation, etc., as a new research trend.

With the rapid development of Transformer in the field of CV, Transformer-based methods have also been widely used in the field of medical image analysis. Whether using Transformer alone21,22 or combining it with CNN, both local and global information of the image can be better captured. He et al.23 noticed the impact of different lesion scales. They progressively integrated multi-scale features from two or three adjacent encoding layers by a progressive feature fusion (PFF) block and then dynamically selected features at different scales by a dynamic attention block (DAB). Huang et al.6 developed a novel network for better interaction between lesions and vessels by introducing self-attention and cross-attention blocks after UNet. They proposed a Global Transformer Block(GTB) inspired by GCNet24 to emphasize more useful channel information in each position and a relation Transformer block (RTB) to explore the dependency among lesions and other retinal tissues.

Blood vessel and lesion segmentation

In an early task of segmenting DR lesions by morphological methods, blood vessels are easily mistaken for red lesions, so morphological structures such as vessels are detected and removed before the actual lesion segmentation. Even when applying deep neural networks(DNN) to DR lesion segmentation, some works still remove blood vessels in the stage of image preprocessing. Jaskirat Kaur et al.25 developed a generalized EX segmentation method consisting of three phases, namely retinal image enhancement, segmentation and elimination of anatomical structures, and EX segmentation, which is robust in the sense that it uses dynamic decision thresholds without considering large variations in retinal fundus images from different datasets. Al-hazaimeh et al.26 detected and removed optic disc and vessels respectively by introducing the Circular Hough Transform (HCT) and the Bias Corrected Separated Possibilistic Neighborhood FCM (BCSPNFCM) algorithm, which adds spatial information to segment vessels to distinguish blood vessels from DR lesions and selects appropriate features for EX, MA, and HE segmentation by feeding the processed feature maps into the Deep Convolutional Neural Network27.

Vascular structure has always been closely associated with DR lesions, and both MAs and EXs around capillaries are caused by vascular atresia. As the disease worsens, vessels may rupture, leading to the formation of HEs and SEs near the arteries from the leaked blood and lipoproteins, respectively. These are consistent with the presentation in fundus images. Huang et al.6 utilized the prior. Due to the lack of labeled fundus datasets for blood vessels and lesions, they used pseudo vascular masks provided by semi-supervised learning and adopted cross-attention mechanisms to assist in detecting lesion areas. However, they failed to fully utilize multi-scale features and only learned at one scale.

Shift block

Shift operation was proposed as an effective alternative to spatial convolution operation in 2017. Specifically, it employs a three-layer structure consisting of two Conv 1×1 and shift operation. In the follow-up work, shift operation is extended into various variants , including active shift28, sparse shift29, and partial shift30.

Wang et al.31 integrated the partial shift with Swin Transformer, introduced a shift block. The block comprises shift operation, LayerNorm, and MLP network. The combined model outperforms the Swin Transformer on tasks such as image classification, object detection, and segmentation, providing evidence for the substitutability of self-attention mechanisms.

In this work, we introduce the shift block into the feature extraction of fundus lesions. Without increasing computational cost, it further expands the receptive field by incorporating spatial neighborhood information to establish better contextual relationships.

Methodology

In this section, we first briefly overview the proposed network and then detail each proposed module, including the shift block, the attention fusion block (AFB), and the scale-adaptive attention (SAA).

Overall architecture

Fig. 2 The overall architecture of the proposed network. (a) is the overall architecture consisting of three components: a dual-branch encoder, a decoder and a classification head, where C1=32, C2=64, C3=128, C4=256 and C5=512 denote the number of feature channels at five scales, respectively, H=512, W=512 denote the scale of pre-processed images. (b) and (c) are the structures of Conv1 and Conv2, where h and w denote the height and width of features at the current scale. In our experiment, c′=2c when downsample is True.

As shown in Fig. 2a, the proposed network adopts a two-branch encoder-decoder architecture. The encoder consists of three main components: a lesion branch including a shift block for processing three-channel color fundus images, a vessel branch for handling single-channel grayscale vessel masks, and an attention fusion transformer encoder (AFTE) that integrates features from both branches. The decoder includes an upsampling structure and a SAA mechanism. Finally, a classification head is employed to map the outputs of the decoder, generating segmentation probabilities for four-category lesions.

The input sizes of the vessel and lesion branches of the encoder are 1×H×W and 3×H×W, respectively. Each branch undergoes convolution operation to obtain features Fves for vessels and Flesion for lesions, as illustrated in Fig. 2b and c for Conv1 and Conv2 structures. For each set of convolution operation, the downsample flag is set to “True” and the input images are downsampled to 1/4 of the original spatial dimensions only in the first convolution block. Before feeding lesion features into the AFTE, a shift block is used to enhance contextual information through shift and linear operation. Finally, the AFTE injects results from the vessel branch into the lesion branch, leveraging the vascular information extracted by the AFB to improve the accuracy of lesion feature learning.

The upsampling in the decoder follows the standard ViT structure, utilizing self-attention head from AFB. The structure of Up Conv2 is similar to Conv2, with the distinction of using Upsample instead of Downsample. Subsequently, the SAA calculates activation weights for each scale, facilitating a weighted fusion of features based on their inherent information to assign attention automatically. Finally, the classification head, a three-layer structure comprising two Conv 1×1 and an Upsample, predicts the final multi-lesion segmentation results based on features incorporating multi-scale information.

Shift block

The structure of the shift block consists of shift operation, LayerNorm, and a Multi-Layer Perceptron (MLP) connected in series, as shown in Fig. 3a. Specifically, shift operation involves shifting a portion of channels of the inputs along four spatial directions (up, down, left, right) by one pixel, as shown in Fig. 3b. Pixels exceeding the boundary are discarded, while blank pixels are filled with zeros. The remaining channels remain unchanged. Take the input Fsh∈Rc×h×w as an example, shift operation can be formulated as follows:1 z^0:h,1:w,0:γc←z0:h,0:w-1,0:γcz^0:h,0:w-1,γc:2γc←z0:h,1:w,γc:2γcz^0:h-1,0:w,2γc:3γc←z1:h,0:w,2γc:3γcz^1:h,0:w,3γc:4γc←z0:h-1,0:w,3γc:4γcz^0:h,0:w,4γc:γc←z0:h,0:w,4γc:γc

where γ is a scaling factor controlling the number of channels to be shifted. In our experiments, γ=112, which results in a total of 1/3 channels being shifted. Shifted features are then passed through a LayerNorm and an MLP. The MLP, consisting of two Linear and a GELU activation function, sequentially increases channels dimension from c to 4c, applies a non-linear activation, and projects channels back to c, yielding the output Fsh_out∈Rc×h×w for the shift block. It is worth noting that shift operation incurs low computational costs and requires no explicit calculations, which reduces a significant number of FLOPs during training.

The shift block is introduced before lesion features entering the AFTE. By repeatedly shifting lesion features in four spatial directions, the spatial receptive field is further expanded to enhance the correlation area between lesions and vessels, which contributes to a better understanding of global information in images. Therefore, shift operation aids the model in being more robust to variations in images, thereby improving the generalization performance of our model.

Attention fusion block

The AFTE follows a standard Transformer encoder structure, comprising three parts: multi-head attention mechanism, LayerNorm, and MLP, as shown in Fig. 3c. It takes two same-sized features, lesion features Flesion from the previous layer and vessel features Fves as input, producing new lesion features as output.Fig. 3 The structure of proposed components. (a) is the structure of shift block. (b) is the details of shift operation. (c) is the structure of the AFTE. (d) is the details of the AFB. (e) is the details of the SAA.

AFB, the core of AFTE as shown in Fig. 3d, introduces a novel fusion of self-attention and cross-attention, effectively capturing inherent dependencies between different lesions and between lesions and vessels simultaneously. In AFB, four heads are utilized, each of which is composed of a Conv 1×1 and a reshape. One head serves as the key generator for Fves, denoted as Kv, while the other three are used as query, key, and value generators for Flesion, denoted as Ql, Kl, Vl, respectively. Given the generated KvFves,QlFlesion,KlFlesion,VlFlesion∈Rhw×c, where h, w, and c represent spatial height, width, and channel number, respectively, the similarity matrix is computed by the dot product of QlFlesion and KlFlesion in the standard self-attention mechanism. While in AFB, spatial relationships between lesions and vessels is introduced by adding vascular information to the key generator. Specifically, KlFlesion and KvFves are added to obtain a new key with vascular information, denoted as Kfusion, and then compute the similarity matrix based on the fusion key. The calculation is described as follows:2 Kfusion=KlFlesionT+λKvFvesT

3 Fsim=KfusionQlFlesion

where Fsim∈Rhw×hw is the similarity matrix, and λ denotes the weight of vessel features. The larger λ is, the more the model focuses on the area around blood vessels, and the greater the role of vascular information in attention is. Subsequently, based on the integrated similarity matrix that incorporates vascular information, the new attention features is calculated as follows:4 Fatt=softmaxFsimVlFlesion

5 Fafb_out=LinearFatt+Flesion

where Linear represents a linear embedding using a Conv 1×1. The subsequent LayerNorm and MLP are the same as those in the shift block.

During the formation of different lesion features, mutual influences manifest in the images as certain spatial relationships between lesions. Additionally, some lesions have a more widespread distribution. The self-attention head can leverage distant contextual information in lesion features, capturing both long and short-distance dependencies between different lesions, which enables the model to learn these semantic relationships better. Furthermore, since the image has similar local regions, the self-attention head can establish global correlations for these regions, aiding the model in understanding the relationships between different parts of the feature map.

Since there are inherent pathological connections between vessels and lesions, vessel features are introduced in the cross-attention head to provide crucial prior knowledge for lesion segmentation. The similarity between lesion features and vessel features is computed by injecting vascular information into the key in the attention mechanism. This utilization of the dependencies between lesions and vessels allows the model to focus sensitively on potential lesion areas around vessels, enhancing the local correlations for better localization of MA and SE. Moreover, AFTE is employed at two scales to exploit the vascular information fully.

Scale-adaptive attention

Different scale information helps enhance the robustness of our model. As shown in Fig. 3e, the proposed SAA is composed of attention calculation and feature fusion.

In the attention calculation, Fsc is input features at four scales, and Fsci is the feature at each scale. Initially, Upsample and Conv 1×1 are sequentially performed to standardize the scale to Fsamesci∈RC4×H2×W2, i∈1,2,3,4, where C4, H2,and W2 represent the channel number, height, and width of the standardized feature, respectively. This operation can be expressed as:6 Fsame_sci=Conv1×1UpsampleFsci

Then, the AdaptivePooling operation is utilized to perform spatial averaging for each channel of each scale, obtaining global features Fspa_avgi∈RC4×1×1, i∈1,2,3,4.

Then, the mean value along the channel is further calculated:7 Fcha_meani=1C4∑j=1C4Fspa_avgi,j,i∈1,2,3,4

where Fspa_avgi,j represents the j-th node in Fspa_avgi. Finally, the Sigmoid activation function is applied to activate the obtained mean value, yielding importance scores for each scale. These scores are then multiplied with their corresponding Fsci, resulting in the activated features for the i-th scale:8 Facti=σFcha_meani×Fsci

In the feature fusion, the activated features for each scale are first concatenated along the channel dimension. Subsequently, the concatenated features are passed through a Conv_Norm layer, comprising a Conv 1×1 and a BatchNorm, resulting in the output Fsc_out∈RC4×H2×W2. This operation can be expressed as:9 Fsc_out=BatchNormConv1×1Fact1,...,Fact4

Among images at each scale, distinct feature information is leveraged to adjust the weights for each scale. Adaptive adjustment of fusion weights is more effective in integrating coarse-grained structure from large-scale features and rich details from small-scale features than treating features from different scales equally.

Experiments

Datasets

Our model is evaluated on two publicly available datasets with pixel-level annotations. DDR dataset32 is a general high-quality dataset of 757 color fundus images suitable for segmentation tasks, where 383 images are used for training, 149 for validation, and 225 for testing. The dataset includes 486 images with EX, 601 with HE, 570 with MA, and 239 with SE. The dataset has some low-quality images with degradation factors such as uneven lighting, underexposure, and overexposure. Therefore, it is the largest and most challenging dataset.

IDRiD dataset33 was released at the International Symposium on Biomedical Imaging (ISBI) in Challenge 2018 for the segmentation and grading of DR lesions. This dataset consists of 81 images with a resolution of 4288×2848, with EX and MA in each image, 80 with HE, and 40 with SE. There is an official split between training and test sets: 54 for training and 27 for testing.

Implementation details

To obtain the vascular structure in the images, a pre-trained model is employed on the CHASE34 and DRIVE35 datasets to extract vascular masks from the raw images. Subsequently, due to limited available video memory, both the larger color fundus images and vascular masks are uniformly resized to 512×512. Finally, considering the scarcity of training data, data augmentation, including random vertical flipping, random horizontal flipping, random scaling and random rotation, is applied to the resized fundus images before training to mitigate overfitting and enhance the model’s generalization.

In this work, all experiments are implemented using PyTorch as the backend and performed on a workstation equipped with two 24GB memory NVIDIA GeForce RTX 4090 GPU cards. The model is trained with a batch size of 4 for 1400 epochs. The model is optimized using the Adam optimizer with a momentum of 0.9 and weight decay of 0.001. The initial learning rate is set to 2×10-4 and adjusted using the ReduceLROnPlateau method. To address the challenge of the generally small foreground proportions in fundus images, binary cross-entropy loss and soft dice loss are employed to train the model for each lesion.

Evaluation metrics

Since fundus images suffer from category imbalance, and the Precision-Recall (PR) curve is more sensitive to imbalanced data due to its focusing on true positive precision and recall, the area under the PR curve (AUPR) is used to measure the performance of the proposed model, which is consistent with the IDRiD challenge. The class-wise IoU score and class-wise F1 score are also calculate, which are common evaluation metrics in the field of medical image analysis. The former indicates the overlap between the predicted and ground true segmented regions, and the latter is the harmonic mean of Precision and Recall. The evaluation metrics are defined as follows:IoU=TPTP+FP+FNF1=2TP2TP+FP+FN

In addition, similar to the definition of mAP in the object detection, the mean AUPR (mAUPR), mean IoU (mIoU), and mean F1 (mF1) are also computed to evaluate the overall performance of the proposed model for the multi-category lesion segmentation.

Performance comparison

Fig. 4 Visualized segmentation results of our model on (a) the DDR dataset and (b) the IDRiD dataset compared with other state-of-the-art models including UNet, DeepLabV3+, UNet++, CE-Net, PSPNet, where EXs, HEs, MAs and SEs are marked in red, green, blue and yellow, respectively. The white arrows indicate the differences between the predictions of all the models and the ground truth (GT).

Our model is compared with 11 state-of-the-art models on two datasets, including UNet36, UNet++37, DeepLabV3+38, CE-Net39, PSPNet40, EAD-Net41, L-Seg11, RILBP-YNet42, RTNet6, PMCNet23, and SAA43. The first five are general semantic segmentation models, and the latter six are lesion-specific segmentation models. The same experimental setup as the proposed model is used for a fair comparison. All models are applied to segment all four-category lesions using a unified structure.Table 1 Performance comparison of multi-lesion segmentation models on the DDR dataset and the IDRiD dataset.

dataset	Method	AUPR	IoU	F1	
Mean	EX	HE	MA	SE	Mean	EX	HE	MA	SE	Mean	EX	HE	MA	SE	
DDR	UNet	0.2881	0.4518	0.409	0.0821	0.2096	0.2185	0.3178	0.2657	0.0859	0.2045	0.3500	0.4823	0.4199	0.1582	0.3396	
DeepLab V3+	0.2927	0.4073	0.401	0.0552	0.3075	0.2151	0.2953	0.2619	0.0697	0.2335	0.345	0.4559	0.415	0.1304	0.3787	
UNet++	0.3057	0.4976	0.3616	0.0697	0.294	0.2172	0.3258	0.2258	0.0858	0.2317	0.3485	0.4915	0.3683	0.158	0.3762	
CE-Net	0.3092	0.4591	0.4146	0.0734	0.2896	0.2366	0.3539	0.2686	0.0817	0.2422	0.3718	0.5228	0.4234	0.1511	0.3900	
PSPNet	0.3221	0.4463	0.425	0.0647	0.3524	0.2412	0.3244	0.2901	0.0751	0.2753	0.3777	0.4898	0.4497	0.1398	0.4317	
L-Seg	0.3208	0.5546	0.3586	0.1052	0.2648	–	–	–	–	–	–	–	–	-	–	
RTNet	0.3362	0.5671	0.3656	0.1176	0.2943	–	–	–	–	–	–	–	–	–	–	
PMCNet	0.3644	0.5430	0.3987	0.1994	0.3164	–	–	–	–	–	–	–	–	–	–	
Ours	0.3902	0.5724	0.4256	0.202	0.3608	0.2834	0.4229	0.3029	0.1557	0.252	0.4328	0.5944	0.4649	0.2694	0.4026	
IDRiD	UNet	0.4058	0.6827	0.4291	0.0782	0.4332	0.3391	0.5316	0.3491	0.0849	0.3907	0.4825	0.6942	0.5176	0.1565	0.5619	
CE-Net	0.4702	0.6394	0.5581	0.1188	0.5646	0.3824	0.5303	0.427	0.1156	0.4567	0.531	0.6931	0.5985	0.2073	0.6271	
PSPNet	0.4867	0.6389	0.5554	0.1464	0.6062	0.3712	0.4545	0.445	0.0961	0.4891	0.5183	0.625	0.6159	0.1753	0.6569	
UNet++	0.4886	0.6584	0.5694	0.1612	0.5653	0.3939	0.5249	0.4203	0.1582	0.4723	0.5488	0.6885	0.5918	0.2732	0.6416	
DeepLab V3+	0.5234	0.7134	0.5700	0.1810	0.6293	0.3208	0.4825	0.3386	0.0633	0.3988	0.4615	0.6509	0.5059	0.119	0.5702	
EAD-Net	0.5489	0.7818	0.5649	0.2408	0.6083	–	–	–	–	–	–	–	–	–	–	
L-Seg	0.6515	0.7945	0.6374	0.4627	0.7113	–	–	–	–	–	–	–	–	–	–	
RILBP-YNet	0.6586	0.7858	0.6914	0.3804	0.7267	–	–	–	–	–	–	–	–	–	–	
SAA	0.6738	0.8812	0.6704	0.4152	0.7281	–	–	–	–	–	–	–	–	–	–	
Ours	0.6755	0.8593	0.6948	0.3983	0.7495	0.4817	0.5147	0.4736	0.1986	0.5397	0.626	0.6796	0.6428	0.3314	0.6604	
Best: bold.

Fig. 5 Visualized segmentation results using different scale combinations and weighting strategies. (a) is the positive effect of complete identification of the arterial trunk on the nearby HE and SE segmentation, whereas (b) is the negative effect of incomplete identification of the capillaries on the EX and MA segmentation. EXs, HEs, MAs and SEs are marked in red, green, blue and yellow, respectively. The white arrows indicate the differences between the ground truth (GT) and the predictions of all the models.

The comparative results of all models on the two datasets are presented in Table 1, and visualization results are presented in Fig. 4a and b respectively. Our model exhibits excellent performance on the DDR dataset in terms of AUPR, IoU, and F1. Particularly in AUPR, the segmentation performance for all four-category lesions reaches optimal, indicating that our model partially overcomes the class imbalance issue. Even for the relatively fewer MAs, our segmentation performance shows a noticeable improvement. For the IDRiD dataset, our model achieves the best performance for HEs and SEs, with AUPR increased by 2.4 and 2.1%, respectively, compared with the suboptimal model (SAA). This improvement may stem from the generally effective segmentation of upper and lower arterial trunks in the vascular masks, allowing the model to focus on SEs closer to the main arteries, as illustrated in Fig. 5b. However, AUPR for EXs and MAs are slightly lower than SAA, with reductions of 2.2% and 1.7%, respectively. This discrepancy may arise from poor segmentation of capillaries in some images, where MAs are distributed around capillaries, and most EXs surround MAs. The inaccurate identification of capillary regions hinders the model from focusing on potential MAs and adjacent EXs in these areas, as shown in Fig. 5a. Furthermore, our model consistently ranks first in mAUPR, mIoU, and mF1 for all four-category lesions on both datasets. Particularly on the DDR dataset, compared with the suboptimal model, our model demonstrates a significant improvement of at least 2.6%, 4.2% and 5.5% in these metrics, indicating superior overall performance in multi-lesion segmentation and highlighting its robust competitiveness.

Ablation studies on the DDR dataset

Ablation experiments are conducted on the DDR dataset to better understand the impact of each component of our model. First, we verify the effectiveness of three important components: shift block, AFTE, and SAA, then analyze their results. Then, to verify the effectiveness of vascular information as prior knowledge for lesion segmentation, we discuss its contribution in terms of attention. Due to the main role of AFTE in achieving the fusion of lesion information and vascular information through AFB, in order to further select the appropriate fusion method in key, we investigate the specific implementation of AFB. Finally, we conduct a comparative study of the scales at which the fused vascular information is available. In these experiments, the remaining model structures remain the same as the proposed model.

Ablation studies of components

Fig. 6 Visualized segmentation results for different combinations of three modules on DDR dataset. (a) demonstrate the segmentation results of different methods on EXs and MAs, showing that our model is better than other combinations, while (b) demonstrate the results on HEs and SEs, showing that the combination (base+AFTE+Shift Block) has the best segmentation. The pink arrows indicate the differences between GT and the results of different combinations.

Figure 6 visualizes the segmentation results of our model with different components, and Table 2 shows the specific comparison results of the AUPR, IoU, F1-score, mean AUPR (mAUPR), mean IoU (mIoU) and mean F1-score (mF1) of four lesions for different components of our model. In Table 2, Row 1 is the results of the baseline, where the encoder includes the lesion branch without shift block and AFTE, and the decoder includes an upsampling structure without SAA. Compared with the baseline, Row 2 demonstrates that the AFTE allows more flexible attention generation, adapting to the context of multi-lesion and vessel features to achieve adaptive coordination of lesion and vascular information. This enhances the model’s ability to model dependencies among multiple lesions and improves the recognition of each lesion, which meets our expectations.Table 2 Ablation studies of different components of our model on the DDR dataset.

Model	AFTE	Shift block	SAA	AUPR	mAUPR	mIoU	mF1	
				EXs	HEs	MAs	SEs				
Base				0.5099	0.4234	0.1229	0.3337	0.3475	0.2458	0.3872	
Base+AFTE	✓			0.5495	0.4077	0.1424	0.3782	0.3694	0.2753	0.423	
Base+AFTE+Shift Block	✓	✓		0.5631	0.4287	0.1669	0.3793	0.3845	0.2764	0.4254	
Base+AFTE+SAA	✓		✓	0.5649	0.4078	0.1693	0.3612	0.3758	0.2655	0.4121	
Base+AFTE+Shift Block+SAA	✓	✓	✓	0.5724	0.4256	0.202	0.3608	0.3902	0.2834	0.4328	
Best: bold; second best: underlined.

Row 3 confirms that shift block can help capture spatial relationships of small-scale structures by using minor pixel shifts, improving sensitivity to EXs and MAs. Additionally, as the shapes and sizes of the four-category lesions vary, shift block introduces non-linear transformations, diversifying feature representations and aiding the model in identifying the complex texture features of HEs and SEs so that the best segmentation effect is achieved for HEs and SEs, as shown in Fig. 6b. Moreover, the combination of AFTE and shift block (Row 4) outperforms the use of AFTE alone, which indicates that shift block enhances the correlation in the regions between lesions and vessels by shifting features within the channels to take advantage of spatial neighbors, enabling AFTE to learn the dependencies between lesions and vessels better.

It is observed from Row 5 that SAA can effectively integrate the features of EXs, HEs and MAs at different scales. Since irregularly shaped SEs may have large differences at different scales, the SAA is unable to integrate these inconsistent feature representation adequately and efficiently, which results in a degradation of segmentation performance.

Although our components do not consistently yield significant improvements for each lesion individually, their combined use brings about greater overall improvement. Compared with the baseline, our model achieves a 4.2% increase in mAUPR, a 3.6% increase in mIoU, and a 4.4% increase in mF1, demonstrating the best overall segmentation performance.Table 3 Ablation studies of different implementations of AFB on the DDR dataset, where [self, cross] denotes the simple concatenation of the self-attention mechanism and cross-attention mechanism over the channels, concat. denotes the concatenation of the lesion-feature based and vascular-feature based keys over the channels, and + denotes the spatial summation of these two keys.

Fusion method	[Self, Cross]	AFB	AUPR	mAUPR	
		Kl	Kv	EXs	HEs	MAs	SEs		
N/A	✓			0.5522	0.2302	0.1235	0.2428	0.2872	
N/A		✓		0.5473	0.455	0.1725	0.3144	0.3723	
Concat.		✓	✓	0.5499	0.4413	0.1623	0.3316	0.3713	
+(ours)		✓	✓	0.5724	0.4256	0.202	0.3608	0.3902	
Best: bold.

Ablation studies of AFB

The impact of the key on segmentation performance is investigated, as shown in Table 3, where kl and kv denotes generated attention weights based on multi-lesion features alone and vessel features alone, respectively. Row 2, 3 and 4 list the AUPR, indicating the impact of only lesion features, fused vessel features and lesion features by concatenation and by summation on multi-lesion segmentation.

Row 4 shows a significant improvement in AUPR for SEs, EXs and MAs, which conforms to our previous analysis that different vascular structures are strongly associated with MAs and SEs. At the same time, EXs have a certain distribution relationship with MAs. However, the segmentation of capillary limits the attention to MAs, resulting in less pronounced improvement in AUPR for MAs compared with Row 2.

Additionally, it is found that introducing vascular information influences HE segmentation. This may be because HEs, compared with other lesions, share more similar features, specifically color, with vessels. When the model attempts to learn lesions around vessels, the less distinctive features of HEs can be learnt, making cross-learning more challenging. Nevertheless, incorporating vessel features as prior knowledge is beneficial for enhancing the overall performance of multi-lesion segmentation.

Having established that vascular information contributes to multi-lesion segmentation, the specific method for fusing vascular information is further discussed. The results for two fusion methods are presented in Rows 3 and 4 of Table 3. Here, the symbol “+” represents the direct addition of the two keys generated based on lesion features and vessel features, while “concat” represents the concatenation of the two keys along the channel dimension. It can be observed that introducing vascular information through concatenation leads to a decline in segmentation for three lesions and the overall model due to information conflicts. This indicates that the channel-wise concatenation cannot effectively leverage vascular information. In contrast, the spatial summation allows better fusion between the two types of information and improves the ability to model complex features.

Additionally, the AFB is compared with the RTB in RTNet, which injects vessel features using a standard cross-attention mechanism and then connects it with multi-lesion features processed through a common self-attention mechanism, denoted as [Self, Cross]. Different from the RTB, which incorporates vascular information in both the key and value generators, the AFB incorporates vascular information in the key generator only. From Table 3, it can be seen that our model can focus on the region around vessels more precisely without being disturbed by unnecessary noise because the reduction of incorporating vascular information effectively reduces noise from vessel features.

Analysis on λ

The hyperparameter λ in AFB denotes the percentage of vascular information in attention. Table 4 show the performance comparison of segmentation by setting λ to 0.4, 0.5, and 0.6, respectively. The results indicates that the segmentation is more accurate in the case of 0.5. Compared with the optimal setting(λ=0.5), increasing or decreasing the vascular information by 10% leads to a loss of about 1.7%.Table 4 Segmentation result comparison of our model under different λ on the DDR dataset.

λ	AUPR	mAUPR	
	EX	HE	MA	SE		
0.4	0.5164	0.4354	0.1765	0.359	0.3718	
0.5	0.5724	0.4256	0.202	0.3608	0.3902	
0.6	0.5522	0.4370	0.1787	0.3221	0.3725	
Best: bold.

Analysis on fusion scale

Experiments on two combinations of three scales:H4×W4 and H16×W16, H8×W8 and H16×W16 are conducted. Deeper features corresponding to larger scales contain more contextual relationships, which facilitates capturing the global structures. Introducing vessel features into deep features allows the combination of global structural information with local details around vessels, thereby improving the segmentation of lesions, as shown in Table 5.Table 5 Segmentation result comparison with the introduction of vascular information at different scales on the DDR dataset.

Scale	AUPR	mAUPR	
H/4×W/4	H/8×W/8	H/16×W/16	EXs	HEs	MAs	SEs		
✓		✓	0.5545	0.4255	0.154	0.3267	0.3652	
	✓	✓	0.5724	0.4256	0.202	0.3608	0.3902	
Best: bold.

Conclusion

In this study, we propose a prior-guided attention fusion Transformer for multi-lesion segmentation of diabetic retinopathy. In this model, we propose a new AFB which combines self-attention and cross-attention mechanisms to incorporate both lesion information and vascular information in the key generator. The self-attention is used to find long-distance dependency relationships between lesions, and the cross-attention is used to effectively integrate valuable vascular information which guides the model more accurately locating different lesions in complex fundus images as prior knowledge. To our best knowledge, this is the first work to employ such prior-guided attention fusion method in fundus lesion segmentation. In order to further extract complex lesion features and improve the sensitivity of the model to structural features at small scales, the shift block is introduced to our model to improve the robustness of the model. The SAA is proposed to calculate the multi-scale attention of features generated during the upsampling process to address the issue of inconsistent lesion scales in fundus images. The segmentation results on two public datasets indicate that our model performs well overall in various multi-lesion segmentation tasks.

However, due to the lack of datasets providing both pixel-level multi-lesion labels and vascular annotations, vascular masks obtained from pretrained models are inevitably coarse-grained, which causes incomplete utilization of vascular information and introduction of noise, resulting in a degradation of the HE segmentation.

In the future, we will explore an unsupervised learning strategy to extract vascular information from fundus images, and continue improving the multi-scale fusion method to accurately focus on details or semantic information in lesion features at different scales where vascular information is incorporated.

Acknowledgements

This work was supported by national natural science foundation of China under Grant 82071995, and key research and development program of Jilin Province, China under Grant 20220201141GX.

Author contributions

Chenfangqian Xu and Xiaoxin Guo conceived and conducted the experiments, and wrote the main manuscript text, Guangqi Yang and Yihao Cui prepared and preprocessed the experimental data, Longchen Su and Hongliang Dong provided the software support, and Xiaoying Hu and Songtian Che analyzed and discussed the experimental results. All authors reviewed the manuscript.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Thomas R Halim S Gurudas S Sivaprasad S Owens D Idf diabetes atlas: A review of studies utilising retinal photography on the global prevalence of diabetes related retinopathy between 2015 and 2018 Diabetes Res. Clin. Pract. 2019 157 107840 10.1016/j.diabres.2019.107840 31733978
Thomas, R., Halim, S., Gurudas, S., Sivaprasad, S. & Owens, D. Idf diabetes atlas: A review of studies utilising retinal photography on the global prevalence of diabetes related retinopathy between 2015 and 2018. Diabetes Res. Clin. Pract. 157, 107840 (2019).31733978 10.1016/j.diabres.2019.107840
2. Ciulla TA Amador AG Zinman B Diabetic retinopathy and diabetic macular edema: Pathophysiology, screening, and novel therapies Diabetes Care 2003 26 2653 2664 10.2337/diacare.26.9.2653 12941734
Ciulla, T. A., Amador, A. G. & Zinman, B. Diabetic retinopathy and diabetic macular edema: Pathophysiology, screening, and novel therapies. Diabetes Care 26, 2653–2664 (2003).12941734 10.2337/diacare.26.9.2653
3. Raman R Gella L Srinivasan S Sharma T Diabetic retinopathy: An epidemic at home and around the world Indian J. Ophthalmol. 2016 64 69 10.4103/0301-4738.178150 26953027
Raman, R., Gella, L., Srinivasan, S. & Sharma, T. Diabetic retinopathy: An epidemic at home and around the world. Indian J. Ophthalmol. 64, 69 (2016).26953027 10.4103/0301-4738.178150
4. Wong TY Guidelines on diabetic eye care: the international council of ophthalmology recommendations for screening, follow-up, referral, and treatment based on resource settings Ophthalmology 2018 125 1608 1622 10.1016/j.ophtha.2018.04.007 29776671
Wong, T. Y. et al. Guidelines on diabetic eye care: the international council of ophthalmology recommendations for screening, follow-up, referral, and treatment based on resource settings. Ophthalmology 125, 1608–1622 (2018).29776671 10.1016/j.ophtha.2018.04.007
5. Liu Q Liu H Ke W Liang Y Automated lesion segmentation in fundus images with many-to-many reassembly of features Pattern Recogn. 2023 136 109191 10.1016/j.patcog.2022.109191
Liu, Q., Liu, H., Ke, W. & Liang, Y. Automated lesion segmentation in fundus images with many-to-many reassembly of features. Pattern Recogn. 136, 109191 (2023).10.1016/j.patcog.2022.109191
6. Huang S Li J Xiao Y Shen N Xu T Rtnet: relation transformer network for diabetic retinopathy multi-lesion segmentation IEEE Trans. Med. Imaging 2022 41 1596 1607 10.1109/TMI.2022.3143833 35041595
Huang, S., Li, J., Xiao, Y., Shen, N. & Xu, T. Rtnet: relation transformer network for diabetic retinopathy multi-lesion segmentation. IEEE Trans. Med. Imaging 41, 1596–1607 (2022).35041595 10.1109/TMI.2022.3143833
7. Guo S Bin loss for hard exudates segmentation in fundus images Neurocomputing 2020 392 314 324 10.1016/j.neucom.2018.10.103
Guo, S. et al. Bin loss for hard exudates segmentation in fundus images. Neurocomputing 392, 314–324. 10.1016/j.neucom.2018.10.103 (2020).10.1016/j.neucom.2018.10.103
8. Mo J Zhang L Feng Y Exudate-based diabetic macular edema recognition in retinal images using cascaded deep residual networks Neurocomputing 2018 290 161 171 10.1016/j.neucom.2018.02.035
Mo, J., Zhang, L. & Feng, Y. Exudate-based diabetic macular edema recognition in retinal images using cascaded deep residual networks. Neurocomputing 290, 161–171. 10.1016/j.neucom.2018.02.035 (2018).10.1016/j.neucom.2018.02.035
9. Playout C Duval R Cheriet F A novel weakly supervised multitask architecture for retinal lesions segmentation on fundus images IEEE Trans. Med. Imaging 2019 38 2434 2444 10.1109/TMI.2019.2906319 30908197
Playout, C., Duval, R. & Cheriet, F. A novel weakly supervised multitask architecture for retinal lesions segmentation on fundus images. IEEE Trans. Med. Imaging 38, 2434–2444. 10.1109/TMI.2019.2906319 (2019).30908197 10.1109/TMI.2019.2906319
10. Kumar S Adarsh A Kumar B Singh AK An automated early diabetic retinopathy detection through improved blood vessel and optic disc segmentation Optics Laser Technol. 2020 121 105815 10.1016/j.optlastec.2019.105815
Kumar, S., Adarsh, A., Kumar, B. & Singh, A. K. An automated early diabetic retinopathy detection through improved blood vessel and optic disc segmentation. Optics Laser Technol. 121, 105815 (2020).10.1016/j.optlastec.2019.105815
11. Guo S L-seg: An end-to-end unified framework for multi-lesion segmentation of fundus images Neurocomputing 2019 349 52 63 10.1016/j.neucom.2019.04.019
Guo, S. et al. L-seg: An end-to-end unified framework for multi-lesion segmentation of fundus images. Neurocomputing 349, 52–63. 10.1016/j.neucom.2019.04.019 (2019).10.1016/j.neucom.2019.04.019
12. Xie, S. & Tu, Z. Holistically-nested edge detection. In 2015 IEEE International Conference on Computer Vision (ICCV), 1395–1403, 10.1109/ICCV.2015.164 (2015).
13. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (2015).
14. Vaswani A Attention is all you need Adv. Neural Inf. Process. Syst. 2017 30 1
Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 1 (2017).
15. Egonmwan, E. & Chali, Y. Transformer and seq2seq model for paraphrase generation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, 249–255 (2019).
16. Shi, Y. et al. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6783–6787 (IEEE, 2021).
17. Wang, X., Girshick, R., Gupta, A. & He, K. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7794–7803 (2018).
18. Carion, N. et al. End-to-end object detection with transformers. In European conference on computer vision, 213–229 (Springer, 2020).
19. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
20. Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012–10022 (2021).
21. Le Dinh T Lee S-H Kwon S-G Kwon K-R Covid-19 chest x-ray classification and severity assessment using convolutional and transformer neural networks Appl. Sci. 2022 12 4861 10.3390/app12104861
Le Dinh, T., Lee, S.-H., Kwon, S.-G. & Kwon, K.-R. Covid-19 chest x-ray classification and severity assessment using convolutional and transformer neural networks. Appl. Sci. 12, 4861 (2022).10.3390/app12104861
22. Krishnan, K. S. & Krishnan, K. S. Vision transformer based covid-19 detection using chest x-rays. In 2021 6th International Conference on Signal Processing, Computing and Control (ISPCC), 644–648 (IEEE, 2021).
23. He A Progressive multiscale consistent network for multiclass fundus lesion segmentation IEEE Trans. Med. Imaging 2022 41 3146 3157 10.1109/TMI.2022.3177803 35613070
He, A. et al. Progressive multiscale consistent network for multiclass fundus lesion segmentation. IEEE Trans. Med. Imaging 41, 3146–3157 (2022).35613070 10.1109/TMI.2022.3177803
24. Ni J Wu J Tong J Chen Z Zhao J Gc-net: Global context network for medical image segmentation Comput. Methods Programs Biomed. 2020 190 105121 10.1016/j.cmpb.2019.105121 31623863
Ni, J., Wu, J., Tong, J., Chen, Z. & Zhao, J. Gc-net: Global context network for medical image segmentation. Comput. Methods Programs Biomed. 190, 105121 (2020).31623863 10.1016/j.cmpb.2019.105121
25. Kaur J Mittal D A generalized method for the segmentation of exudates from pathological retinal fundus images Biocybernetics Biomed. Eng. 2018 38 27 53 10.1016/j.bbe.2017.10.003
Kaur, J. & Mittal, D. A generalized method for the segmentation of exudates from pathological retinal fundus images. Biocybernetics Biomed. Eng. 38, 27–53. 10.1016/j.bbe.2017.10.003 (2018).10.1016/j.bbe.2017.10.003
26. Al-hazaimeh, O. M., Abu-Ein, A. A., Tahat, N. M., Al-Smadi, M. A. & Al-Nawashi, M. M. Combining artificial intelligence and image processing for diagnosing diabetic retinopathy in retinal fundus images. International Journal of Online & Biomedical Engineering 18 (2022).
27. Sun Y Xue B Zhang M Yen GG Lv J Automatically designing cnn architectures using the genetic algorithm for image classification IEEE Trans. Cybernetics 2020 50 3840 3854 10.1109/TCYB.2020.2983860
Sun, Y., Xue, B., Zhang, M., Yen, G. G. & Lv, J. Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Trans. Cybernetics 50, 3840–3854. 10.1109/TCYB.2020.2983860 (2020).10.1109/TCYB.2020.2983860
28. Jeon, Y. & Kim, J. Constructing fast network through deconstruction of convolution. Advances in neural information processing systems 31 (2018).
29. Chen, W., Xie, D., Zhang, Y. & Pu, S. All you need is a few shifts: Designing efficient convolutional neural networks for image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7241–7250 (2019).
30. Lin, J., Gan, C. & Han, S. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF international conference on computer vision, 7083–7093 (2019).
31. Wang, G., Zhao, Y., Tang, C., Luo, C. & Zeng, W. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence 36, 2423–2430 (2022).
32. Li T Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening Inf. Sci. 2019 501 511 522 10.1016/j.ins.2019.06.011
Li, T. et al. Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Inf. Sci. 501, 511–522 (2019).10.1016/j.ins.2019.06.011
33. Porwal P Idrid: Diabetic retinopathy-segmentation and grading challenge Med. Image Anal. 2020 59 101561 10.1016/j.media.2019.101561 31671320
Porwal, P. et al. Idrid: Diabetic retinopathy-segmentation and grading challenge. Med. Image Anal. 59, 101561 (2020).31671320 10.1016/j.media.2019.101561
34. Owen CG Retinal arteriolar tortuosity and cardiovascular risk factors in a multi-ethnic population study of 10-year-old children; the child heart and health study in england (chase) Arterioscler. Thromb. Vasc. Biol. 2011 31 1933 1938 10.1161/ATVBAHA.111.225219 21659645
Owen, C. G. et al. Retinal arteriolar tortuosity and cardiovascular risk factors in a multi-ethnic population study of 10-year-old children; the child heart and health study in england (chase). Arterioscler. Thromb. Vasc. Biol. 31, 1933–1938 (2011).21659645 10.1161/ATVBAHA.111.225219
35. Staal J Abràmoff MD Niemeijer M Viergever MA Van Ginneken B Ridge-based vessel segmentation in color images of the retina IEEE Trans. Med. Imaging 2004 23 501 509 10.1109/TMI.2004.825627 15084075
Staal, J., Abràmoff, M. D., Niemeijer, M., Viergever, M. A. & Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23, 501–509 (2004).15084075 10.1109/TMI.2004.825627
36. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, 234–241 (Springer, 2015).
37. Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N. & Liang, J. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, 3–11 (Springer, 2018).
38. Chen Y Meng Q Zhang J Effects of the notch angle, notch length and injection rate on hydraulic fracturing under true triaxial stress: An experimental study Water 2018 10 801 10.3390/w10060801
Chen, Y., Meng, Q. & Zhang, J. Effects of the notch angle, notch length and injection rate on hydraulic fracturing under true triaxial stress: An experimental study. Water 10, 801 (2018).10.3390/w10060801
39. Gu Z Ce-net: Context encoder network for 2d medical image segmentation IEEE Trans. Med. Imaging 2019 38 2281 2292 10.1109/TMI.2019.2903562 30843824
Gu, Z. et al. Ce-net: Context encoder network for 2d medical image segmentation. IEEE Trans. Med. Imaging 38, 2281–2292. 10.1109/TMI.2019.2903562 (2019).30843824 10.1109/TMI.2019.2903562
40. Zhao, H., Shi, J., Qi, X., Wang, X. & Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2881–2890 (2017).
41. Wan C Ead-net: A novel lesion segmentation method in diabetic retinopathy using neural networks Disease Markers 2021 2021 6482665 10.1155/2021/6482665 34512815
Wan, C. et al. Ead-net: A novel lesion segmentation method in diabetic retinopathy using neural networks. Disease Markers 2021, 6482665 (2021).34512815 10.1155/2021/6482665
42. Pavani PG Biswal B Gandhi TK Simultaneous multiclass retinal lesion segmentation using fully automated rilbp-ynet in diabetic retinopathy Biomed. Signal Process. Control 2023 86 105205 10.1016/j.bspc.2023.105205
Pavani, P. G., Biswal, B. & Gandhi, T. K. Simultaneous multiclass retinal lesion segmentation using fully automated rilbp-ynet in diabetic retinopathy. Biomed. Signal Process. Control 86, 105205 (2023).10.1016/j.bspc.2023.105205
43. Bo, W., Li, T., Liu, X. & Wang, K. Saa: scale-aware attention block for multi-lesion segmentation of fundus images. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 1–5 (IEEE, 2022).
