
==== Front
Ophthalmol Ther
Ophthalmol Ther
Ophthalmology and Therapy
2193-8245
2193-6528
Springer Healthcare Cheshire

39127983
1009
10.1007/s40123-024-01009-7
Original Research
Ocular Disease Detection with Deep Learning (Fine-Grained Image Categorization) Applied to Ocular B-Scan Ultrasound Images
Ye Xin 16
He Shucheng 1
Dan Ruilong 3
Yang Shangchao 2
Xv Jiahao 4
Lu Yang 5
Wu Bole 5
Zhou Congying 2
Xu Han 2
Yu Jiafeng 1
Xie Wenbin 6
Wang Yaqi wangyaqi@cuz.edu.cn

37
http://orcid.org/0000-0002-7518-2932
Shen Lijun slj@mail.eye.ac.cn

12
1 grid.417401.7 0000 0004 1798 6507 Center for Rehabilitation Medicine, Department of Ophthalmology, Zhejiang Provincial People’s Hospital (Affiliated People’s Hospital, Hangzhou Medical College), Hangzhou, Zhejiang China
2 https://ror.org/00rd5t069 grid.268099.c 0000 0001 0348 3990 School of Ophthalmology and Eye Hospital, Wenzhou Medical University, 270 West Xueyuan Road, Wenzhou, Zhejiang China
3 https://ror.org/0576gt767 grid.411963.8 0000 0000 9804 6672 Hangzhou Dianzi University, Computer and Software School, Hangzhou, Zhejiang China
4 https://ror.org/05v58y004 grid.415644.6 0000 0004 1798 6662 Department of Ophthalmology, Shaoxing People’s Hospital, Shaoxing, Zhejiang China
5 grid.459700.f Department of Ophthalmology, Lishui People’s Hospital, Lishui, Zhejiang China
6 Bijie Hospital of Zhejiang Provincial People’s Hospital, Bijie, Guizhou China
7 https://ror.org/04t7gxr16 grid.449896.e 0000 0004 1755 0017 College of Media Engineering, Communication University of Zhejiang, Xueyuan Road 998, Hangzhou, China
11 8 2024
11 8 2024
10 2024
13 10 26452659
7 4 2024
22 7 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
Introduction

The aim of this work is to develop a deep learning (DL) system for rapidly and accurately screening for intraocular tumor (IOT), retinal detachment (RD), vitreous hemorrhage (VH), and posterior scleral staphyloma (PSS) using ocular B-scan ultrasound images.

Methods

Ultrasound images from five clinically confirmed categories, including vitreous hemorrhage, retinal detachment, intraocular tumor, posterior scleral staphyloma, and normal eyes, were used to develop and evaluate a fine-grained classification system (the Dual-Path Lesion Attention Network, DPLA-Net). Images were derived from five centers scanned by different sonographers and divided into training, validation, and test sets in a ratio of 7:1:2. Two senior ophthalmologists and four junior ophthalmologists were recruited to evaluate the system's performance.

Results

This multi-center cross-sectional study was conducted in six hospitals in China. A total of 6054 ultrasound images were collected; 4758 images were used for the training and validation of the system, and 1296 images were used as a testing set. DPLA-Net achieved a mean accuracy of 0.943 in the testing set, and the area under the curve was 0.988 for IOT, 0.997 for RD, 0.994 for PSS, 0.988 for VH, and 0.993 for normal. With the help of DPLA-Net, the accuracy of the four junior ophthalmologists improved from 0.696 (95% confidence interval [CI] 0.684–0.707) to 0.919 (95% CI 0.912–0.926, p < 0.001), and the time used for classifying each image reduced from 16.84 ± 2.34 s to 10.09 ± 1.79 s.

Conclusions

The proposed DPLA-Net showed high accuracy for screening and classifying multiple ophthalmic diseases using B-scan ultrasound images across mutiple centers. Moreover, the system can promote the efficiency of classification by ophthalmologists.

Keywords

Ocular B-scan ultrasound images
Multi-center study
Fine-grained image categorization
Deep learning
http://dx.doi.org/10.13039/501100017594 Medical Science and Technology Project of Zhejiang Province 2023KY915 Ye Xin issue-copyright-statement© Springer Healthcare Ltd., part of Springer Nature 2024
==== Body
pmcKey Summary Points

Why carry out this study?	
Ultrasound examination is one of the fastest and simplest imaging methods for eyes, which is helpful when screening for and diagnosing the most common abnormalities.	
The automated classification of ocular disease using B-scan ultrasound images remains a challenge.	
What was learned from the study?	
This study developed the Dual-Path Lesion Attention Network (DPLA-Net) for the detection of four ocular diseases and normal eyes based on ultrasound images.	
DPLA-Net can assist ophthalmologists in classifying common abnormalities using ocular B-scan ultrasound images.	

Introduction

Intraocular tumor (IOT), retinal detachment (RD), and vitreous hemorrhage (VH) are common blinding diseases in the primary or emergency care setting. Uveal melanoma is the most common malignant IOT, with an annual incidence of 0.2–6 cases per million [1]. Retinoblastoma is the second most common malignant IOT, affecting up to 1 in 20,000 live births in the UK, and the most common primary intraocular malignancy of childhood [2]. RD has been reported to have a 5-year cumulative incidence of 0.21%, and the incidence rate was 42/100,000 person-years [3]. The average annual national hospital incidence rate of hematogenous RD was 21.97 ± 1.04 per 10,0000 population in France [4]. For these diseases, early screening and accurate diagnosing can avoid vision loss and are essential in the primary or emergency care setting.

Ocular ultrasonography is a non-invasive measure and has the advantage of detecting abnormalities in subtle or peripheral lesions and describing the shape of the entire eyeball [5]. The B-scan ultrasound image is comparable to a histological section through the eye and can effectively provide posterior segment information regardless of the opacification of the light-conducting media, such as that present in cataracts or vitreous hemorrhage. Due to the popularity of ultrasonography in primary hospitals and the low cost of the examination [6], it still cannot be replaced. Ocular ultrasonography has been widely applied to the diagnosis of IOT, posterior scleral staphyloma (PSS), VH, RD, and so on. As the population ages, the number of people who need to be screened for blinding eye diseases is increasing. However, the number of ophthalmologists is limited, and there are fewer experienced ophthalmologists who are able to read ultrasound reports [7].

Artificial intelligence (AI) has greatly improved the diagnostic accuracy and efficiency for some medical diseases. Deep learning (DL) shows potential in lesion detection, classification, and segmentation using medical images [8–10]. Previous studies have reported that DL models based on ultrasound images can be used in thyroid, liver, uterus, breast, heart, and kidney diseases [11–15]. However, only a few studies have focused on the application of AI models to ocular ultrasound images. The interpretation of ocular ultrasound images relies on the clinical experience of the ophthalmologist. An accurate and efficient AI-assisted identification system has the potential to reduce visual loss due to a delayed or incorrect diagnosis.

In this study, we developed an AI system to classify IOT, RD, PSS, VH, and normal eyes using ocular B-scan ultrasound images. We evaluated the model’s performance in an independent test which including images collected from six hospitals. In addition, a study was conducted to evaluate this system’s performance in assisting ophthalmologists in classification.

Methods

Multi-center Ocular Ultrasonography Process

This multi-center study was approved by the ethics committees for six medical centers, including Zhejiang Provincial People’s Hospital (QT2023086), Lishui People’s Hospital (LLW-FO-403), Shaoxing People’s Hospital (2022-102-Y-01), and the Hangzhou, Wenzhou, and Zhijiang Xihu branches of the Affiliated Eye Hospital of Wenzhou Medical University (H2023-004-K-02). Written informed consent was obtained from all patients involved during the initial collection. The patient’s consent for their data to be used in the future, but not specifically for this study, was obtained. Ocular ultrasonic images obtained from 2017 to 2021 were collected. The instruments used in this study were the Aviso (Quantel Medical, Clermont-Ferrand, France) and SW-2100 (Tianjin Sauvy Electronic Technology, China). The images were taken by senior sonographers with over 10 years of clinical experience. All the patients were examined by B-scan ultrasonography, with the probe frequency set to 10 MHz. The patient was in a supine position and the ultrasound probe was applied to the closed eyelids during scanning.

Study Sample

Ophthalmologists made a clinical diagnosis based on the medical history, ocular ultrasonic report, and other examinations, such as fundus images, optical coherence tomography, fundus fluorescein angiography, and so on. Patients who were clinically diagnosed with IOT, RD, PSS, VH, and normal eyes were included in this study. The included images were stored in JPG or PDF format. The exclusion criteria were as follows: (1) images with more than two lesion characteristics and some other abnormalities except for IOT, RD, PSS, and VH; (2) abnormal situations where ultrasound was not the primary imaging method for screening and diagnosis; (3) images with inferior quality (such as those taken in the presence of severe vitreous opacity), as this made it difficult for ophthalmologists to interpret the ocular ultrasound images.

The ultrasound images were tagged with the clinical diagnosis. Four ophthalmologists with 10 years of clinical experience divided these images into five categories: IOT, RD, PSS, VH, and normal. Due to the large size of the annotated sample in this study, four ophthalmologists were first assigned to annotate before the most experienced doctor among them double-checked all the examination results. If there was any disagreement, a discussion among the four ophthalmologists yielded the final result.

Development of the Deep-Learning Model

Firstly, the original B-scan ultrasound images were pre-processed before training and testing. Specifically, irrelevant features in the raw ultrasound images, including patient information and the details of imaging device parameters, were removed since these features were detrimental when optimizing the deep models. Then, the original ultrasound images were center cropped to a fixed size (224 × 224) for reasoning. The images were all normalized according to the mean and variance of the dataset. We employed a data augmentation technique, as labeled data are precious in medical image analysis for the optimization of deep models. Specifically, Albumentations, a well-known Python library, was utilized in our DL pipeline to alleviate the overfitting issue and enhance the model’s robustness in multi-site scenarios. During training, the augmentation pipeline would produce new variations of a batch of selected original images for inputs. We especially used a random combination from a pre-defined list of transformations for augmentation, including flip, rotation, affine transformation, and contrast limited adaptive histogram equalization (CLAHE).

We proposed the Dual-Path Lesion Attention Network (DPLA-Net) for fine-grained image categorization (FGIC) using ocular B-scan ultrasound images. Our DPLA-Net simulated the experienced ophthalmologists, who divided these images into five categories: IOT, RD, PSS, VH, and normal. Our framework was developed via Python and the PyTorch framework with two NVIDIA RTX 2080Ti GPUs with 11 GB of memory. The output of the DL model ranged from 0 to 1, representing the probability of the existence of each category.

As shown in Fig. 1, DPLA-Net consisted of two main parts: the Dual-Path Convolutional Neural Network (DPNN) and the Prediction Fusion (PF) module. The CNN backbone in the macro path first extracted semantic features and outputted coarse predictions. We adopted the Efficientnet-b3 [16] architecture, a network automatically designed by a neural architecture search (NAS), as our CNN backbone. Moreover, lesion attention maps produced by the macro path were also utilized to focus on the skeptical regions in the images for fine diagnosis. Like the macro path, the CNN in the micro path extracted features for output diagnosis. Finally, the PF module fused the coarse and fine predictions from the macro and micro paths for ensemble diagnosis prediction.Fig. 1 The workflow of this study. (1) Data collection and preparation. Patients who were clinically diagnosed with IOT, RD, PSS, VH, and normal were included in this study. The ultrasound images were tagged with the clinical diagnosis. Ocular B-scan ultrasound images were collected from six centers and poor-quality images were eliminated. Four ophthalmologists divided these images into five categories: IOT, RD, PSS, VH, and normal. (2) System development. Ocular B-scan ultrasound images were independently passed to the AI system as the input. The AI system first outputted coarse predictions made through the macro path and then focused on the micro path to give a refined diagnosis. Then the AI system combined the predictions from both the macro and micro paths to give an ensemble diagnosis prediction. (3) Output and evaluation. The classification results and corresponding heat maps were given as the output. Two senior ophthalmologists and four junior ophthalmologists were recruited to evaluate the system's performance. A comparison of the performance of the ophthalmologists with and without the assistance from the AI system was conducted. AI artificial intelligence, HWH Hangzhou branch of the Affiliated Eye Hospital of Wenzhou Medical University, HWW Wenzhou branch of the Affiliated Eye Hospital of Wenzhou Medical University, HWZ Zhijiang branch of the Affiliated Eye Hospital of Wenzhou Medical University, LPH Lishui People’s Hospital, SPH Shaoxing People’s Hospital, ZPH Zhejiang Provincial People’s Hospital, IOT intraocular tumor, RD retinal detachment, PSS posterior scleral staphyloma, VH vitreous hemorrhage

Performance Evaluation of the Model

The performance of DPLA-Net was evaluated using an internal test dataset consisting of 1296 images. For further performance validation, we randomly selected a subset of 300 images from the test dataset, asked six clinicians (including two senior ophthalmologists and four junior ophthalmologists) to provide diagnoses independently, and compared their results with those from the AI system. After 2 weeks of washing out, six ophthalmologists gave separate diagnoses of the same subset with the assistance of a reference diagnosis from the AI system. Their diagnostic performance was compared to that achieved without the AI assistant.

Statistical Analysis

The accuracy, sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve (AUC) along with their 95% confidence interval (CIs) were calculated using the R packages Hmisc_4.2–0 and pRoc_1.15.3 to assess the performance of the DL model. The difference in accuracy between the model and the ophthalmologists was analyzed by a χ2 test. The Mann–Whitney U test was applied to compare the accuracy of the ophthalmologists with and without the assistance from the DL model. p < 0.05 was considered to indicate a significant difference. All the results were analyzed with SPSS software.

Results

Details of the Training, Validation, and Test Datasets

After the disqualified images had been filtered out, 6054 images from 2176 patients were included. Among these images, 884 images were of IOT, 1875 images were of RD, 1022 images were of PSS, 1341 images were of VH, and 932 images were of normal eyes. The images were divided into a training set (4139 images), a validation set (619 images), and a test set (1296 images); the ratio between the sets was close to 7:1:2. The distribution details of the training, validation, and test datasets are shown in Table 1. Table 1 Number of images included in the training, validation, and test sets

Diagnosis	Training	Validation	Test	
Normal	656	90	186	
IOT	603	100	181	
PSS	710	101	211	
RD	1197	231	447	
VH	973	97	271	
Total	4139	619	1296	
IOT intraocular tumor, PSS posterior scleral staphyloma, RD retinal detachment, VH vitreous hemorrhage

Classification Performance of DPLA-Net

When classifying the five type images in the test set, DPLA-Net achieved a mean accuracy of 0.943, a mean sensitivity of 0.933, a mean specificity of 0.986, and a mean AUC of 0.992 (as shown in Table 2). The AUC was 0.988 for IOT, 0.997 for RD, 0.994 for PSS, 0.988 for VH, and 0.993 for normal eyes (as shown in Fig. 2). The confusion matrix results are shown in Fig. 3A. Table 2 The diagnostic performance of DPLA-Net, the ophthalmologists alone, and the DPLA-Net-assisted ophthalmologists

Model	F1 score	Accuracy	Sensitivity	Specificity	AUC	
First test	0.931	0.941	0.930	0.986	0.992	
Second test	0.935	0.944	0.935	0.986	0.992	
Third test	0.936	0.944	0.933	0.986	0.993	
Average	0.934	0.943	0.933	0.986	0.992	
AUC area under the receiver operating characteristic (ROC) curve, DPLA-Net Dual-Path Lesion Attention Network

Fig. 2 Diagnostic performance of DPLA-Net and the ophthalmologists. The AUCs were 0.988, 0.997, 0.994, 0.988, and 0.993 for IOT, RD, PSS, VH, and normal morphological features, respectively. The performance of DPLA-Net compared with that of each ophthalmologist in recognizing IOT (A), VH (B), RD (C), PSS (D), and normal eyes (E). AUC area under the curve, DPLA-Net Dual-Path Lesion Attention Network, IOT intraocular tumor, RD retinal detachment, PSS posterior scleral staphyloma, VH vitreous hemorrhage

Fig. 3 A Confusion matrix generated by the proposed DPLA-Net when identifying different lesions. The numbers of correct classifications are found on the diagonal. B The training and validation cross entropy (CE) loss curves for our network. C The training and validation accuracy curves for our network. DPLA-Net Dual-Path Lesion Attention Network, IOT intraocular tumor, PSS posterior scleral staphyloma, RD retinal detachment, VH vitreous hemorrhage

The hyperparameters of the models were optimized according to their performance for the validation dataset. A huge number of experiments were conducted during the process, and we only selected the models with the best performance for the dataset. The loss curve during our training (see Fig. 3B) indicated that the overall loss basically converged at about 64 epochs. The results supported the notion that our system showed good generalization for the test dataset, which was completely independent of the training and test sets, and it achieved good sensitivity and specificity.

Comparison of DPLA-Net and the Ophthalmologists

A subset of the testing set (300 images) was used to compare the performance of DPLA-Net and Ophthalmologists (see Table 3). DPLA-Net achieved an accuracy of 0.965 (95% CI 0.955–0.975) for normal eyes, 0.990 (95% CI 0.985–0.995) for IOT, 0.966 (95% CI 0.956–0.976) for PSS, 0.985 (95% CI 0.978–0.991) for RD, and 0.982 (95% CI 0.974–0.989) for VH. The two senior ophthalmologists achieved an accuracy of 0.958 (95% CI 0.951–0.965), a sensitivity of 0.895 (95% CI 0.884–0.906), and a specificity of 0.974 (95% CI 0.968–0.980). The four junior ophthalmologists achieved an accuracy of 0.878 (95% CI 0.870–0.887), a sensitivity of 0.696 (95% CI 0.684–0.707), and a specificity of 0.924 (95% CI 0.917–0.931). Table 3 Comparison of the performance of the ophthalmologists with and without the assistance of DPLA-Net

	DPLA-Net	Without AI assistant	With AI assistant	
SO	JO	SO	JO	
Normal	
 Sensitivity (95% CI)	0.941 (0.928, 0.954)	0.940 (0.921, 0.959)	0.776 (0.752, 0.800)	0.940 (0.921, 0.959)	0.957* (0.945, 0.968)	
 Specificity (95% CI)	0.969 (0.960, 0.979)	0.944 (0.926, 0.963)	0.788 (0.765, 0.811)	0.979† (0.969, 0.991)	0.935* (0.921, 0.949)	
 Accuracy (95% CI)	0.965 (0.955, 0.975)	0.943 (0.925, 0.962)	0.786 (0.763, 0.809)	0.972† (0.958, 0.985)	0.939* (0.926, 0.953)	
IOT	
 Sensitivity (95% CI)	0.939 (0.926, 0.952)	0.892 (0.868, 0.917)	0.850 (0.830, 0.870)	0.946† (0.928, 0.964)	0.931* (0.916, 0.945)	
 Specificity (95% CI)	0.998 (0.996, 1.000)	1.000 (1.000, 1.000)	0.905 (0.889, 0.922)	0.998 (0.994, 1.000)	0.992* (0.987, 0.997)	
 Accuracy (95% CI)	0.990 (0.985, 0.995)	0.977 (0.965, 0.989)	0.893 (0.876, 0.911)	0.986 (0.978, 0.996)	0.982* (0.974, 0.989)	
PSS	
 Sensitivity (95% CI)	0.853 (0.834, 0.872)	0.804 (0.772, 0.836)	0.525 (0.496, 0.553)	0.931† (0.911, 0.952)	0.892* (0.875, 0.910)	
 Specificity (95% CI)	0.988 (0.982, 0.994)	0.994 (0.988, 1.000)	0.991 (0.986, 0.996)	0.990 (0.982, 0.998)	0.992 (0.987, 0.997)	
 Accuracy (95% CI)	0.966 (0.956, 0.976)	0.962 (0.946, 0.977)	0.912 (0.896, 0.928)	0.980† (0.969, 0.991)	0.975* (0.966, 0.984)	
RD	
 Sensitivity (95% CI)	0.973 (0.964, 0.982)	0.976 (0.964, 0.988)	0.710 (0.685, 0.736)	0.992 (0.985, 0.999)	0.913* (0.897, 0.929)	
 Specificity (95% CI)	0.991 (0.985, 0.996)	0.954 (0.937, 0.970)	0.984 (0.977, 0.991)	0.977† (0.965, 0.989)	0.995* (0.990, 0.999)	
 Accuracy (95% CI)	0.985 (0.978, 0.991)	0.958 (0.942, 0.974)	0.927 (0.912, 0.941)	0.980† (0.969, 0.991)	0.978* (0.969, 0.986)	
VH	
 Sensitivity (95% CI)	0.971 (0.961, 0.980)	0.907 (0.884, 0.930)	0.587 (0.559, 0.615)	0.927† (0.908, 0.949)	0.901* (0.884, 0.918)	
 Specificity (95% CI)	0.984 (0.978, 0.991)	0.961 (0.945, 0.976)	0.950 (0.938, 0.963)	0.992† (0.984, 0.999)	0.982* (0.975, 0.990)	
 Accuracy (95% CI)	0.982 (0.974, 0.989)	0.950 (0.933, 0.967)	0.874 (0.855, 0.893)	0.978† (0.967, 0.990)	0.965* (0.955, 0.975)	
AI artificial intelligence, SO senior ophthalmologists, JO junior ophthalmologists, DPLA-Net Dual-Path Lesion Attention Network, IOT intraocular tumor, PSS posterior scleral staphyloma, RD retinal detachment, VH vitreous hemorrhage, CI confidence interval

*The diagnostic performance of junior ophthalmologists achieved with AI assistance.was statistically significantly different from that achieved without AI assistance at the level of p < 0.05 †The diagnostic performance of senior ophthalmologists achieved with AI assistance was statistically significantly different from that achieved without AI assistance at the level of p < 0.05

Visualization of the Fine-Grained Classification Process of DPLA-Net Using Heatmaps

The AI system provided heatmaps to demonstrate that DPLA-Net made the classification based on the correct lesion. Regions highlighted with warmer colors represented areas that were more critical to the final class determination. Most lesions are concentrated in the posterior pole of the eyeball, and some are very subtle. Moreover, partial lesions are hard to detect using traditional AI systems. Our system extracts multi-scale features from macro views, marks saliency regions in micro views, and finally utilizes lesion-specific micro features to improve classification performance. The regions of interest were captured precisely, and results were compatible with the judgment of the ophthalmologists (Fig. 4).Fig. 4 Visualization of the macro and the micro views (at the attention center) of four lesions and the corresponding attention heatmaps. The first two images represent the global views and corresponding attention maps. The following two images represent the local views (at the attention center) and corresponding attention maps. A An example of an IOT lesion detected by our AI system. B An example of an RD lesion detected by our AI system. C An example of a PSS lesion detected by our AI system. D An example of a VH lesion detected by our AI system. E An example of a normal eye. F An example of the AI system misclassifying RD lesions as PSS lesions due to the presence of normal tissue. AI artificial intelligence, IOT intraocular tumor, RD retinal detachment, PSS posterior scleral staphyloma, VH vitreous hemorrhage

Comparison of the Performance of the Ophthalmologists with and without the Assistance of DPLA-Net

The four junior ophthalmologists had an accuracy of 0.878 (95% CI 0.870–0.887) without the DL model assistant and 0.968 (95% CI 0.963–0.972) with the assistant, indicating that there was an significant improvement with the DL model assistant (p < 0.001). The two senior ophthalmologists had an accuracy of 0.958 (95% CI 0.951–0.965) without the DL model assistant and 0.979 (95% CI 0.974–0.984) accuracy with the assistant, implying that there was an significant improvement with the assistant (p < 0.001).

In addition, the time used for classifying each image reduced from 16.84 ± 2.34 s to 10.09 ± 1.79 s for junior ophthalmologists. The time used for classifying each image reduced from 14.35 ± 9.69 s to 8.1 ± 4.15 s for senior ophthalmologists. The results are shown in Table 3 and Fig. 5.Fig. 5 The difference in performance of each ophthalmologist with and without AI assistance. Horizontal lines represent the individual change in performance, the gray dot represents the performance without AI assistance, and the red dot represents the performance with AI assistance. A The change in individual sensitivity. B The change in individual specificity. C The change in individual accuracy. D The change in mean diagnosis time per image. AI artificial intelligence, JO junior ophthalmologists, SO senior ophthalmologists

Discussion

In this study, a DL model (DPLA-Net) was developed for detecting and classifying abnormal findings and highlighting the locations of lesions in B-scan ultrasound images. The DPLA-Net architecture achieved a mean accuracy of 0.943 in recognizing IOT, RD, PSS, VH, and normal eyes in a test set (1296 images), and the area under the curve was 0.988 for IOT, 0.997 for RD, 0.994 for PSS, 0.988 for VH, and 0.993 for normal eyes. In an experiment comparing the performance of DPLA-Net and that of the ophthalmologists using a subset of 300 images, DPLA-Net presented a level of performance that was better than that of junior ophthalmologists and close to that of senior ophthalmologists.

Ocular ultrasonography is one of the quickest and simplest imaging modalities. Due to the aging of global populations, the number of patients who need early screening for blinding eye diseases has significantly increased. However, the number of ophthalmologists who can interpret ultrasound images is not enough to meet the needs of large-scale screening work. Combining AI systems with digital health systems has the potential to establish a remote healthcare system and overcome the shortage of medical resources. If we can screen out patients with common abnormalities using DL models, the workload of ophthalmologists will be greatly reduced. Although AI systems have been widely applied in ultrasound image analysis and processing [9, 17, 18], achieving a sufficiently high DL model performance to promote the clinical diagnosis of ocular disease using B-scan ultrasound images remains a challenge. Previous studies reported that the sensitivity of RD diagnosis using ultrasound images was 92.3 to 94.2% [19–21]. The sensitivity and specificity of the diagnosis of VH by human experts were reported to be 81.9% and 82.3%, respectively [22]. Meanwhile, DPLA-Net achieved a mean accuracy of 94.3%, a mean sensitivity of 93.3%, and a mean specificity of 98.6% for classifying common ocular abnormal findings using B-scan ultrasound images. In addition, the AUC was 0.988 for IOT, 0.997 for RD, 0.994 for PSS, 0.988 for VH, and 0.993 for normal eyes. These results show that DL models have the potential to approach the performance level of human experts using B-scan ultrasound images.

Moreover, human performance in interpreting ultrasound images depends on the human expert’s knowledge and experience. There are misdiagnoses and delayed diagnoses due to poor consistency among ophthalmologists. An efficient and accurate DL model may be a good solution to this problem. FGIC has been proposed to help with disease judgment based on images with a high degree of similarity. Some networks, including SPP-Net [23], P-CNN [24], and SR-GNN [25], have been designed to focus on the discriminative parts of images. This study developed the Dual-Path Lesion Attention Network (DPLA-Net) for the detection of four ocular diseases and normal eyes based on ultrasound images, and it achieved a mean sensitivity of 0.935 and a mean specificity of 0.986. DPLA-Net can automatically focus on basic features and recognize subtle features to a level beyond human visual observation ability. To figure out whether DPLA-Net could assist ophthalmologists in classifying common abnormalities using ocular B-scan ultrasound images, two senior ophthalmologists and four junior ophthalmologists were recruited to the experiments. The performance of the ophthalmologists with and without the assistance of DPLA-Net was compared, and the results showed that with the help of DPLA-Net, the accuracy of the four junior ophthalmologists improved from 0.696 (95% CI 0.684–0.707) to 0.919 (95% CI 0.912–0.926, p < 0.001), and the time used for classifying each image reduced from 16.84 ± 2.34 s to 10.09 ± 1.79 s. Therefore, the DL model may improve the accuracy, efficiency, and stability of diagnosis in clinical practice. It will be helpful in large-scale screening and provide useful classification advice for inexperienced clinical doctors.

As ultrasonography is widely used for detecting ocular lesions, the abundance of ocular ultrasound image datasets has fueled the development of AI research in this field in recent years. It is difficult to determine the extent of retinal detachment in the presence of vitreous hemorrhage through traditional fundus photography. Chen et al. developed an AI system for screening abnormal findings using ocular ultrasound images and reported a sensitivity of 0.920 and 0.790 in recognizing RD and VH, respectively [26]. Feng et al. proposed a DL model for detecting VH that was based on GoogLeNet Inception V1 and achieved a sensitivity of 0.96 [27]. Wang et al. proposed an AI model that achieved AUCs of 0.97 and 0.99 for the automatic recognition of IOT and PSS and could detect microtumors and mild PSS based on irregular curvature features of the posterior of the eyeball [28]. Ito et al. developed an automated algorithm that was able to detect the posterior eyewall and quantify its shape, which led to good classification performance, with a maximum validation AUC of 0.897 [29]. Different from these DL models for general image categorization, the DPLA-Net architecture proposed in this study used an attention mechanism to capture more detailed features and weigh important features for distinguishing between lesions. The DPLA-Net architecture achieved a mean accuracy of 0.943 in recognizing IOT, RD, PSS, VH, and normal eyes in the test set (1296 images), and the area under the curve was 0.988 for IOT, 0.997 for RD, 0.994 for PSS, 0.988 for VH, and 0.993 for normal eyes. In the experiment comparing the performance of DPLA-Net with that of the ophthalmologists using a subset of 300 images, DPLA-Net gave a level of performance that was better than that of the ophthalmologists. The FGIC architecture recognizes subcategories of the same category by mining subtle and discriminative regions. Thus, DPLA-Net first outputs a coarse prediction through the macro path and then, after focusing on micro regions, provides a refined classification. This dual-path setup combines multi-scale and multi-view predictions, thus concentrating on subtle lesions and enhancing the robustness of this AI system.

It is worth noting that adequate training samples were important for developing a DL model with good performance. The benefits of expanding the sample sizes were that it negates the impact of overfitting in machine learning and enhances the precision of the DL model [30]. A large-scale database was built in this study to lay the foundations for developing a more robust model. To improve the generalization ability and expand the usage scenarios of the DL model [31], data were collected from six centers. However, the heterogeneity of the data, which results from variations in the inspection instruments and the skills of operators, remains a key challenge to deploying AI models in real-world clinical settings [32]. To address this challenge, the proposed DPLA-Net architecture used representation learning to enhance its robustness to multi-center data [33]. Moreover, we applied various data augmentation techniques that mimic challenging images from multiple sources to enhance the robustness and precision of our DL-based model [34].

In addition, attention heatmaps were generated to allow us to directly observe how DPLA-Net made the prediction and classification, which may help to determine the contributions of intraocular features to the final diagnosis. As we all know, one of the most significant disadvantages of DL-based models is the lack of interpretability of the model’s prediction, which is a critical factor when applying the deep model in clinical scenarios. Common interpretable techniques such as Grad-CAM are widely utilized to generate saliency maps for explaining the predictions of DL models [35]. However, deep models cannot easily incorporate these interpretable techniques to achieve more robust prediction results. In contrast, the attention mechanism and its variants are more suitable for use in this work, as they can be utilized to enhance deep features and provide specific highlight regions for interpreting prediction results. Specifically, spatial lesion attention was used to localize the ocular lesions in B-mode ultrasound images without a reliance on manual annotations [36].

There were some limitations to our study. Firstly, the DL model used for ultrasound image classification cannot make the final clinical diagnosis, which needs to be made by the doctor, who will utilize other clinical information on the patient as well. Secondly, DPLA-Net is designed to make the classification based on still images instead of videos captured by sonographers, which means that the accurate detection of an abnormal region requires the experience of inspectors. Thirdly, although DPLA-Net achieved a mean sensitivity of 0.935 and a mean specificity of 0.986, some false-positive and false-negative findings still need to be discussed. As shown in Fig. 4, when it comes to shallow RD (especially lesions that cannot even be determined by professional doctors), ocular ultrasound images cannot provide information on the whole eye. In this situation, local normal tissue or signal interference may affect the recognition performance of the AI system. Lastly, DPLA-Net cannot give multiple classifications for one image, but it may be improved by allowing it to give multiple classifications based on video data in the future.

Conclusions

This study developed the Dual-Path Lesion Attention Network (DPLA-Net) for the detection of four common abnormal findings using B-scan ocular ultrasound images across mutiple centers. DPLA-Net provided good accuracy in five-category classification and can assist ophthalmologists in terms of their accuracy and efficiency.

Author Contributions

Study concept and design: Xin Ye, Yaqi Wang, and Lijun Shen. Acquisition, analysis, or interpretation of data: Xin Ye, Shucheng He, Ruilong Dan, Shangchao Yang, Wenbin Xie, Jiafeng Yu, Jiahao Xv, Yang Lu, Bole Wu, Congying Zhou, Han Xu. Drafting of the manuscript: Xin Ye, Shucheng He, Ruilong Dan, Shangchao Yang. Critical revision of the manuscript for important intellectual content: Lijun Shen, Yaqi Wang. Final approval of the version to be published: Lijun Shen, Xin Ye. Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved: Lijun Shen, Yaqi Wang.

Funding

This work was supported in part by the Medical Science and Technology Project of Zhejiang Province (no. 2023KY915). The journal’s Rapid Service Fee was funded by the authors.

Data Availability

Data are available from Zhejiang Provincial People's Hospital (contact Dr Xin Ye, yexinsarah@163.com) for researchers who meet the criteria for access to confidential data.

Declarations

Conflict of Interest

The authors report no conflicts of interest in this study. Xin Ye, Shucheng He, Ruilong Dan, Shangchao Yang, Jiahao Xv, Yang Lu, Bole Wu, Congying Zhou, Han Xu, Jiafeng Yu, Wenbin Xie, Yaqi Wang, and Lijun Shen declare that they have no competing interests.

Ethical Approval

The study was approved by the Ethics Committee of Zhejiang Provincial People’s Hospital (QT2023086), the School of Ophthalmology and Eye Hospital of Wenzhou Medical University (the Hangzhou, Wenzhou, and Zhijiang Xihu branches) (H2023-004-K-02), Shaoxing People’s Hospital (2022–102-Y-01), and Lishui People’s Hospital (LLW-FO-403). Written informed consent was obtained from all patients involved during the initial collection. The patient’s consent for their data to be used in the future, but not specifically for this study, was obtained.

Xin Ye, Shucheng He and Ruilong Dan have contributed equally to this work.
==== Refs
References

1. Manchegowda P, Singh AD, Shields C, Kaliki S, Shah P, Gopal L, Rishi P. Uveal melanoma in Asians: a review. Ocul Oncol Pathol. 2021;7(3):159–67.
2. Jenkinson H. Retinoblastoma: diagnosis and management—the UK perspective. Arch Dis Child. 2015;100(11):1070–5.
3. Gerstenberger E Stoffelns B Nickels S Münzel T Wild PS Beutel ME Schmidtmann I Lackner KJ Pfeiffer N Schuster AK Incidence of retinal detachment in Germany: results from the Gutenberg Health Study Ophthalmologica 2021 244 2 133 140 10.1159/000513080 33197918
Gerstenberger E, Stoffelns B, Nickels S, Münzel T, Wild PS, Beutel ME, Schmidtmann I, Lackner KJ, Pfeiffer N, Schuster AK. Incidence of retinal detachment in Germany: results from the Gutenberg Health Study. Ophthalmologica. 2021;244(2):133–40.33197918
4. Ben Ghezala I Mariet AS Benzenine E Gabrielle PH Baudin F Bron AM Quantin C Creuzot-Garcher CP Incidence of rhegmatogenous retinal detachment in France from 2010 to 2016: seasonal and geographical variations Br J Ophthalmol 2022 106 8 1093 1097 33658231
Ben Ghezala I, Mariet AS, Benzenine E, Gabrielle PH, Baudin F, Bron AM, Quantin C, Creuzot-Garcher CP. Incidence of rhegmatogenous retinal detachment in France from 2010 to 2016: seasonal and geographical variations. Br J Ophthalmol. 2022;106(8):1093–7.33658231
5. Lorente-Ramos RM Armán JA Muñoz-Hernández A Gómez JM de Torre SB US of the eye made easy: a comprehensive how-to review with ophthalmoscopic correlation Radiographics 2012 32 5 E175 E200 10.1148/rg.325115105 22977037
Lorente-Ramos RM, Armán JA, Muñoz-Hernández A, Gómez JM, de Torre SB. US of the eye made easy: a comprehensive how-to review with ophthalmoscopic correlation. Radiographics. 2012;32(5):E175–200.22977037
6. Bates A, Goett HJ. Ocular ultrasound. StatPearls. Treasure Island: StatPearls Publishing LLC; 2024.
7. Kim DJ Francispragasam M Docherty G Silver B Prager R Lee D Maberley D Test characteristics of point-of-care ultrasound for the diagnosis of retinal detachment in the emergency department Acad Emerg Med 2019 26 1 16 22 10.1111/acem.13454 29774966
Kim DJ, Francispragasam M, Docherty G, Silver B, Prager R, Lee D, Maberley D. Test characteristics of point-of-care ultrasound for the diagnosis of retinal detachment in the emergency department. Acad Emerg Med. 2019;26(1):16–22.29774966
8. Koh JEW Raghavendra U Gudigar A Ping OC Molinari F Mishra S Mathavan S Raman R Acharya UR A novel hybrid approach for automated detection of retinal detachment using ultrasound images Comput Biol Med 2020 120 103704 10.1016/j.compbiomed.2020.103704 32250849
Koh JEW, Raghavendra U, Gudigar A, Ping OC, Molinari F, Mishra S, Mathavan S, Raman R, Acharya UR. A novel hybrid approach for automated detection of retinal detachment using ultrasound images. Comput Biol Med. 2020;120: 103704.32250849
9. Huang X He S Wang J Yang S Wang Y Ye X Lesion detection with fine-grained image categorization for myopic traction maculopathy (MTM) using optical coherence tomography Med Phys 2023 50 9 5398 5409 10.1002/mp.16623 37490302
Huang X, He S, Wang J, Yang S, Wang Y, Ye X. Lesion detection with fine-grained image categorization for myopic traction maculopathy (MTM) using optical coherence tomography. Med Phys. 2023;50(9):5398–409.37490302
10. Ye X, Gao K, He S, Zhong X, Shen Y, Wang Y, Shao H, Shen L. Artificial intelligence-based quantification of central macular fluid volume and VA prediction for diabetic macular edema using OCT images. Ophthalmol Ther. 2023;12(5):2441–52.
11. Zhang P, Huang H, Xiong Q, He X, Liu Y. Feature analysis and automatic classification of B-mode ultrasound images of fatty liver. Biomed Signal Process Control. 2023;79: 104073.
12. He H Liu R Zhou X Zhang Y Yu B Xu Z Huang H B-ultrasound image analysis of intrauterine pregnancy residues after mid-term pregnancy based on smart medical big data J Healthcare Eng 2022 2022 1 11
He H, Liu R, Zhou X, Zhang Y, Yu B, Xu Z, Huang H. B-ultrasound image analysis of intrauterine pregnancy residues after mid-term pregnancy based on smart medical big data. J Healthcare Eng. 2022;2022:1–11.
13. Yao S Shen P Dai T Dai F Wang Y Zhang W Lu HJI Human understandable thyroid ultrasound imaging AI report system—a bridge between AI and clinicians iScience 2023 26 4 106530 10.1016/j.isci.2023.106530 37123225
Yao S, Shen P, Dai T, Dai F, Wang Y, Zhang W, Lu HJI. Human understandable thyroid ultrasound imaging AI report system—a bridge between AI and clinicians. iScience. 2023;26(4): 106530.37123225
14. Yaqub M Kelly B Noble J Papageorghiou AJ Gynecology, OC05. 01: an AI system to support sonologists during fetal ultrasound anomaly screening Ultrasound Obstet Gyne 2018 10.1002/uog.19232
Yaqub M, Kelly B, Noble J, Papageorghiou AJ. Gynecology, OC05. 01: an AI system to support sonologists during fetal ultrasound anomaly screening. Ultrasound Obstet Gyne. 2018. 10.1002/uog.19232.
15. Browne JL, Pascual MÁ, Perez J, Salazar S, Valero B, Rodriguez I, Cassina D, Alcázar JL, Guerriero S, Graupera BJD. AI: can it make a difference to the predictive value of ultrasound breast biopsy? Diagnostics (Basel). 2023;13(4):811.
16. Tan M, Le Q. EfficientNet: rethinking model scaling for convolutional neural networks. In: Kamalika C, Ruslan S (eds) Proceedings of the 36th International Conference on Machine Learning. PMLR 2019;97:6105--14.
17. Ye X Wang J Chen Y Lv Z He S Mao J Xu J Shen L Automatic screening and identifying myopic maculopathy on optical coherence tomography images using deep learning Transl Vis Sci Technol 2021 10 13 10 10.1167/tvst.10.13.10 34751744
Ye X, Wang J, Chen Y, Lv Z, He S, Mao J, Xu J, Shen L. Automatic screening and identifying myopic maculopathy on optical coherence tomography images using deep learning. Transl Vis Sci Technol. 2021;10(13):10.34751744
18. Dan R Li Y Wang Y Chen X Jia G Wang S Ge R Qian G Jin Q Ye J Wang Y CDNet: contrastive disentangled network for fine-grained image categorization of ocular B-scan ultrasound IEEE J Biomed Health Inform 2023 27 7 3525 3536 10.1109/JBHI.2023.3271696 37126620
Dan R, Li Y, Wang Y, Chen X, Jia G, Wang S, Ge R, Qian G, Jin Q, Ye J, Wang Y. CDNet: contrastive disentangled network for fine-grained image categorization of ocular B-scan ultrasound. IEEE J Biomed Health Inform. 2023;27(7):3525–36.37126620
19. Parchand S Singh R Bhalekar S Reliability of ocular ultrasonography findings for pre-surgical evaluation in various vitreo-retinal disorders Semin Ophthalmol 2014 29 4 236 241 10.3109/08820538.2013.821506 24409949
Parchand S, Singh R, Bhalekar S. Reliability of ocular ultrasonography findings for pre-surgical evaluation in various vitreo-retinal disorders. Semin Ophthalmol. 2014;29(4):236–41.24409949
20. Demir G Arici M Alkin Z Preoperative evaluation of tractional retinal detachment with B-mode ultrasonography in diabetic vitreous hemorrhage Beyoglu Eye J 2021 6 1 49 53 35005492
Demir G, Arici M, Alkin Z. Preoperative evaluation of tractional retinal detachment with B-mode ultrasonography in diabetic vitreous hemorrhage. Beyoglu Eye J. 2021;6(1):49–53.35005492
21. Gottlieb M Holladay D Peksa GD Point-of-care ocular ultrasound for the diagnosis of retinal detachment: a systematic review and meta-analysis Acad Emerg Med 2019 26 8 931 939 10.1111/acem.13682 30636351
Gottlieb M, Holladay D, Peksa GD. Point-of-care ocular ultrasound for the diagnosis of retinal detachment: a systematic review and meta-analysis. Acad Emerg Med. 2019;26(8):931–9.30636351
22. Lahham S Shniter I Thompson M Le D Chadha T Mailhot T Kang TL Chiem A Tseeng S Fox JC Point-of-care ultrasonography in the diagnosis of retinal detachment, vitreous hemorrhage, and vitreous detachment in the emergency department JAMA Netw Open 2019 2 4 e192162 10.1001/jamanetworkopen.2019.2162 30977855
Lahham S, Shniter I, Thompson M, Le D, Chadha T, Mailhot T, Kang TL, Chiem A, Tseeng S, Fox JC. Point-of-care ultrasonography in the diagnosis of retinal detachment, vitreous hemorrhage, and vitreous detachment in the emergency department. JAMA Netw Open. 2019;2(4): e192162.30977855
23. He K Zhang X Ren S Sun J Spatial pyramid pooling in deep convolutional networks for visual recognition IEEE Trans Pattern Anal Mach Intell 2015 37 9 1904 1916 10.1109/TPAMI.2015.2389824 26353135
He K, Zhang X, Ren S, Sun J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell. 2015;37(9):1904–16.26353135
24. Han J Yao X Cheng G Feng X Xu D P-CNN: part-based convolutional neural networks for fine-grained visual categorization IEEE Trans Pattern Anal Mach Intell 2022 44 2 579 590 10.1109/TPAMI.2019.2933510 31398107
Han J, Yao X, Cheng G, Feng X, Xu D. P-CNN: part-based convolutional neural networks for fine-grained visual categorization. IEEE Trans Pattern Anal Mach Intell. 2022;44(2):579–90.31398107
25. Bera A Wharton Z Liu Y Bessis N Behera A SR-GNN: spatial relation-aware graph neural network for fine-grained image categorization IEEE Trans Image Process 2022 10.1109/TIP.2022.3205215 36103441
Bera A, Wharton Z, Liu Y, Bessis N, Behera A. SR-GNN: spatial relation-aware graph neural network for fine-grained image categorization. IEEE Trans Image Process. 2022. 10.1109/TIP.2022.3205215.36103441
26. Chen D Yu Y Zhou Y Peng B Wang Y Hu S Tian M Wan S Gao Y Wang Y Yan Y Wu L Yao L Zheng B Wang Y Huang Y Chen X Yu H Yang Y A deep learning model for screening multiple abnormal findings in ophthalmic ultrasonography (with video) Transl Vis Sci Technol 2021 10 4 22 10.1167/tvst.10.4.22 34004002
Chen D, Yu Y, Zhou Y, Peng B, Wang Y, Hu S, Tian M, Wan S, Gao Y, Wang Y, Yan Y, Wu L, Yao L, Zheng B, Wang Y, Huang Y, Chen X, Yu H, Yang Y. A deep learning model for screening multiple abnormal findings in ophthalmic ultrasonography (with video). Transl Vis Sci Technol. 2021;10(4):22.34004002
27. Feng L Zhang Y Wei W Qiu H Shi M Applying deep learning to recognize the properties of vitreous opacity in ophthalmic ultrasound images Eye (Lond) 2024 38 2 380 385 10.1038/s41433-023-02705-7 37596401
Feng L, Zhang Y, Wei W, Qiu H, Shi M. Applying deep learning to recognize the properties of vitreous opacity in ophthalmic ultrasound images. Eye (Lond). 2024;38(2):380–5.37596401
28. Wang Y Xu Z Dan R Yao C Shao J Sun Y Wang Y Ye J Automated classification of multiple ophthalmic diseases using ultrasound images by deep learning Br J Ophthalmol 2023 10.2139/ssrn.4258958 38164563
Wang Y, Xu Z, Dan R, Yao C, Shao J, Sun Y, Wang Y, Ye J. Automated classification of multiple ophthalmic diseases using ultrasound images by deep learning. Br J Ophthalmol. 2023. 10.2139/ssrn.4258958.38164563
29. Ito K Lye TH Dan YS Yu JDG Silverman RH Mamou J Hoang QV Automated classification and detection of staphyloma with ultrasound images in pathologic myopia eyes Ultrasound Med Biol 2022 48 12 2430 2441 10.1016/j.ultrasmedbio.2022.06.010 36096896
Ito K, Lye TH, Dan YS, Yu JDG, Silverman RH, Mamou J, Hoang QV. Automated classification and detection of staphyloma with ultrasound images in pathologic myopia eyes. Ultrasound Med Biol. 2022;48(12):2430–41.36096896
30. Hohmann E. Editorial commentary: Big data and machine learning in medicine. Arthroscopy. 2022;38(3):848–9.
31. Riley RD Ensor J Snell KI Debray TP Altman DG Moons KG Collins GS External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges BMJ 2016 353 i3140 10.1136/bmj.i3140 27334381
Riley RD, Ensor J, Snell KI, Debray TP, Altman DG, Moons KG, Collins GS. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016;353: i3140.27334381
32. Tian Y Chen W Zhou T Li J Ding K Li J Establishment and evaluation of a multicenter collaborative prediction model construction framework supporting model generalization and continuous improvement: a pilot study Int J Med Inform 2020 141 104173 10.1016/j.ijmedinf.2020.104173 32531725
Tian Y, Chen W, Zhou T, Li J, Ding K, Li J. Establishment and evaluation of a multicenter collaborative prediction model construction framework supporting model generalization and continuous improvement: a pilot study. Int J Med Inform. 2020;141: 104173.32531725
33. Yue H Liu J Li J Kuang H Lang J Cheng J Peng L Han Y Bai H Wang Y Wang Q Wang J MLDRL: multi-loss disentangled representation learning for predicting esophageal cancer response to neoadjuvant chemoradiotherapy using longitudinal CT images Med Image Anal 2022 79 102423 10.1016/j.media.2022.102423 35429696
Yue H, Liu J, Li J, Kuang H, Lang J, Cheng J, Peng L, Han Y, Bai H, Wang Y, Wang Q, Wang J. MLDRL: multi-loss disentangled representation learning for predicting esophageal cancer response to neoadjuvant chemoradiotherapy using longitudinal CT images. Med Image Anal. 2022;79: 102423.35429696
34. Akkus Z Cai J Boonrod A Zeinoddini A Weston AD Philbrick KA Erickson BJ A survey of deep-learning applications in ultrasound: artificial intelligence-powered ultrasound for improving clinical workflow J Am Coll Radiol 2019 16 9 Pt B 1318 1328 10.1016/j.jacr.2019.06.004 31492410
Akkus Z, Cai J, Boonrod A, Zeinoddini A, Weston AD, Philbrick KA, Erickson BJ. A survey of deep-learning applications in ultrasound: artificial intelligence-powered ultrasound for improving clinical workflow. J Am Coll Radiol. 2019;16(9 Pt B):1318–28.31492410
35. Selvaraju RR Cogswell M Das A Vedantam R Parikh D Batra D Grad-CAM: visual explanations from deep networks via gradient-based localization Int J Comput Vision 2020 128 2 336 359 10.1007/s11263-019-01228-7
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int J Comput Vision. 2020;128(2):336–59.
36. Li J Wang Y Wang S Wang J Liu J Jin Q Sun L Multiscale attention guided network for COVID-19 diagnosis using chest X-ray images IEEE J Biomed Health Inform 2021 25 5 1336 1346 10.1109/JBHI.2021.3058293 33560995
Li J, Wang Y, Wang S, Wang J, Liu J, Jin Q, Sun L. Multiscale attention guided network for COVID-19 diagnosis using chest X-ray images. IEEE J Biomed Health Inform. 2021;25(5):1336–46.33560995
