
==== Front
Nat Commun
Nat Commun
Nature Communications
2041-1723
Nature Publishing Group UK London

39223149
51888
10.1038/s41467-024-51888-4
Article
Learning co-plane attention across MRI sequences for diagnosing twelve types of knee abnormalities
http://orcid.org/0009-0002-8628-5061
Qiu Zelin 1
Xie Zhuoyao 2
http://orcid.org/0000-0002-6799-9352
Lin Huangjing 3
Li Yanwen 3
Ye Qiang 2
Wang Menghong 2
Li Shisi 2
Zhao Yinghua zhaoyh@smu.edu.cn

2
http://orcid.org/0000-0002-8400-3780
Chen Hao jhc@cse.ust.hk

145
1 https://ror.org/00q4vv597 grid.24515.37 0000 0004 1937 1450 Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China
2 grid.413107.0 Department of Radiology, The Third Affiliated Hospital of Southern Medical University (Academy of Orthopedics, Guangdong Province), Guangzhou, Guangdong China
3 AI Research Lab, Imsight Technology Co., Ltd., Shenzhen, Guangdong China
4 grid.24515.37 0000 0004 1937 1450 Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Hong Kong, China
5 https://ror.org/00q4vv597 grid.24515.37 0000 0004 1937 1450 Division of Life Science, The Hong Kong University of Science and Technology, Hong Kong, China
2 9 2024
2 9 2024
2024
15 76372 8 2023
17 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Multi-sequence magnetic resonance imaging is crucial in accurately identifying knee abnormalities but requires substantial expertise from radiologists to interpret. Here, we introduce a deep learning model incorporating co-plane attention across image sequences to classify knee abnormalities. To assess the effectiveness of our model, we collected the largest multi-sequence knee magnetic resonance imaging dataset involving the most comprehensive range of abnormalities, comprising 1748 subjects and 12 types of abnormalities. Our model achieved an overall area under the receiver operating characteristic curve score of 0.812. It achieved an average accuracy of 0.78, outperforming junior radiologists (accuracy 0.65) and remains competitive with senior radiologists (accuracy 0.80). Notably, with the assistance of model output, the diagnosis accuracy of all radiologists was improved significantly (p < 0.001), elevating from 0.73 to 0.79 on average. The interpretability analysis demonstrated that the model decision-making process is consistent with the clinical knowledge, enhancing its credibility and reliability in clinical practice.

The authors present a deep learning model that incorporates co-plane attention across image sequences with a performance comparable to senior radiologists in classifying 12 knee abnormalities from MRI. The model significantly improves diagnostic performance and aligns with clinical observations.

Subject terms

Magnetic resonance imaging
Machine learning
Data processing
501100010877 Shenzhen Science and Technology Innovation Commission SGDX20210823103201011 501100005847 Food and Health Bureau of the Government of the Hong Kong Special Administrative Region | Health and Medical Research Fund (HMRF) 20211021 501100011002 National Science Foundation of China | National Natural Science Foundation of China-Yunnan Joint Fund (NSFC-Yunnan Joint Fund) 81871510 501100011002 National Science Foundation of China | National Natural Science Foundation of China-Yunnan Joint Fund (NSFC-Yunnan Joint Fund) 82172014 501100003453 Natural Science Foundation of Guangdong Province (Guangdong Natural Science Foundation) 2023A030313574 issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

The knee joint is a complex hinge joint and one of the main load-bearing joints of the human body, supporting the completion of complex movements in daily activities1. Various knee abnormalities can arise from aging or injury, including meniscal tears, cartilage damage, anterior cruciate ligament tear, posterior cruciate ligament injury, medial collateral ligament injury, lateral collateral ligament injury, patellar retinaculum injury, infrapatellar fat pad injury, synovial plica, cysts (such as popliteal cyst, ganglion cyst, meniscus cyst, and synovial cyst), bone contusion, and joint effusion2,3. These 12 types of knee abnormalities are relatively common and often result in knee pain and dysfunction4, significantly impacting patients’ quality of life and psychological well-being and imposing a substantial social cost5–7. Different types of knee abnormalities often occur together but are best treated with quite different methods: mild abnormalities (e.g., bone contusion, infrapatellar fat pad injury, and effusion) usually require only conservative treatment and follow-up observation; and severe abnormalities (e.g., anterior cruciate ligament tear and meniscal tear) require surgical treatment8,9. Therefore, the accurate diagnosis of knee abnormalities is crucial for developing personalized treatment plans and improving the overall quality of life for patients.

Arthroscopy is widely considered the gold standard for diagnosing the pathology of knee abnormalities10. However, this inspection method cannot detect some knee abnormalities located outside the joint cavity (e.g., bone contusion, infrapatellar fat pad injury, and patellar retinaculum injury)11 and carries the risk of additional trauma and serious complications (e.g., joint infection and deep vein thrombosis)12–14. Magnetic resonance imaging (MRI) is a non-invasive method for accurate evaluation of knee pathology and provides results comparable to those of arthroscopy15. In addition to conventional T1- and T2-weighted (T1W and T2W) imaging sequences, knee MRI requires proton density-weighted (PDW) imaging sequence because of its high signal-to-noise ratio and spatial resolution to accurately detect abnormalities16,17. However, knee MRI interpretation is very time-consuming and labor-intensive. Some subtle lesions of the knee joint can easily be overlooked by radiologists with insufficient work experience. Therefore, automatic methods have a huge clinical demand for accurate diagnosis of knee abnormalities.

In recent years, several automated approaches, including classical methods and DL models, have been proposed to assist in knee abnormality diagnosis from MRI18–23, with the corresponding knee MRI dataset collected to facilitate related research24,25. However, the existing studies mainly focused on a few common abnormalities, such as tears in the meniscal and anterior cruciate ligaments, which limits the model when adapting to more complex real-life cases.

In this paper, we construct the largest knee MRI dataset involving the most comprehensive range of knee abnormalities. We gathered data from 1748 patients covering 12 types of knee abnormalities from five clinical centers. The images and annotations that combine arthroscopy results are utilized in developing our deep learning model using Co-Plane Attention across MRI Sequences (CoPAS). In our experiments, the model outperforms other methods and demonstrates the ability to assist radiologists by providing competitive results with senior radiologists, subsequently improving the diagnostic accuracy of junior radiologists.

Results

Study design

The overview of this work is illustrated in Fig. 1. We first constructed a multi-center knee MRI dataset to develop and later evaluate our model. In total, 1748 patients were collected from five clinical centers in China: The Third Affiliated Hospital of Southern Medical University, The Seventh Affiliated Hospital of Southern Medical University, Zhujiang Hospital of Southern Medical University, Foshan Hospital of Traditional Chinese Medicine, and The Fifth Affiliated Hospital of Sun Yat-sen University, denoted as center A to E, respectively. The images include PDW sequences taken from sagittal, coronal, and axial planes, the coronal T1W, and the sagittal T2W sequences.Fig. 1 Overview of this study.

First, we constructed a multi-center knee MRI dataset collected from five clinical centers and encompassed 1748 patients in total. The dataset comprises multiple MRI sequences, including PDW, T1W and T2W. Twelve knee abnormalities are covered in this study, where the patient-level annotations are obtained by combining MRI and arthroscopy results. We develop a deep-learning model (CoPAS) based on the dataset and conduct evaluations with radiologists. The results demonstrate its competitiveness compared to senior radiologists and the potential to enhance the accuracy of radiologists' diagnoses. This figure was created with BioRender.com released under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International license (https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en).

The data in this study covers 12 types of knee abnormalities: meniscal tear (MENI); anterior cruciate ligament tear (ACL); cartilage damage (CART); posterior cruciate ligament injury (PCL); medial collateral ligament injury (MCL); lateral collateral ligament injury (LCL); joint effusion (EFFU); bone contusion (CONT); synovial plica (PLICA); cyst (CYST); infrapatellar fat pad injury (IFP); and patellar retinaculum injury (PR). The inclusion criteria and brief description of the abnormalities are illustrated in Table 1. Specifically, we only included patients who underwent knee arthroscopy. Patients without preoperative MRI examination data or patients with MRI images containing artifacts (motion artifacts or metal artifacts) were excluded. Patient-level annotations of the abnormalities were presented by experienced doctors using both arthroscopy and MRI. More details about the dataset, including MRI sequence setting, scanning protocols, and annotations, can be found in the supplementary materials.Table 1 Brief descriptions of twelve types of knee abnormalities in this study

Types	Brief descriptions	
Meniscal tear (MENI)	Intact (normal or degenerative changes without tear) or tear (tears of the medial, lateral, anterior horn, posterior horn, body, or root of the meniscus) diagnosed with arthroscopy51	
Anterior cruciate ligament tear (ACL)	Intact (normal, mucoid degeneration, and sprain) or tear (incomplete tear or complete tear) diagnosed with arthroscopy52	
Cartilage damage (CART)	Intact (cartilage surface intact) or damage (cartilage surface damage including cracks, fissures, fibrillation, fragmentation, or bone exposed on the patella, femora, or tibia) diagnosed with arthroscopy53	
Posterior cruciate ligament injury (PCL)	Normal or injury (mucoid degeneration, sprain, partial tear, or complete tear, which presented as hypointensity on T1-weighted images and hyperintensity on T2-weighted images or proton density-weighted images within the ligament in at least 1 slice) diagnosed with MRI54	
Medial collateral ligament injury (MCL)	Normal or injury (mucoid degeneration, sprain, partial tear, or complete tear, which presented as hypointensity on T1-weighted images and hyperintensity on T2-weighted images or proton density-weighted images within the ligament in at least 1 slice) diagnosed with MRI55	
Lateral collateral ligament injury (LCL)	Normal or injury (mucoid degeneration, sprain, partial tear, or complete tear, which presented as hypointensity on T1-weighted images and hyperintensity on T2-weighted images or proton density-weighted images within the ligament in at least 1 slice) diagnosed with MRI56	
Joint effusion (EFFU)	Absence or presence (fluid in the articular space and bursae with hypointensity on T1-weighted images and hyperintensity on T2-weighted images or proton density-weighted images) diagnosed with MRI15	
Bone contusion (CONT)	Absence or presence (wedge-shaped or hemispherical regions in the bone marrow with hypointensity on T1-weighted images and hyperintensity on T2-weighted images or proton density-weighted images) diagnosed with MRI57	
Synovial plica (PLICA)	Absence or presence (band-like structures of hypointensity and variable size and thickness on T2-weighted images or proton density-weighted images) diagnosed with MRI58	
Cyst (including popliteal cyst, ganglion cyst, meniscus cyst, synovial cyst, and bone cyst)	Absence or presence (encapsulated fluid collections around the knee with hypointensity on T2-weighted images or proton density-weighted images) diagnosed with MRI59	
Infrapatellar fat pad injury (IFP)	Normal or injury (areas of hyperintensity within Hoffa’s fat pad on sagittal plane) diagnosed with MRI60	
Patellar retinaculum injury (PR)	Normal or injury (increased signal within the ligament with intact borders; partial or total fiber discontinuity in at least 1 slice on T2-weighted images or proton density-weighted images) diagnosed with MRI61	

In our experiments, the patients from center A (n = 1103) and center B-E (n = 645) were included as the internal dataset and the external set. We separate the internal dataset with a ratio of 7:1:2, resulting in 773 patients for training, 110 for validation, and 220 for testing. We trained and assessed the diagnosis performance of our CoPAS model against existing DL methods using the internal data and conducted the ablation study. The generalization ability is evaluated on the external data with different sequence settings. Up to six human radiologists, including junior and senior professionals, are involved in simulating and verifying the effectiveness of the model’s assistance in the clinical environment. Their involvement includes independently providing their assessments initially and subsequently re-evaluating along with our model’s results. Finally, we validated the effectiveness and compatibility of our model design by examining the output patterns and comparing them to the empirical diagnostic preferences observed in clinical settings. Details of the experiment steps will be described in the method section and supplementary materials.

Accurate and generalizable knee abnormality diagnosis by CoPAS

We evaluate the diagnostic performance of our model and make comparisons against some existing knee abnormality diagnosis methods. We adapt three existing methods to our tasks, MRNet24, MPFuseNet26, and ELNet27. (See supplementary materials for the details of adaptation). The diagnostic task is formulated as twelve binary classifications, as shown in Table 2, where we report the area under the receiver operating characteristic curve (AUC-ROC) as an evaluation metric. The results show that our method outperforms other models with an average AUC-ROC of 0.812. Specifically, our CoPAS outperformed the three extant models in 8 out of 12 abnormalities, demonstrating the potential efficacy of our attention-based framework. MRNet exhibits greater accuracy in detecting medial collateral ligament tear (MCL) and effusion (EFFU), while MPFuseNet attains the highest score in anterior cruciate ligament tear (ACL).Table 2 Comparison with other methods in the multi-center dataset

Method	MENI	ACL	CART	PCL	MCL	LCL	EFFU	CONT	PLICA	CYST	IFP	PR	Average	
Center A (PDW+T1W+T2W)	
MRNet (Baseline)24	0.659	0.950	0.778	0.759	0.802	0.763	0.789	0.820	0.759	0.819	0.732	0.744	0.781	
MPFuseNet26	0.686	0.963	0.767	0.757	0.782	0.672	0.751	0.774	0.664	0.778	0.744	0.717	0.754	
ELNet27	0.744	0.902	0.750	0.714	0.773	0.738	0.694	0.810	0.766	0.770	0.684	0.754	0.760	
CoPAS (Ours)	0.763	0.949	0.790	0.800	0.782	0.816	0.787	0.821	0.877	0.862	0.734	0.767	0.812	
Center B+C (PDW)	
MRNet	0.648	0.717	0.682	0.715	0.730	0.743	0.697	0.682	0.590	0.679	0.723	0.718	0.694	
MPFuseNet	0.618	0.739	0.656	0.680	0.694	0.736	0.646	0.608	0.623	0.715	0.682	0.672	0.672	
ELNet	0.674	0.705	0.672	0.703	0.699	0.753	0.688	0.663	0.617	0.695	0.670	0.717	0.688	
CoPAS	0.688	0.755	0.692	0.731	0.722	0.789	0.718	0.698	0.666	0.741	0.715	0.732	0.721	
Center D+E (PDW+T1W/T2W)	
MRNet	0.664	0.701	0.653	0.722	0.724	0.613	0.726	0.676	0.740	0.757	0.743	0.718	0.703	
MPFuseNet	0.681	0.718	0.658	0.741	0.734	0.661	0.700	0.663	0.733	0.735	0.622	0.693	0.695	
ELNet	0.626	0.696	0.624	0.737	0.758	0.543	0.708	0.622	0.750	0.756	0.745	0.697	0.689	
CoPAS	0.690	0.760	0.676	0.750	0.772	0.658	0.739	0.675	0.748	0.771	0.745	0.723	0.726	
The area under ROC curve (AUC-ROC) is the reported metric. Our model is superior in most of the classes and the average AUC-ROC. Highest scores are indicated in bold text.

In addition, we evaluate the generalizability of our model with different sequence combinations on the external dataset. We conducted cross-validation by training on the internal dataset using the same sequence settings as the external dataset, and testing on the external data. The result is shown in Table 2, a decline of the average AUC-ROCs from 0.812 to 0.721 and 0.726 is observed when transitioning from the internal dataset to the two external datasets. The results indicate a strong reliance on T1W and T2W sequences for diagnosing bone contusion (CONT), as evidenced by a drop in AUC-ROC from 0.821 to 0.698. Surprisingly, certain abnormalities that could be better diagnosed using T1W and T2W MRI (e.g., IFP) do not exhibit a significant decline in performance. We assume that the PDW images contained relevant information but were subtle to be seen with bare eyes, which can be captured by the model.

In our point of view, the performance drop is due to fewer sequences available in the external dataset, as well as the data distribution shifting between different centers caused by multiple factors including scanning parameters and patient demographics. In our further exploration of different subgroups (see supplementary file), we found that both unbalanced sex and age have influence on the model performance. Nevertheless, our model still maintains its leading position in all classes against other methods, showing its robustness in encountering multi-center changes. Our ablation study (see supplementary materials for details) also illustrates the performance gain from the components in CoPAS, including the innovative co-plane attention learning modules. The results demonstrate the model’s proficiency in incorporating spatial information and identifying sequence-specific information, thereby promoting the classification outcomes.

CoPAS offers effective diagnostic assistance to radiologists

To further explore the potential of our model in clinical settings, we compare the classification results of CoPAS with those of both senior and junior radiologists. The internal data results are illustrated in Fig. 2, involving one senior and one junior radiologist. The classification accuracy (ACC) for each abnormality is reported. Our model surpasses the junior radiologist on all abnormalities and outperforms the senior radiologist in 5 out of the 12 abnormalities (i.e., MENI, ACL, CART, PCL, and MCL), but it has slight drawbacks in terms of average accuracy (ACC 0.78 to 0.80). Notably, our model demonstrates higher accuracy in tasks involving complex spatial morphology, such as the cruciate ligament, which is visible across multiple slices. This indicates that our model excels in handling spatial relations between slices and sequences compared to the radiologists.Fig. 2 The diagnosis accuracy in our internal testing set.

CoPAS, the junior radiologist, and the senior radiologist achieved average accuracy of 0.78, 0.65, and 0.80, respectively. Our model surpasses the junior radiologist on all abnormalities. It outperforms the senior radiologist in nearly half of the classes. Note that our model appears to be more accurate for complex cruciate ligament tears (e.g., ACL, PCL, MCL, and LCL), which is attributed to its ability to capture spatial information through co-plane attention. Source data are provided as a Source Data file.

The same experiments are conducted in external data involving six radiologists in a subgroup that sampled 220 cases with the same class distribution as the internal dataset. Additionally, we evaluated the model’s ability to assist human radiologists in diagnosis by comparing the performance of radiologists with and without the model’s classification output. We found that, despite the model performance decline due to the data variations across multiple centers, CoPAS still outperforms certain senior radiologists. Moreover, it significantly enhances the diagnostic performance of all radiologists with its classification output. Detailed results are reported in Supplementary material. The results demonstrate that our CoPAS is competitive with senior radiologists. Most importantly, it highlights the model’s capability to provide powerful assistance in diagnosing complex knee abnormalities in clinical practice.

Consistency found between CoPAS and clinical knowledge

Considering the distinctive nature of MRI’s multi-sequence characteristic, it is evident that images from different planes can provide variant perspectives and play diverse roles in the diagnosis of abnormalities. Each knee abnormalities have its’ optimal diagnosis plane, for example, it is recommended that meniscal tears be evaluated first on sagittal MR images and then on coronal images28. Because sagittal images can detect 97% of medial meniscal tears and 96% of lateral meniscal tears. Assessment of the anterior and posterior cruciate ligaments is typically performed on sagittal T2 images29. The medial and lateral collateral ligaments are best visualized on coronal images30. Therefore, in CoPAS, we designed a specialized adaptable matrix to excavate the correlations between MRI planes and abnormalities.

We resort to an empirical table provided by radiologists with their preference in choosing MRI planes during abnormality diagnosis. As shown in Fig. 3(a), we use lines to link the abnormalities (from the left) with planes (to the right), with thicker lines indicating a higher likelihood of observing this abnormality in that particular plane. Similarly, we measure the certainty level and accuracy of the model output from each plane, and demonstrate in Fig. 3(b) and (c). For quantitative comparison, we treat each abnormality as a series and calculate the corresponding correlation coefficient, as presented in Table 3. The detailed quantification method can be found in supplementary materials.Fig. 3 The consistency in plane-abnormality correlations.

Demonstrating the correlation derived by a clinical observation probability, b model prediction certainty, and c model prediction accuracy. Thicker lines indicate the stronger relations between the abnormality and plane. The similar patterns in black boxes suggest that the model has learned a diagnosis preference that aligns with clinical observations. Source data are provided as a Source Data file.

Table 3 Quantitative analysis of the correlation coefficient between clinical observation probability and our prediction results in each plane

	MENI	ACL	CART	PCL	MCL	LCL	EFFU	CONT	PLICA	CYST	IFP	PR	
Accuracy	0.998	0.069	0.803	0.961	0.999	0.778	0.721	0.989	-0.629	0.690	1.000	0.732	
Certainty	0.898	0.004	0.995	0.990	−0.940	0.738	0.711	-0.697	0.998	0.922	0.971	−0.735	
Noting that certain classes (ACL, MCL, CONT, and PR) exhibit opposite trends because the model yields similar results in three planes.

Generally, there are noticeable similar patterns in Fig. 3, where we marked out with black boxes. In most cases, there is a high positive correlation between the model results and clinical preference, except for CONT and PLICA. Meanwhile, the prediction certainty of each plane matches the probability of observation in the clinic, while the classes ACL, MCL, CONT, and PR show the opposite results. This is because the three planes contribute equally to the model, given the close certainty values. In essence, our model exhibits plane-related preferences similar to those of human radiologists. It has derived a set of corresponding rules that maximizes the prediction probability during decision-making, which allows it to produce more reliable results during clinical implementation.

Discussion

In this paper, we create a multi-center multi-sequence knee MRI dataset to conduct comprehensive research on an attention-based model for assisting knee abnormality classification. Our model has established a benchmark for this dataset, which will prove beneficial for future research endeavors.

In our experiment, we conclude that the high performance of our model benefits from the multi-task framework design. This framework not only mitigates the computational demands of large single-task ensemble models but also facilitates learning by enabling the sharing of feature representations between tasks. However, we found that single-task methods, such as the MRNet, may outperform our approach in some tasks. The underlying reason is that certain characteristics of these abnormalities may be misguided by other tasks in our multi-task model, but more discriminate for a single-task network, as previously noted. For instance, the intensity enhancement of effusion resembles that of a cyst. Our ablation study with single-task variants verified this assumption. Notably, our approach attains optimal performance in patellar retinaculum injury (PR) cases, which have a limited number of training samples. This achievement highlights the efficacy of our multi-task model, which can identify hidden correlations among abnormalities, thereby enhancing robustness and ensuring high performance when working with small data.

In addition, the incorporation of co-plane attention in our model makes a significant contribution to classification accuracy. The experiment also demonstrates the high clinical value of our model, as its successful implementation can alleviate the burden on radiologists and improve their diagnostic accuracy. Moreover, our findings reveal consistency between clinical observations and the predictions made by our model. This not only demonstrated interpretability in our model’s predictions but also increases the possibility of identifying and validating new clinical insights, such as abnormality-plane correlations.

Here, we illustrated the region of interest in the decision-making process of the model with the class activation mapping (CAM) technique. As shown in Fig. 4, we leverage EigenCAM31 with no class discrimination and GradCam++32 to illustrate the activated regions in three planes on a patient with MENI, ACL, CART, and EFFU. Our model successfully classified the MENI, ACL, CART, and EFFU, but misclassified MCL and PLICA. The model exhibits distinct spatial preferences across different planes. Specifically, in the EigenCAM image, the cartilage in the sagittal and coronal planes is highlighted, which is easy to identify as worn-out cartilage. In the sagittal plane, other activated regions indicate the model’s focus on the meniscus and suspected effusion, with corresponding activation observed in the axial plane as well.Fig. 4 Illustration of class activation map on a patient with MENI, ACL, CART, and EFFU.

The EigenCAM with no class discrimination and Grad-CAM++ are utilized. The model correctly classified MENI, ACL, CART, and EFFU but yielded false positives in MCL and PLICA. We can see that the active region shows a spatial correlation (e.g., cartilage in sagittal and coronal plane, collateral ligament in coronal and axial plane) which indicates the model can learn the spatial information from other planes.

Additionally, the collateral ligaments in the coronal and axial planes are activated simultaneously. However, it is worth noting that this leads to a false positive prediction on MCL. These pairs of activation across different planes illustrate that the model is effectively learning the spatial relationships among the three planes with co-plane attention. Even though the axial plane may not be the most optimal choice for accurately classifying MENI and ACL, the attention mechanism enhances the spatial-related information and enables the model to identify subtle yet similar features in the axial plane.

However, it is important to acknowledge several limitations in this study. Firstly, compared to other classification models, CoPAS has a relatively large number of parameters due to its utilization of separate encoders for MRI sequences. Consequently, training the model with limited data may take more time to converge and be prone to overfitting, posing challenges in its implementation. Second, the requirement for MRI scans from three different planes limits the flexibility of the model. The experiment has shown that incorporating more sequences leads to performance improvement. However, in practice, acquiring a dataset with such a comprehensive sequence set can be challenging.

Furthermore, the model presented in this paper is trained in a fully-supervised manner, which relies on labeled data for training. However, considering the risks associated with arthroscopy procedures and the substantial workload for annotating images, it becomes impractical to collect large-scale datasets with full annotation. There are two aspects in the future work addressing this problem. First, it would be preferable to train the model in an unsupervised or semi-supervised manner by utilizing a combination of limited labeled data and a larger quantity of potentially unlabeled data. Second, we can shorten the label-making time of the radiologists. Intuitively, leveraging the CAM technique with a pre-trained model to generate a heat map indicating the areas of potential abnormalities, we can significantly expedite the diagnosis process.

Methods

This study received ethical approval from the Institutional Review Board of The Third Affiliated Hospital of Southern Medical University (No. 201501003) and was also approved by centers B, C, D, and E. Informed consent was waived due to the retrospective nature of the study and anonymity of the analyzed data. This study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines33.

Previous works on automatic knee abnormality classification

MRI sequences are three-dimensional volumes that consist of a series of single-channel slices. Traditional image analysis methods that rely on hand-crafted features are less effective in handling complex 3D images. Deep learning can automatically model the intricate relations between high-dimensional data and labels, making it feasible for many challenging tasks in medical image analysis34,35. It has sparked research interest in knee abnormality classification20–22. For example, Bien et al.24 proposed MRNet, which separately predicts abnormalities from three MRI planes, and combines their results by max pooling. This kind of method that assigns each abnormality to one specific network requires large computational resources under a multi-abnormality setting. Similarly, ensemble models, such as in Ref. 36, are usually criticized for being redundant and lacking immediate explanation.

Some other methods are specifically designed for specific abnormality types (e.g., cruciate ligament tears)37,38. The specific tailoring of these single-task models presents challenges for their transferability to other tasks, thereby limiting their applicability in clinical practice. Therefore, utilizing multi-task learning is more realistic and can benefit from the shared features39. However, a common phenomenon in multi-task learning is that the model may be misled when trying to optimize for all tasks at once due to the task heterogeneity and experiences negative transfer of the performance40.

In previous works, the number of MRI sequences is usually less than three, which means they rarely need to pay attention to the complex integration strategy of the inputs. Belton et al.26 explored different strategies for fusing multi-planar MRI. The paper uses simple concatenation for late fusion. This equal-contribution structure ignores the implicit correlations in low-level features and thus makes it vulnerable to noise when the number of sequences increases. Characterizing the inter-sequence representations with attention learning, as in our work, will help in the selection of the most representative features and achieve better classification results.

Model overview

An overview of our model is illustrated in Fig. 5. The original data comprises PDW MR scans in different planes (sagittal, coronal, and axial) and two sequences with different contrasts (T1W in the coronal plane and T2W in the sagittal plane). All images are pre-processed into cubic volumes to serve as the model input, resulting in five original volumes (on the left) and six synthetic volumes obtained by rotating PDW volumes (on the right). Notice that this set of images is a specific case to demonstrate and verify the capacity of our model on the internal dataset. In other words, we are proposing a general solution for knee abnormality diagnosis using a set of multi-planar multi-contrast MRIs.Fig. 5 The pipeline of our approach.

We perform rotation on the PDW volumes to other planes. The volumes will be fed into three network branches (as shown on the bottom left) representing three planes, and integrated by co-plane attention learning (details are shown in Fig. 6). The predictions from the three branches will be further integrated using a probability matrix, which can be analyzed to explore the correlation between the planes and abnormalities. This integration process will ultimately yield the diagnostic prediction for the patient. This figure was created with BioRender.com released under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International license (https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en).

We draw inspiration from clinical knowledge that different MRI planes account differently for diagnosis, we design three branches to extract cross-plane information from three MRI planes, denoted by blue, green, and orange. Detailed network structures are illustrated in Fig. 6. The volumes from different planes are fed into their respective branches, allowing for the integration of information by cross-plane and cross-sequence attention. The predictions from each plane will then be integrated by a probability matrix that characterizes the joint distribution of the classified abnormalities. Finally, a correlation mining module is employed to obtain the final classification result for the 12 abnormalities.Fig. 6 Detail of the three-branch network structure.

The coronal, sagittal, and axial branches (represented by orange, blue, and green blocks) aim at different planes. For all branches, the cross-plane volume will be encoded by a weight-sharing encoder (ResNet3D), and cross-plane spatial information will be decoupled from each volume and integrated using an attention mechanism. Notably, for sagittal and coronal branches, the PDW features will join the T2W and T1W features for co-plane cross-sequence integration. In this way, the three branches will yield individual prediction results.

Pre-processing with image cropping and volume rotation

Firstly, to establish a reference point for the knee, we segment the meniscus by U-Net41 as an upstream task. Note that a coarse segmentation is adequate because we only need a rough region of interest (ROI) to identify the center of the meniscus. We crop the original images according to the segmentation mask to remove irrelevant areas and reduce the overall size of the image.

In a clinical setting, the utilization of multi-planar MRI offers a comprehensive perspective for diagnosing abnormalities. It is important to recognize that each MRI plane contains specific information that is crucial for decision-making purposes. Therefore, we tailor three individual branches in our model to specialize and extract features from different planes. Given that the slice thickness (i.e., the distance between two scanned slices) is relatively larger, resulting in anisotropic spatial resolution across different axes. Conventionally, using interpolation will reduce this imbalance, but the interpolation with one image itself does not provide any information gain and may introduce additional errors.

To solve this problem, we incorporate the information from other sequences by leveraging cross-attention learning. Specifically, we utilize the high-resolution information from other MRI volumes obtained in orthogonal planes. As shown in Fig. 5, the sagittal volume (224 × 224 × 24) has a high resolution in the sagittal plane but only has 24 slices. When the left side of this volume is rotated to the front, it can be regarded as a coronal-plane image of 24 × 224 × 224 (note that this is still the same volume). We believe that this synthetic coronal plane volume (denoted as ‘Sag → Cor’ in Fig. 5) from the sagittal plane contains complement information in the coronal plane. Therefore, we generate eleven volumes (nine PDW, one T1W, and one T2W) in total from five images. Simply, we name the synthetic volume ‘cross-plane volumes’ as they are converted from other MRI planes. Recall also that, essentially, there are only three different PDW volumes.

Cross-plane attention enhancing spatial information

The success of the deep learning model is attributable to its capability of automatically extracting high-dimensional features with rich semantic information. Attention mechanisms42, inspired by human vision, have shown their power in eliminating the interference of irrelevant features. Similarly, we can adopt cross-attention to enhance the spatial information from cross-plane volumes. Figure 6 demonstrates the detailed structures of the three branches of our model.

We will take the coronal branch as an example. As shown in Fig. 6, the coronal image is regarded as the main image, while the volume converted from the sagittal and axial images serves to provide additional information. Our model utilizes ResNet3D with 18 layers43 as the basic encoder for the network. The 3-dimensional convolution in ResNet3D enables the learning of patterns from the third dimension44, which, in our case, corresponds to the spatial information along slices. To prioritize the main volume for plane-specific feature extraction, we adopt weight sharing in the encoder across all volumes. This means that the weights are initially updated based on the main volume, allowing the network to locate lesions guided by high-resolution slices.

Moreover, the final pooling layer operates on the last two dimensions to preserve depth-wise information, which is then transposed. That is, the feature map will be pooled and reshaped from RC×D×H×W to RD×C, where C, D, H, and W represent the channel, depth, height, and width of the feature. This will transform the volumes into embeddings while keeping the order of slices, i.e., generate the representations of each slice. In this way, the model can learn to decouple three plane-specific features that are dominated by the three MRI planes.

To alleviate the information loss from the large slice thickness, we further integrate the cross-plane information by characterizing the spatial correspondence with attention. As illustrated in Fig. 6, a linear transformation with weight WQ is applied to the main feature to obtain the query Q. Then we use separate transformations with weight WKi and WVi to obtain Ki and Vi which represent keys and values. The attention output F is then calculated as follows:1 F=∑n=12SoftmaxQKnTdKnVn,

where dKn=C represents the dimension of the keys. The features will pass through normalization and linear layers. We then add the original main feature for the model to learn residues between the original and attention features. In this way, the model will identify and absorb similar features from the cross-plane volumes, thereby enhancing the features with rich spatial information that captures more fine-grained intensity changes along the slices.

Learning anatomical information with co-plane cross-sequence attention

Considering the PDW images carry more informative details about abnormalities during the diagnostic process. On the other hand, T1W and T2W images provide a clear view of anatomical structures. To leverage the advantages of each image type, we introduce T1W and T2W as the contrast reference to PDW to demonstrate the basis of multi-sequence attention learning in our model. In conventional feature fusion strategies, the auxiliary features are usually added to the main features using methods such as summation or concatenation. However, given that sequences of different contrasts contain both sequence-specific and shared information, we propose to leverage their feature attention to filter out irrelevant features and, therefore, improve the accuracy of the model.

As illustrated in Fig. 6, we utilize the contrast between two sequences to calculate the corresponding channel-wise attention. Unlike the cross-plane attention, which can be regarded as an enhancement of the main image to provide details in the depth dimension, the T1W and T2W sequence acts more like a filter to re-weight different features. Therefore, considering the feature extractors should have individual weights other than those we used in the PDW sequences, the encoder in this module is trained separately to reduce computational complexity.

To be specific, a linear layer will learn the sequence-specific information with the guidance of the co-plane information from upstream PDW volumes and project it as attention factors to the mixture of two features. Here, we employ the sum operation to combine the two features, as we believe that the model can learn by adding the corresponding features and guiding the integration process.

Plane-aware feature integration with abnormality probability matrix

Considering the attention mechanism will naturally enhance the similar (closer) features, the extracted features are still predominantly influenced by the main volume. Therefore, it is undeniable that each branch in our model retains plane-specific features. As mentioned earlier, the radiological diagnosis of abnormalities is closely linked to the MRI planes. In other words, certain abnormalities exhibit distinct characteristics in specific MRI planes. In our model, the prediction from each branch does not directly correspond to the final outcome. This situation is akin to multi-instance learning, where only the patient-level label is available for training. Since we want to classify multiple abnormalities at once, it is challenging for the model to learn the relations with only the concatenation of features that lack task-related guidance. Wang et al.45 proposed a method of fusing multi-view information by discovering correlations within class labels. Similarly, in terms of the MRI, we propose a fusion strategy for the three branches in our model to excavate the relations between MRI planes and abnormalities.

Figure 5 shows the predictions from the different branches representing multiple planes marked by orange, blue, and green. We form a plane-aware matrix by producing the dot product of each element from the predictions. The element xi,j,k is calculated by multiplying the ith, jth, and kth elements in sagittal, coronal, and axial predictions, which represent the possibility of a corresponding abnormality. Thus, the matrix contains both inter-class correlation, which is learned from multi-task learning, and abnormality-plane correlation information. We then use the correlation mining module to excavate the hidden pattern, i.e., the distribution of this probability. The diagonal of the plane-aware matrix represents the joint probability of each class from three planes, which is treated as the base prediction in this module. Meanwhile, a 12-channel convolution is used to learn position-related information in the matrix. Similar to SENet46, the matrix is squeezed into a vector, and then linear transformation is applied to generate the weight of basic predictions. Then the final prediction is yielded by scaling the diagonal vector with the weight. In the discussion section, we analyze the model predictions with CAM by backwarding from the final prediction.

As the outputs of the three branches are expected to be the probability of abnormalities, we apply supervision on both the final result and the branch outputs. Specifically, we use Focal Loss47 to measure the distance between the final result y and label y^, which can help the model to focus on the difficult samples. For the prediction of three branches ybranch, the binary cross entropy (BCE) loss is applied. We use hyper-parameters α to balance the losses, and the total loss can be formulated as follows:2 Ltotal=α∑branch∑i=112BCE(ybranchi,y^i)+∑i=112FL(yi,y^i).

Observer study

The clinical applicability of the model was evaluated in an observer study with six radiologists on the external test set II. Four junior musculoskeletal radiologists (C.X.Q., S.Y.Y., Z.Y.X., and Y.W.D., with 2, 2, 5, and 7 years of experience, respectively) and two senior musculoskeletal radiologists (C.L.L. and Y.H.Z., with 10 and 30 years of experience, respectively) who were blind to the reference standard independently performed MRI interpretations. They viewed MR images using RadiAnt DICOM Viewer software (Version 2020.1, Medixant, Poland) and performed binary classifications for each type of knee abnormality. After a washout period of 30 days, they performed repeated interpretations with model assistance. The comparison was performed between radiologists with and without model assistance.

Statistical analysis

In this paper, the Kruskal-Wallis test was used to compare the ages of patients in different centers, and the chi-square test was used to compare sex. Patient demographics were analyzed using SPSS software (version 23.0, IBM, Armonk, NY). The experiments are conducted with the evaluation metrics of AUC-ROC and ACC. The threshold value for deciding whether a patient has an abnormality is determined by maximizing the F1 score. If not otherwise stated, the significance of the difference in AUC-ROC and ACC is validated by Delong-test48 and T-test, respectively, with P ≤ 0.05 as significant. In experiments that involve radiologists, the 95% confidence intervals were extracted using bootstrapping with 1000 redraws. Fleiss’ kappa score49 is calculated to ensure the inter-reader agreement in dataset labeling.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Supplementary information

Supplementary Information

Peer Review File

Reporting Summary

Source data

Source Data

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-024-51888-4.

Acknowledgements

This work was supported by the Shenzhen Science and Technology Innovation Committee Fund (No. SGDX20210823103201011), received by H.C., Health and Medical Research Fund (No. 20211021), the Health Bureau, The Government of the Hong Kong Special Administrative Region, received by H.C., the National Natural Science Foundation of China (Nos. 81871510, 82172014), received by Y.Z., and the Natural Science Foundation of Guangdong Province (No. 2023A030313574), received by Y.Z. We thank Guobin Hong (a radiologist at the Department of Radiology in The Fifth Affiliated Hospital of Sun Yat-sen University), Jianxiang Yuan (a radiologist at the Department of Radiology in Foshan Hospital of Traditional Chinese Medicine), Bomiao Lin (a radiologist at the Department of Radiology in Zhujiang Hospital of Southern Medical University) and Caolin Liu (a radiologist at the Department of Radiology in Foshan Nanhai District People’s Hospital) for providing the external datasets.

Author contributions

H.C., Y.Z., Z.Q., Y.L., and H.L. provided study conception and design; Z.Q., Y.L., Z.X., Q.Y., S.L., and M.W. conducted the experiment; H.C., Y.Z., Z.Q., and Z.X. analyzed the results; Y.Z., Z.X., Q.Y., and M.W. collected data; H.C., Z.Q., and Z.X. wrote the manuscript.

Peer review

Peer review information

Nature Communications thanks Jo Schlemper, Tianrui Luo, and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

The raw data collected and processed in this study are supervised under corresponding institutions. The data are available under restricted access, which can be obtained by emailing the corresponding author with all requests for academic use. The requirements will be evaluated concerning institutional policies, and data can only be shared for non-commercial academic usage with a formal material transfer agreement. All requests will be promptly reviewed within a timeframe of 15 working days. The result data generated in this study are provided in the Source Data file. The data that helps to reproduce this study are available in the code repository. Source data are provided in this paper.

Code availability

The pipeline development and experiments are conducted in Python with PyTorch as a primary tool. All code for reproducing this study can be found at https://github.com/zqiuak/CoPAS50.

Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

These authors contributed equally: Zelin Qiu, Zhuoyao Xie.
==== Refs
References

1. Hirschmann MT üllerM W Complex function of the knee joint: the current understanding of the knee Knee Surg. Sports Traumatol. Arthrosc. 2015 23 2780 2788 10.1007/s00167-015-3619-3 25962963
Hirschmann, M. T. & üllerM, W. Complex function of the knee joint: the current understanding of the knee. Knee Surg. Sports Traumatol. Arthrosc. 23, 2780–2788 (2015).25962963 10.1007/s00167-015-3619-3
2. Roos EM Lohmander LS The knee injury and osteoarthritis outcome score (KOOS): from joint injury to osteoarthritis Health Qual. Life Outcomes 2003 1 1 8 10.1186/1477-7525-1-1 12605709
Roos, E. M. & Lohmander, L. S. The knee injury and osteoarthritis outcome score (KOOS): from joint injury to osteoarthritis. Health Qual. Life Outcomes 1, 1–8 (2003).12605709 10.1186/1477-7525-1-1
3. Kim-Wang SY Distribution of bone contusion patterns in acute noncontact anterior cruciate ligament–torn knees Am. J. Sports Med. 2021 49 404 409 10.1177/0363546520981569 33411563
Kim-Wang, S. Y. et al. Distribution of bone contusion patterns in acute noncontact anterior cruciate ligament–torn knees. Am. J. Sports Med. 49, 404–409 (2021).33411563 10.1177/0363546520981569
4. Yusuf E Kortekaas MC Watt I Huizinga TWJ Kloppenburg M Do knee abnormalities visualised on MRI explain knee pain in knee osteoarthritis? a systematic review Ann. Rheum. Dis. 2011 70 60 67 10.1136/ard.2010.131904 20829200
Yusuf, E., Kortekaas, M. C., Watt, I., Huizinga, T. W. J. & Kloppenburg, M. Do knee abnormalities visualised on MRI explain knee pain in knee osteoarthritis? a systematic review. Ann. Rheum. Dis. 70, 60–67 (2011).20829200 10.1136/ard.2010.131904
5. Truong LK Psychological, social and contextual factors across recovery stages following a sport-related knee injury: a scoping review Br. J. Sports Med. 2020 54 1149 1156 10.1136/bjsports-2019-101206 32060141
Truong, L. K. et al. Psychological, social and contextual factors across recovery stages following a sport-related knee injury: a scoping review. Br. J. Sports Med. 54, 1149–1156 (2020).32060141 10.1136/bjsports-2019-101206
6. Connelly JW Patient acceptable symptom state at 1 and 3 years after total knee arthroplasty: Thresholds for the knee injury and osteoarthritis outcome score (KOOS) JBJS 2019 101 995 1003 10.2106/JBJS.18.00233
Connelly, J. W. et al. Patient acceptable symptom state at 1 and 3 years after total knee arthroplasty: Thresholds for the knee injury and osteoarthritis outcome score (KOOS). JBJS 101, 995–1003 (2019).10.2106/JBJS.18.00233
7. Adams BG Houston MN Cameron KL The epidemiology of meniscus injury Sports Med. Arthrosc. Rev. 2021 29 e24 e33 10.1097/JSA.0000000000000329 34398119
Adams, B. G., Houston, M. N. & Cameron, K. L. The epidemiology of meniscus injury. Sports Med. Arthrosc. Rev. 29, e24–e33 (2021).34398119 10.1097/JSA.0000000000000329
8. Musahl V Karlsson J Anterior cruciate ligament tear N. Engl. J. Med. 2019 380 2341 2348 10.1056/NEJMcp1805931 31189037
Musahl, V. & Karlsson, J. Anterior cruciate ligament tear. N. Engl. J. Med. 380, 2341–2348 (2019).31189037 10.1056/NEJMcp1805931
9. Doral MN Bilge O Huri G Turhan E Verdonk R Modern treatment of meniscal tears EFORT open Rev. 2018 3 260 268 10.1302/2058-5241.3.170067 29951265
Doral, M. N., Bilge, O., Huri, G., Turhan, E. & Verdonk, R. Modern treatment of meniscal tears. EFORT open Rev. 3, 260–268 (2018).29951265 10.1302/2058-5241.3.170067
10. Patel KA Hartigan DE Makovicka JL Dulle III DL Chhabra A Diagnostic evaluation of the knee in the office setting using small-bore needle arthroscopy Arthrosc. Tech. 2018 7 e17 e21 10.1016/j.eats.2017.08.044 29379709
Patel, K. A., Hartigan, D. E., Makovicka, J. L., Dulle III, D. L. & Chhabra, A. Diagnostic evaluation of the knee in the office setting using small-bore needle arthroscopy. Arthrosc. Tech. 7, e17–e21 (2018).29379709 10.1016/j.eats.2017.08.044
11. Diederichs G Issever AS Scheffler S Mr imaging of patellar instability: Injury patterns and assessment of risk factors Radiographics 2010 30 961 981 10.1148/rg.304095755 20631363
Diederichs, G., Issever, A. S. & Scheffler, S. Mr imaging of patellar instability: Injury patterns and assessment of risk factors. Radiographics 30, 961–981 (2010).20631363 10.1148/rg.304095755
12. Friberger Pajalic K Turkiewicz A Englund M Update on the risks of complications after knee arthroscopy BMC Musculoskelet. Disord. 2018 19 1 7 10.1186/s12891-018-2102-y 29304778
Friberger Pajalic, K., Turkiewicz, A. & Englund, M. Update on the risks of complications after knee arthroscopy. BMC Musculoskelet. Disord. 19, 1–7 (2018).29304778 10.1186/s12891-018-2102-y
13. Felson DT Arthroscopy as a treatment for knee osteoarthritis Best. Pract. Res. Clin. Rheumatol. 2010 24 47 50 10.1016/j.berh.2009.08.002 20129199
Felson, D. T. Arthroscopy as a treatment for knee osteoarthritis. Best. Pract. Res. Clin. Rheumatol. 24, 47–50 (2010).20129199 10.1016/j.berh.2009.08.002
14. Kim S Bosque J Meehan JP Jamali A Marder R Increase in outpatient knee arthroscopy in the United States: a comparison of national surveys of ambulatory surgery, 1996 and 2006 JBJS 2011 93 994 1000 10.2106/JBJS.I.01618
Kim, S., Bosque, J., Meehan, J. P., Jamali, A. & Marder, R. Increase in outpatient knee arthroscopy in the United States: a comparison of national surveys of ambulatory surgery, 1996 and 2006. JBJS 93, 994–1000 (2011).10.2106/JBJS.I.01618
15. Nacey NC Geeslin MG Miller GW Pierce JL Magnetic resonance imaging of the knee: an overview and update of conventional and state of the art imaging J. Magn. Reson. Imaging 2017 45 1257 1275 10.1002/jmri.25620 28211591
Nacey, N. C., Geeslin, M. G., Miller, G. W. & Pierce, J. L. Magnetic resonance imaging of the knee: an overview and update of conventional and state of the art imaging. J. Magn. Reson. Imaging 45, 1257–1275 (2017).28211591 10.1002/jmri.25620
16. Chien A Weaver JS Kinne E Omar I Magnetic resonance imaging of the knee Pol. J. Radiol. 2020 85 509 531 10.5114/pjr.2020.99415
Chien, A., Weaver, J. S., Kinne, E. & Omar, I. Magnetic resonance imaging of the knee. Pol. J. Radiol. 85, 509–531 (2020).10.5114/pjr.2020.99415
17. Tyler P Datir A Saifuddin A Magnetic resonance imaging of anatomical variations in the knee Skelet. Radiol. 2010 39 1175 1186 10.1007/s00256-010-0904-6
Tyler, P., Datir, A. & Saifuddin, A. Magnetic resonance imaging of anatomical variations in the knee. Skelet. Radiol. 39, 1175–1186 (2010).10.1007/s00256-010-0904-6
18. Astuto B Automatic deep learning–assisted detection and grading of abnormalities in knee MRI studies Radiology: Artif. Intell. 2021 3 e200165
Astuto, B. et al. Automatic deep learning–assisted detection and grading of abnormalities in knee MRI studies. Radiology: Artif. Intell. 3, e200165 (2021).
19. Atito, S., Anwar, S. M., Awais, M. & Kittler, J. SB-SSL: Slice-based self-supervised transformers for knee abnormality classification from MRI. In Workshop on Medical Image Learning with Limited and Noisy Data, pages 86–95. Springer, (2022).
20. Huang, C. et al. DADP: Dynamic abnormality detection and progression for longitudinal knee magnetic resonance images from the osteoarthritis initiative. Medical Image Analysis, 102343, (2022).
21. Liu F Deep learning approach for evaluating knee MR images: Achieving high diagnostic performance for cartilage lesion detection Radiology 2018 289 160 169 10.1148/radiol.2018172986 30063195
Liu, F. et al. Deep learning approach for evaluating knee MR images: Achieving high diagnostic performance for cartilage lesion detection. Radiology 289, 160–169 (2018).30063195 10.1148/radiol.2018172986
22. Azcona, D., McGuinness, K. & Smeaton, A. F. A comparative study of existing and new deep learning methods for detecting knee injuries using the MRNET dataset. In 2020 International Conference on Intelligent Data Science Technologies and Applications (IDSTA), 149–155. IEEE, (2020).
23. Zhao, J. et al. Adaptive fusion of deep learning with statistical anatomical knowledge for robust patella segmentation from CT images. IEEE Journal of Biomedical and Health Informatics, (2024).
24. Bien N Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet PLoS Med. 2018 15 e1002699 10.1371/journal.pmed.1002699 30481176
Bien, N. et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLoS Med. 15, e1002699 (2018).30481176 10.1371/journal.pmed.1002699
25. Štajduhar I Mamula M Miletić D Uenal G Semi-automated detection of anterior cruciate ligament injury from MRI Comput. Methods Prog. Biomed. 2017 140 151 164 10.1016/j.cmpb.2016.12.006
Štajduhar, I., Mamula, M., Miletić, D. & Uenal, G. Semi-automated detection of anterior cruciate ligament injury from MRI. Comput. Methods Prog. Biomed. 140, 151–164 (2017).10.1016/j.cmpb.2016.12.006
26. Belton, N. et al. Optimising knee injury detection with spatial attention and validating localisation ability. In Annual Conference on Medical Image Understanding and Analysis, pages 71–86. Springer, (2021).
27. Tsai, C.-H., Kiryati, N., Konen, E., Eshed, I. & Mayer, A. Knee injury detection using MRI with efficiently-layered network (elnet). In Medical Imaging with Deep Learning, 784–794. PMLR, (2020).
28. De Smet AA How I diagnose meniscal tears on knee MRI Am. J. Roentgenol. 2012 199 481 499 10.2214/AJR.12.8663 22915388
De Smet, A. A. How I diagnose meniscal tears on knee MRI. Am. J. Roentgenol. 199, 481–499 (2012).22915388 10.2214/AJR.12.8663
29. Barberie JE Carson BW Finnegan M Wong AD Oblique sagittal view of the anterior cruciate ligament: comparison of coronal vs. axial planes as localizing sequences J. Magn. Reson. Imaging: Off. J. Int. Soc. Magn. Reson. Med. 2001 14 203 206 10.1002/jmri.1174
Barberie, J. E., Carson, B. W., Finnegan, M. & Wong, A. D. Oblique sagittal view of the anterior cruciate ligament: comparison of coronal vs. axial planes as localizing sequences. J. Magn. Reson. Imaging: Off. J. Int. Soc. Magn. Reson. Med. 14, 203–206 (2001).10.1002/jmri.1174
30. Khanna AJ Magnetic resonance imaging of the knee: current techniques and spectrum of disease JBJS 2001 83 S128 141 10.2106/00004623-200100022-00011
Khanna, A. J. et al. Magnetic resonance imaging of the knee: current techniques and spectrum of disease. JBJS 83, S128–141 (2001).10.2106/00004623-200100022-00011
31. Muhammad, M. B. & Yeasin, M. Eigen-CAM: class activation map using principal components. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, (2020).
32. Chattopadhay, A., Sarkar, A., Howlader, P. & Balasubramanian, V. N. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847. IEEE, (2018).
33. Collins GS Reitsma JB Altman DG Moons KGM Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): the tripod statement Ann. Intern. Med. 2015 162 55 63 10.7326/M14-0697 25560714
Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): the tripod statement. Ann. Intern. Med. 162, 55–63 (2015).25560714 10.7326/M14-0697
34. Shen D Wu G Suk H-I Deep learning in medical image analysis Annu. Rev. Biomed. Eng. 2017 19 221 248 10.1146/annurev-bioeng-071516-044442 28301734
Shen, D., Wu, G. & Suk, H.-I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017).28301734 10.1146/annurev-bioeng-071516-044442
35. Qiu, Z. et al. Predicting symptoms from multiphasic MRI via multi-instance attention learning for hepatocellular carcinoma grading. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 439–448. Springer, (2021).
36. Gupta, S., Pawar, P. M. & Tamizharasan, P.S. Intelligent detection of knee injury in MRI exam. International Journal of Information Technology, 1–7, (2022).
37. Namiri NK Deep learning for hierarchical severity staging of anterior cruciate ligament injuries from MRI Radiology: Artif. Intell. 2020 2 e190207
Namiri, N. K. et al. Deep learning for hierarchical severity staging of anterior cruciate ligament injuries from MRI. Radiology: Artif. Intell. 2, e190207 (2020).
38. Chang PD Wong TT Rasiej MJ Deep learning for detection of complete anterior cruciate ligament tear J. Digital Imaging 2019 32 980 986 10.1007/s10278-019-00193-4
Chang, P. D., Wong, T. T. & Rasiej, M. J. Deep learning for detection of complete anterior cruciate ligament tear. J. Digital Imaging 32, 980–986 (2019).10.1007/s10278-019-00193-4
39. Graham S One model is all you need: multi-task learning enables simultaneous histology image segmentation and classification Med. Image Anal. 2023 83 102685 10.1016/j.media.2022.102685 36410209
Graham, S. et al. One model is all you need: multi-task learning enables simultaneous histology image segmentation and classification. Med. Image Anal. 83, 102685 (2023).36410209 10.1016/j.media.2022.102685
40. Liu, S., Liang, Y. & Gitter, A. Loss-balanced task weighting to reduce negative transfer in multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 33, 9977–9978, (2019).
41. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, 234–241. Springer, (2015).
42. Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).
43. Tran, D. et al. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6450–6459, (2018).
44. Hara, K., Kataoka, H. & Satoh, Y. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and Imagenet? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6546–6555, (2018).
45. Wang, L., Ding, Z., Tao, Z., Liu, Y. & Fu, Y. Generative multi-view human action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6212–6221, (2019).
46. Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141, (2018).
47. Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, (2017).
48. DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, pages 837–845, (1988).
49. Fleiss JL Cohen J The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability Educ. Psychol. Meas. 1973 33 613 619 10.1177/001316447303300309
Fleiss, J. L. & Cohen, J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ. Psychol. Meas. 33, 613–619 (1973).10.1177/001316447303300309
50. Qiu, Z. et al. Learning co-plane attention across MRI sequences assists radiologists in diagnosing twelve types of knee abnormalities. CoPAS, 10.5281/ZENODO.12680648 (2024).
51. Quinn SF Brown TF Meniscal tears diagnosed with mr imaging versus arthroscopy: how reliable a standard is arthroscopy? Radiology 1991 181 843 847 10.1148/radiology.181.3.1947108 1947108
Quinn, S. F. & Brown, T. F. Meniscal tears diagnosed with mr imaging versus arthroscopy: how reliable a standard is arthroscopy? Radiology 181, 843–847 (1991).1947108 10.1148/radiology.181.3.1947108
52. Altman, R. D. and Kates, J. Arthroscopy of the knee. In Seminars in arthritis and rheumatism, 13, pages 188–199. Elsevier, (1983).
53. Noyes FR Stabler CL A system for grading articular cartilage lesions at arthroscopy Am. J. Sports Med. 1989 17 505 513 10.1177/036354658901700410 2675649
Noyes, F. R. & Stabler, C. L. A system for grading articular cartilage lesions at arthroscopy. Am. J. Sports Med. 17, 505–513 (1989).2675649 10.1177/036354658901700410
54. Margheritini F Mariani PP Diagnostic evaluation of posterior cruciate ligament injuries Knee Surg. Sports Traumatol. Arthrosc. 2003 11 282 288 10.1007/s00167-003-0409-0 12955348
Margheritini, F. & Mariani, P. P. Diagnostic evaluation of posterior cruciate ligament injuries. Knee Surg. Sports Traumatol. Arthrosc. 11, 282–288 (2003).12955348 10.1007/s00167-003-0409-0
55. Craft JA Kurzweil PR Physical examination and imaging of medial collateral ligament and posteromedial corner of the knee Sports Med. Arthrosc. Rev. 2015 23 e1 e6 10.1097/JSA.0000000000000066 25932881
Craft, J. A. & Kurzweil, P. R. Physical examination and imaging of medial collateral ligament and posteromedial corner of the knee. Sports Med. Arthrosc. Rev. 23, e1–e6 (2015).25932881 10.1097/JSA.0000000000000066
56. Grawe B Schroeder AJ Kakazu R Messer MS Lateral collateral ligament injury about the knee: anatomy, evaluation, and management JAAOS-J. Am. Acad. Orthop. Surg. 2018 26 e120 e127 10.5435/JAAOS-D-16-00028
Grawe, B., Schroeder, A. J., Kakazu, R. & Messer, M. S. Lateral collateral ligament injury about the knee: anatomy, evaluation, and management. JAAOS-J. Am. Acad. Orthop. Surg. 26, e120–e127 (2018).10.5435/JAAOS-D-16-00028
57. Jackson DW Jennings LD Maywood RM Berger PE Magnetic resonance imaging of the knee Am. J. Sports Med. 1988 16 29 38 10.1177/036354658801600105 3344877
Jackson, D. W., Jennings, L. D., Maywood, R. M. & Berger, P. E. Magnetic resonance imaging of the knee. Am. J. Sports Med. 16, 29–38 (1988).3344877 10.1177/036354658801600105
58. Vassiou K Synovial plicae of the knee joint: the role of advanced MRI Postgrad. Med. J. 2015 91 35 40 10.1136/postgradmedj-2013-132176 25476020
Vassiou, K. et al. Synovial plicae of the knee joint: the role of advanced MRI. Postgrad. Med. J. 91, 35–40 (2015).25476020 10.1136/postgradmedj-2013-132176
59. Marra MD Mri features of cystic lesions around the knee Knee 2008 15 423 438 10.1016/j.knee.2008.04.009 18559292
Marra, M. D. et al. Mri features of cystic lesions around the knee. Knee 15, 423–438 (2008).18559292 10.1016/j.knee.2008.04.009
60. Saddik D McNally EG Richardson M Mri of Hoffa’s fat pad Skelet. Radiol. 2004 33 433 444 10.1007/s00256-003-0724-z
Saddik, D., McNally, E. G. & Richardson, M. Mri of Hoffa’s fat pad. Skelet. Radiol. 33, 433–444 (2004).10.1007/s00256-003-0724-z
61. Thawait SK High resolution magnetic resonance imaging of the patellar retinaculum: normal anatomy, common injury patterns, and pathologies Skelet. Radiol. 2012 41 137 148 10.1007/s00256-011-1291-3
Thawait, S. K. et al. High resolution magnetic resonance imaging of the patellar retinaculum: normal anatomy, common injury patterns, and pathologies. Skelet. Radiol. 41, 137–148 (2012).10.1007/s00256-011-1291-3
