
==== Front
J Immunother Cancer
J Immunother Cancer
jitc
jitc
Journal for Immunotherapy of Cancer
2051-1426
BMJ Publishing Group BMA House, Tavistock Square, London, WC1H 9JR

39231545
10.1136/jitc-2024-009348
jitc-2024-009348
Original Research
Immunotherapy Biomarkers
2437
1506
Non-invasive multimodal CT deep learning biomarker to predict pathological complete response of non-small cell lung cancer following neoadjuvant immunochemotherapy: a multicenter study
Ye Guanchao 120guanchaoye@qq.com

Wu Guangyao 30guangyao.w@foxmail.com

Qi Yu 10qiyu@zzu.edu.cn

Li Kuo 2lktj126@126.com

Wang Mingliang 4wml6667@163.com

Zhang Chunyang 1zcy198200@126.com

Li Feng 1lf_0604@163.com

Wee Leonard 56leonard.wee@maastro.nl

Dekker Andre 6andre.dekker@maastro.nl

Han Chu 789*1hanchu@gdph.org.cn

Liu Zaiyi 78*1liuzaiyi@gdph.org.cn

Liao Yongde 2*1liaoyongde@hust.edu.cn

http://orcid.org/0000-0003-1305-5935
Shi Zhenwei 7891shizhenwei@gdph.org.cn

1 Department of Thoracic Surgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
2 Department of Thoracic Surgery, Huazhong University of Science and Technology Tongji Medical College Union Hospital, Wuhan, Hubei, China
3 Department of Radiology, Huazhong University of Science and Technology Tongji Medical College Union Hospital, Wuhan, Hubei, China
4 Department of Thoracic Surgery, Henan Provincial People's Hospital, Zhengzhou, Henan, China
5 Clinical Data Science, Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
6 Department of Radiation Oncology (Maastro), GROW School of Oncology and Reproduction, Maastricht University Medical Centre+, Maastricht, The Netherlands
7 Department of Radiology, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China
8 Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou, China
9 Medical Research Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Additional supplemental material is published online only. To view, please visit the journal online (https://doi.org/10.1136/jitc-2024-009348).

None declared.

ProfessorZhenweiShi; shizhenwei@gdph.org.cn
ProfessorYongdeLiao; liaoyongde@hust.edu.cn
ProfessorZaiyiLiu; liuzaiyi@gdph.org.cn
ProfessorChuHan; hanchu@gdph.org.cn
GY, GW and YQ contributed equally.

CH, ZL, YL and ZS are joint senior authors.

2024
3 9 2024
12 9 e00934804 8 2024
Copyright © Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See http://creativecommons.org/licenses/by-nc/4.0/.

Abstract

Objectives

Although neoadjuvant immunochemotherapy has been widely applied in non-small cell lung cancer (NSCLC), predicting treatment response remains a challenge. We used pretreatment multimodal CT to explore deep learning-based immunochemotherapy response image biomarkers.

Methods

This study retrospectively obtained non-contrast enhanced and contrast enhancedbubu CT scans of patients with NSCLC who underwent surgery after receiving neoadjuvant immunochemotherapy at multiple centers between August 2019 and February 2023. Deep learning features were extracted from both non-contrast enhanced and contrast enhanced CT scans to construct the predictive models (LUNAI-uCT model and LUNAI-eCT model), respectively. After the feature fusion of these two types of features, a fused model (LUNAI-fCT model) was constructed. The performance of the model was evaluated using the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. SHapley Additive exPlanations analysis was used to quantify the impact of CT imaging features on model prediction. To gain insights into how our model makes predictions, we employed Gradient-weighted Class Activation Mapping to generate saliency heatmaps.

Results

The training and validation datasets included 113 patients from Center A at the 8:2 ratio, and the test dataset included 112 patients (Center B n=73, Center C n=20, Center D n=19). In the test dataset, the LUNAI-uCT, LUNAI-eCT, and LUNAI-fCT models achieved AUCs of 0.762 (95% CI 0.654 to 0.791), 0.797 (95% CI 0.724 to 0.844), and 0.866 (95% CI 0.821 to 0.883), respectively.

Conclusions

By extracting deep learning features from contrast enhanced and non-contrast enhanced CT, we constructed the LUNAI-fCT model as an imaging biomarker, which can non-invasively predict pathological complete response in neoadjuvant immunochemotherapy for NSCLC.

Immunotherapy
Pathologic complete response - pCR
Biomarker
Lung Cancer
Neoadjuvant
National Key Research and Development Program 2023YFC2508603 National Science Foundation for Young Scientists of China 62102103 82102034 82202148 Wuhan Science and Technology Bureau Knowledge Innovation Special Project 2023020201010164 International Cooperation Project of Hubei Provincial Department of Science and Technology 2023EHA060
==== Body
pmcWHAT IS ALREADY KNOWN ON THIS TOPIC

Current biomarkers for immunotherapy, such as programmed cell death ligand 1, require invasive procedures and have limited accuracy. While previous studies have demonstrated that deep learning approaches based on pretreatment unimodal CT scans can predict the treatment response in non-small cell lung cancer (NSCLC) patients undergoing neoadjuvant immunochemotherapy, no research has explored the potential of deep learning approaches using both pre-treatment non-contrast and contrast enhanced CT scans to non-invasively predict the pathological complete response (pCR) in NSCLC patients after neoadjuvant immunochemotherapy.

WHAT THIS STUDY ADDS

This multicenter study used non-contrast enhanced and contrast enhanced CT scans to extract deep learning features and develop predictive models (LUNAI-uCT, LUNAI-eCT, and LUNAI-fCT) for pCR in NSCLC patients undergoing neoadjuvant immunochemotherapy. For the first time, this study compared the performance of a multimodal fusion prediction model with single-mode models and found that the multimodal fusion model significantly outperformed the single-mode models.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

As an imaging biomarker, the LUNAI-fCT model can non-invasively predict pCR in NSCLC patients treated with neoadjuvant immunochemotherapy. This capability could enhance clinical decision-making for resectable NSCLC patients, help avoid overtreatment, and promote personalized, precise cancer care.

Introduction

Lung cancer remains the most lethal malignancy worldwide, characterized by a substantial incidence and mortality rate.1 Prognosis in advanced-stage non-small cell lung cancer (NSCLC) is relatively poor, making neoadjuvant treatment essential for tumor downstaging prior to surgical resection. Recently, the integration of immunochemotherapy into neoadjuvant regimens has marked a significant advancement, yielding improved progression-free survival and overall survival.2

The advent of neoadjuvant immunochemotherapy has been paralleled by an increase in patients achieving a pathological complete response (pCR), with reported rates ranging from 24% to 63%.35 The extent of residual tumor cells postsurgery is a critical prognostic factor, with lower residual tumor burden correlating with a more favorable prognosis.6 Confirming pCR in patients with NSCLC is a valuable predictor of their survival time.7 Furthermore, patients who achieve pCR may benefit from a “wait-and-see” strategy, similar to observation strategies employed in rectal cancer8 and esophageal cancer.9 10 Implementing this strategy can help prevent overtreatment and potentially lead to long-term survival.11 Studies have shown that excessive lymph node dissection in lung cancer can reduce the effectiveness of immunotherapy.12 For patients with pCR, avoiding thorough lymph node dissection may help achieve better outcomes in immune maintenance treatment. However, the definitive assessment of pCR still primarily relies on histopathological examination of the resected tumor, underscoring the need for a non-invasive, accurate method to identify pCR post-immunochemotherapy.

CT imaging is a standard, cost-effective tool for monitoring tumor response to treatment in lung cancer.13 However, when evaluating treatment response to neoadjuvant immunochemotherapy, there exists a substantial disparity between radiological and pathological evaluations. In the NADIM trial, 33% of patients with stable disease and 73% with partial response on imaging were found to have pCR.6 This difference is often due to pseudo-lesions resulting from lymphocyte infiltration, where radiological findings might not reflect actual tumor regression. Therefore, it is necessary to refine methodologies to bridge the gap between radiological and pathological evaluations in the context of neoadjuvant immunochemotherapy.

With the development of artificial intelligence (AI), extracting radiomics or deep learning features from CT can provide additional information, enhancing diagnosis, treatment, and prognosis in lung cancer.14 15 Several studies have investigated the predictive value of quantitative imaging features for neoadjuvant immunochemotherapy response in NSCLC.4 16 17 However, conventional deep learning feature extraction methods are typically based on models trained on natural images or limited datasets, limiting their applicability in medical research. To address this limitation, we previously developed a pretrained foundation model for CT images in lung cancer (namely FM-LCT) using datasets with large sample size and ample diversity. Notably, there has been no prior research exploring the use of non-contrast enhanced and contrast enhanced CT images to extract deep learning features and build prediction models for neoadjuvant immunochemotherapy response in NSCLC. In this study, we used the FM-LCT model to extract deep learning features from non-contrast enhanced and contrast enhanced CT images obtained before treatment. We then fused these deep learning features and constructed a machine learning model to predict pCR. This approach is expected to achieve more accurate prediction performance.

Here, we introduce an AI-based prediction model that uses deep learning features from pretreatment multimodal CT scans to predict pCR in patients with NSCLC undergoing neoadjuvant immunochemotherapy. Our method enables the multidimensional characterization of tumor features during neoadjuvant immunochemotherapy in NSCLC and has successfully developed a non-invasive biomarker to distinguish pCR. This approach has the potential to aid clinical decision-making in resectable NSCLC patients, prevent overtreatment, and facilitate personalized and precise cancer treatment.

Materials and methods

Study data

This retrospective study is registered at http://clinicaltrials.gov (identifier: NCT06285058). We screened 295 patients with NSCLC who underwent surgery after neoadjuvant immunochemotherapy at Center A (Tongji Medical College Affiliated Union Hospital), Center B (Zhengzhou University First Affiliated Hospital), Center C (Yichang Central Hospital), and Center D (Anyang Cancer Hospital) between August 2019 and February 2023.

Inclusion criteria were: (1) NSCLC diagnosis confirmed by biopsy pathology, clinically staged IB to III; (2) completion of at least two cycles of neoadjuvant immunochemotherapy; (3) postoperative pathological evaluation of tumor and lymph nodes as per International Association for the Study of Lung Cancer (IASLC) guidelines. Exclusion criteria included: (1) absence of pretreatment CT image; (2) CT imaging more than 1 month before treatment initiation; (3) incomplete clinicopathological data.

The study subjects were presented in the flowchart depicted in figure 1. The clinicopathological data included age, gender, smoking history, tumor and family history, pretreatment neutrophil to lymphocyte ratio, pretreatment serum lactic dehydrogenase, pretreatment albumin, pretreatment clinical stage, tumor location, pathological type, and postoperative pathological response. The tumor, node, metastasis (TNM) staging system used was the 8th edition of the IASLC TNM staging system.18 The reporting of this study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guidelines.19

Figure 1 Flowchart shows patient exclusion for each dataset. IASLC, International Association for the Study of Lung Cancer; NSCLC, non-small cell lung cancer.

Pretreatment evaluation and neoadjuvant administration

Prior to neoadjuvant treatment, a comprehensive evaluation was conducted to diagnose and stage the tumor. This evaluation involved various preoperative examinations, including enhanced MRI of the head, contrast enhanced CT of the chest, abdominal ultrasound, whole-body bone scintigraphy, or positron emission tomography-CT. Additionally, pathological diagnosis was obtained through procedures such as tissue biopsy under bronchoscopy, ultrasound bronchoscopy-guided transbronchial fine needle aspiration, or CT-guided percutaneous puncture. Typically, patients underwent 2–4 cycles of neoadjuvant immunochemotherapy, with a 3-week interval between each cycle. The treatment regimen comprised first-line immune checkpoint inhibitors, such as pembrolizumab (200 mg), nivolumab (360 mg), durvalumab (1500 mg), nivolumab (200 mg), tislelizumab (200 mg), or camrelizumab (200 mg), combined with platinum-based chemotherapy, as recommended by guidelines. Approximately 4 weeks after the completion of neoadjuvant treatment, surgical resection and systematic lymph node dissection were performed. The specific treatment plan, treatment cycles, and operation schedule for each patient were determined through multidisciplinary consultation among the participating treatment institutions.

Histopathological assessment and definition of pCR

Thoracic surgeons (YQ and CZ) with pathological diagnosis expertise collaborated with pathologists to evaluate treatment responses at each site. This evaluation followed the IASLC guidelines,20 defining pCR as the absence of viable tumor cells in both the tumor bed and lymph nodes (ypT0 and ypN0). Discrepancies were resolved through review by two pathologists, reaching a consensus via discussion.

CT acquisition

The study used CT scans with slice thicknesses ranging from 0.625 mm to 1.25 mm alongside the administration of contrast medium. Each patient received an intravenous injection of non-ionic iodinated contrast medium (350 mg/mL) at a dosage of 60–80 mL, with an injection rate of 2–3 mL/min. Bone reconstruction algorithms were used for reconstruction purposes. Detailed acquisition and reconstruction parameters were available in online supplemental file S1. The chest CT images underwent retrospective analysis using a window width of 1600 HU and a window level of −600 HU. A radiologist (GW) and a thoracic surgeon (GY) completed region of interest (ROI) segmentation using ITK-SNAP (V.3.8.0, available at http://www.itksnap.org/). Differences in opinion among the radiologists were resolved through discussion to achieve a consensus. Subsequently, the senior radiologist assessed the quality of the ROI and made necessary adjustments after the primary radiologist completed tumor lesion segmentation. To assess the robustness, 50 cases were randomly selected to estimate the intraclass correlation coefficients, with a value of ≥0.75 indicating robustness. Figure 2 illustrates the overall research design.

Figure 2 An illustration of the overall research design. (a) Pretraining of the foundation model using masked autoencoder method. (b) Deep learning features extraction, and prediction model training and test. (c) Quantitative analysis and evaluation including patient characteristics analysis, prediction model evaluation, and quantitative imaging feature analysis. LUNAI-eCT, contrast enhanced CT deep learning features model; LUNAI-fCT, non-contrast and contrast enhanced CT fused deep learning features model; LUNAI-uCT, non-contrast enhanced CT deep learning features model.

Image preprocessing

Imaging analysis, especially for CT scans, was sensitive to variations in slice thickness and scanner types. To address this, image preprocessing was performed on the original CT images. These steps included voxel resampling to achieve isotropic dimensions of 1 mm using B-spline interpolation and intensity normalization using the z-score method. This meticulous approach ensures a consistent and reliable foundation for subsequent model development and data analysis.

Feature extraction and model development

Given the intricate nature and wide-ranging variations found in lung cancer CT images, traditional feature extraction methods often fail to capture essential effectively. In contrast, deep learning-based feature extraction methods offer significant advantages, such as the ability to autonomously learn abstract and intricate features from images, eliminating the need for manual design. However, conventional deep learning feature extraction methods are typically based on models trained on natural images or limited datasets, limiting their applicability in medical research.

To address this limitation, we employed a pre-existing foundational model called FM-LCT as the feature extractor. The FM-LCT model was trained on a diverse dataset encompassing various lung cancer types and stages, using masked autoencoder (MAE) contrastive learning algorithms.21 For further details, refer to the project’s repository on GitHub (https://github.com/zhenweishi/FM-LCT).

We used the FM-LCT model to extract two feature sets, namely FS-uCT and FS-eCT, from the bounding box of tumors in non-contrast enhanced and contrast enhanced CT images, respectively. Each feature set consisted of a feature vector with a length of 768. A fused feature set (FS-fCT) was generated by performing average pooling on the FS-uCT and FS-eCT feature sets. To prevent model overfitting, we performed feature dimension reduction using principal component analysis, retaining 16 key features. Correlations were assessed using Pearson and Spearman methods. Ultimately, we developed three types of models, named LUng cancer NeoAdjuvant Immunochemotherapy (LUNAI), by the random forest modeling method: (1) using only non-contrast enhanced CT features (LUNAI-uCT model), (2) using only contrast enhanced CT features (LUNAI-eCT model), and (3) using the fused CT features (LUNAI-fCT model). The output probability of LUNAI-fCT model was used to generate an immunochemotherapy treatment response (Immu_TR) score, which represented the likelihood of patients achieving pCR following neoadjuvant immunochemotherapy.

Model evaluation and statistical analysis

Models were evaluated using the area under the receiver operating characteristic curve (AUC) with a 95% CI. Additionally, several metrics were calculated, including accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive (NPV). To quantify the impact of CT imaging features on model prediction, SHapley Additive exPlanations (SHAP) analysis was conducted.22 To gain insights into how our model makes predictions, we employed Gradient-weighted Class Activation Mapping (Grad-CAM) to generate saliency heatmaps.23

Statistical analyses were carried out using R software (V.3.6.0, The R Foundation) and Python SciPy (V.1.8.0, Python Software Foundation). A two-tailed p value of less than 0.05 was considered statistically significant. Fisher’s exact test or Pearson’s χ2 test and the Kruskal-Wallis test were used to compare the characteristics of patients in the training and external test datasets.

Results

Patient characteristics

A total of 225 patients were initially acquired from four centers. Patients were excluded for 37 patients due to missing or inadequate quality of CT, 14 patients with a time interval greater than 1 month between CT imaging and treatment initiation, and 19 patients with incomplete clinicopathological data. This resulted in a final training and internal validation datasets comprising 113 patients from Center A, and the test dataset included 112 patients (Center B n=73, Center C n=20, Center D n=19). Table 1 provides a summary of the clinicopathological information of the patients, while table 2 details the patient characteristics across different cohorts. In the training and internal validation datasets, 111 patients had non-contrast enhanced CT images, 108 patients had contrast enhanced CT images, and 106 patients had both modalities. Correspondingly, the test dataset included 71, 74, and 35 patients, respectively, for the same categories.

Table 1 Comparison of clinical and pathological characteristics between pCR group and non-pCR group

Characteristics	Total(n=225)	Non-pCR(n=145)	pCR(n=80)	P value	
Age (years), mean±SD	60.682±7.750	60.951±7.423	60.193±8.336	0.480	
Gender, n (%)				0.003*	
 Female	26 (11.6)	24 (16.6)	2 (2.5)		
 Male	199 (88.4)	121 (83.4)	78 (97.5)		
Smoking, n (%)				0.134	
 Yes	140 (62.2)	85 (58.6)	55 (68.8)		
 No	85 (37.8)	60 (41.4)	25 (31.2)		
Location of lesion, n (%)				0.245	
 Left	113 (50.2)	77 (53.1)	36 (45.0)		
 Right	112 (49.8)	68 (46.9)	44 (55.0)		
Pathological type, n (%)				0.160	
 Squamous carcinoma	169 (75.1)	103 (71.0)	66 (82.5)		
 Adenocarcinoma	49 (21.8)	37 (25.5)	12 (15.0)		
 Ther	7 (3.1)	5 (3.5)	2 (2.5)		
T stage, n (%)				0.043*	
 T1	29 (12.9)	13 (9.0)	16 (20.0)		
 T2	82 (36.4)	50 (34.5)	32 (40.0)		
 T3	73 (32.5)	52 (35.9)	21 (26.2)		
 T4	41 (18.2)	30 (20.6)	11 (13.8)		
N stage, n (%)				0.768	
 N0	19 (8.4)	13 (9.0)	6 (7.5)		
 N1	44 (19.6)	31 (21.4)	13 (16.2)		
 N2	146 (64.9)	91 (62.8)	55 (68.8)		
 N3	16 (7.1)	10 (6.8)	6 (7.5)		
Clinical stage, n (%)				0.672	
 II	34 (15.1)	23 (15.9)	11 (13.8)		
 III	191 (84.9)	122 (84.1)	69 (86.2)		
 NLR, median (IQR)	3.141(2.158–4.231)	3.378(2.354–4.566)	2.833(1.975–3.712)	0.005*	
 LDH (IU/L), median (IQR)	207.000(178.000–241.000)	208.500(182.00–228.000)	203.000(177.000–246.00)	0.544	
 Albumin (g/dL), median (IQR)	38.200(35.700–41.000)	38.200(35.700–41.050)	38.550(35.325–41.075)	0.907	
* Indicates sStatistical significance; p value <0.05.

IQRinterquartile rangeLDH, lactic dehydrogenaseNLR, neutrophil to lymphocyte ratio; pCR, pathological complete response

Table 2 Patient characteristics across different cohorts

Characteristics	Total(n=225)	Training and Validation(n=113)	Test(n=112)	P value	
Age (years), mean±SD	60.683±7.750	60.312±6.852	61.054±8.577	0.473	
Gender, n (%)				0.035*	
 Female	26 (11.6)	8 (7.1)	18 (16.1)		
 Male	199 (88.4)	105 (92.9)	94 (83.9)		
Smoking, n (%)				0.236	
 Yes	140 (62.2)	66 (58.4)	74 (66.1)		
 No	85 (37.8)	47 (41.6)	38 (33.9)		
Location of lesion, n (%)				0.463	
 Left	113 (50.2)	54 (47.8)	59 (52.7)		
 Right	112 (49.8)	59 (52.2)	53 (47.3)		
Pathological type, n (%)				0.102	
 Squamous carcinoma	169 (75.1)	80 (70.8)	89 (79.5)		
 Adenocarcinoma	49 (21.8)	27 (23.9)	22 (19.6)		
 Ther	7 (3.1)	6 (5.3)	1 (0.9)		
Clinical T stage, n (%)				0.309	
 T1	29 (12.9)	11 (9.7)	18 (16.1)		
 T2	82 (36.4)	47 (41.6)	35 (31.2)		
 T3	73 (32.4)	35 (31.0)	38 (33.9)		
 T4	41 (18.3)	20 (17.7)	21 (18.8)		
Clinical N stage, n (%)				0.414	
 N0	19 (8.4)	8 (7.1)	11 (9.8)		
 N1	44 (19.6)	18 (15.9)	26 (23.2)		
 N2	146 (64.9)	78 (69.0)	68 (60.7)		
 N3	16 (7.1)	9 (8.0)	7 (6.3)		
Clinical stage, n (%)				0.059	
 II	34 (15.1)	12 (10.6)	22 (19.6)		
 III	191 (84.9)	101 (89.4)	90 (80.4)		
pCR, n (%)				0.432	
 Present	80 (35.6)	43 (38.1)	37 (33.0)		
 Absent	145 (64.4)	70 (61.9)	75 (67.0)		
* Indicates sStatistical significance; p value <0.05.

pCR, pathological complete response

Performance evaluation of prediction models

Figure 3a compares the discrimination performance for different prediction models concerning pCR. The LUNAI-uCT, LUNAI-eCT, and LUNAI-fCT models achieved AUCs of 0.762 (95% CI 0.654 to 0.791), 0.797 (95% CI 0.724 to 0.844), and AUC of 0.866 (95% CI 0.821 to 0.883) in the test dataset, respectively. The confusion matrices for these models are shown in figure 3b, with cut-off values of 0.448 for the LUNAI-uCT, 0.458 for LUNAI-eCT, and 0. 434 for LUNAI-fCT. Subgroup analysis based on age (≥60 vs <60), smoking status (smoking vs non-smoking), and pretreatment overall stage (II vs III–IV) were depicted in figure 3c.

Figure 3 Performance of the three developed models for predicting pCR. (a) Receiver operating characteristic curves (ROC); (b) the confusion matrix for the three models; (c) the subgroup analysis includes age, smoking status, and pretreatment overall stage in the test dataset. LUNAI-eCT, contrast enhanced CT deep learning features model; LUNAI-fCT, non-contrast and contrast enhanced CT fused deep learning features model; LUNAI-uCT, non-contrast enhanced CT deep learning features model; pCR, pathological complete response.

The evaluation metrics, including accuracy, sensitivity, specificity, PPV, and NPV, were detailed in table 3. In the test dataset, the LUNAI-uCT model achieved an accuracy of 0.676 (95% CI 0.610 to 0.753). It exhibited high sensitivity (0.958) but lower specificity (0.532), indicating a higher rate of false positives. The PPV was 0.511, while the NPV was 0.962. The LUNAI-eCT model showed improved performance with an accuracy of 0.716 (95% CI 0.668 to 0.758). It had a sensitivity of 0.833 and a specificity of 0.660, reflecting a better balance in identifying positive and negative cases. The PPV for LUNAI-eCT was 0.541, and the NPV was 0.892. The LUNAI-fCT model demonstrated the highest performance, with an accuracy of 0.800 (95% CI 0.772 to 0.841), a sensitivity of 0.917, and a specificity of 0.739. The PPV for LUNAI-fCT was 0.647, and the NPV was 0.944. These results indicate that the LUNAI-fCT model exhibited the highest discrimination performance for predicting pCR, demonstrating its generalization in external unseen test datasets.

Table 3 Discrimination performance comparison of the prediction models for pCR

Model	AUC	ACC	Sensitivity	Specificity	PPV	NPV	
LUNAI-uCT							
 Training	0.827(95% CI 0.743 to 0.899)	0.727(95% CI 0.667 to 0.784)	0.879	0.636	0.592	0.897	
 Validation	0.825(95% CI 0.603 to 0.992)	0.826(95% CI 0.785 to 0.886)	1.000	0.714	0.692	1.000	
 Test	0.762(95% CI 0.654 to 0.791)	0.676(95% CI 0.610 to 0.753)	0.958	0.532	0.511	0.962	
LUNAI-eCT							
 Training	0.849(95% CI 0.771 to 0.918)	0.756(95% CI 0.701 to 0.790)	0.970	0.623	0.615	0.971	
 Validation	0.839(95% CI 0.638 to 0.982)	0.818(95% CI 0.729 to 0.863)	0.875	0.786	0.700	0.917	
 Test	0.797(95% CI 0.724 to 0.844)	0.716(95% CI 0.668 to 0.758)	0.833	0.660	0.541	0.892	
LUNAI-fCT							
 Training	0.927(95% CI 0.869 to 0.974)	0.845(95% CI 0.802 to 0.901)	0.906	0.808	0.774	0.933	
 Validation	0.884(95% CI 0.708 to 1.000)	0.773(95% CI 0.752 to 0.834)	1.000	0.643	0.615	1.000	
 Test	0.866(95% CI 0.821 to 0.883)	0.800(95% CI 0.772 to 0.841)	0.917	0.739	0.647	0.944	
ACC, accuracy; AUC, area under curve; LUNAI-eCTcontrast enhanced CT deep learning features modelLUNAI-fCTnon-contrast and contrast enhanced CT fused deep learning features modelLUNAI-uCTnon-contrast enhanced CT deep learning features modelNPV, negative predictive valuepCR, pathological complete response; PPV, positive predictive value

The Kolmogorov-Smirnov (KS) statistic test of the Immu_TR score was illustrated in figure 4a–c. The Immu_TR score derived from fused CT imaging features achieved the highest score of 0.825 (p<0.001) in distinguishing pCR and non-pCR groups across the entire patient population. The KS scores for non-contrast enhanced and contrast enhanced CT images were 0.704 (p<0.001) and 0.752 (p<0.001), respectively. The Immu_TR scores were visualized using t-distributed Stochastic Neighbor Embedding (figure 4d–f) and Uniform Manifold Approximation and Projection (figure 4g–i). These visualizations showed that the Immu_TR scores based on fused features provide clearer separation between pCR and non-pCR groups compared with the other methods.

Figure 4 Statistical analysis of the generated Immu_TR scores between pCR and non-pCR groups in all patient populations. Immu_TR, immunochemotherapy treatment response; pCR, pathological complete response.

Important model features and example model interpretation

The deep learning features in the FS-uCT, FS-eCT, and FS-fCT feature sets demonstrated low correlations (see online supplemental file). SHAP summary plots in figure 5 illustrate feature importance and their effects on the model’s prediction. In the LUNAI-uCT model (figure 5a), features 1 and 2 had the highest absolute SHAP values (longest bars), indicating a strong influence on predictions. For the LUNAI-eCT model (figure 5b), features 2 and 3 contributed the most significantly. For the LUNAI-fCT model (figure 5c), features 2 and 14 had the highest SHAP values, highlighting their substantial contributions to the model’s prediction.

Figure 5 Prediciton model interpretation by SHAP analysis for LUNAI-uCT, LUNAI-eCT, and LUNAI-fCT models in pCR prediction. LUNAI-eCT, contrast enhanced CT deep learning features model; LUNAI-fCT, non-contrast and contrast enhanced CT fused deep learning features model; LUNAI-uCT, non-contrast enhanced CT deep learning features model; pCR, pathological complete response; SHAP, SHapley Additive exPlanations.

Figure 6 shows an example implementation of the Immu_TR score generated by the LUNAI-fCT model for two patients with similar clinical characteristics. Both patient A and patient B had the same clinical T, N, and overall stages, smoking status, pathological type, lesion location, and similar ages and tumor volumes prior to neoadjuvant immunochemotherapy. Grad-CAM saliency maps indicated that the deep learning features were extracted from intra-tumoral regions. The Immu_TR scores generated for these patients were 0.791 and 0.213. Pathological evaluation confirmed that patient A achieved pCR, whereas patient B did not.

Figure 6 Schemtic shows an example implementation of the Immu_TR score in two patients with similar clinical characteristics, where patient A with pCR has a high Immu_TR score, and patient B with non-pCR has a low Immu_TR score. Grad-CAM saliency maps were generated for both non-contrast enhanced and contrast enhanced CT images. Grad-CAM, Gradient-weighted Class Activation Mapping; Immu_TR, immunochemotherapy treatment response; LUNAI-fCT, non-contrast and contrast enhanced CT fused deep learning features model; pCR, pathological complete response.

Discussion

In our study, we harnessed the capability of the FM-LCT foundation model to extract pertinent features from non-contrast enhanced and contrast enhanced CT images. The developed models (LUNAI-uCT, LUNAI-eCT, and LUNAI-fCT) demonstrated moderated performance in predicting pCR in NSCLC patients undergoing neoadjuvant immunochemotherapy. Notably, the LUNAI-fCT, which used fused features from both non-contrast enhanced and contrast enhanced CT images, achieved the best performance with an AUC of 0.866 (95% CI 0.821 to 0.883) in the test dataset. A novel Immu_TR score, derived from the predictive probability of the developed models, showed significant differences between pCR and non-pCR groups across all patient populations. This score offered a non-invasive, accurate method for predicting pCR in NSCLC patients to neoadjuvant immunochemotherapy, potentially aiding in identifying individuals most likely to benefit from this treatment strategy.

The burgeoning field of immunotherapy represents a transformative approach in the neoadjuvant treatment landscape of lung cancer. Its role in orchestrating the suppression of tumor-mediated immune evasion, coupled with the amplification of the host’s immune response to eradicate malignancies, is well-documented.3 24 Furthermore, there is a gradual increase in the number of lung cancer patients achieving pCR after receiving neoadjuvant immunochemotherapy.25 The ratio of residual tumor cells after surgery is closely linked to prognosis, confirming that pCR is valuable in predicting the survival of patients with NSCLC.26 Studies have demonstrated that an observational management approach, which avoids surgical organ preservation, is an effective choice for patients who achieve pCR.11 To prevent overtreatment, lung cancer management can draw on the observational strategies employed in colorectal cancer8 and esophageal cancer.9 10 Excessively aggressive lymph node dissection may diminish the efficacy of immunotherapy,12 thus the necessity of comprehensive lymph node dissection in all patients remains a topic of ongoing debate. Patients who achieve pCR may benefit from avoiding complete lymph node dissection, leading to improved immune maintenance therapy. However, it is important to note that confirmation of pCR can only be obtained through histopathological examination of surgical specimens. Consequently, the development of a non-invasive and accurate method to safely identify pCR after immunochemotherapy remains a significant challenge.

SITC Clinical Immuno-Oncology Network pointed out that immuno-oncology biomarker goals can be divided into risk biomarkers, diagnostic biomarkers, prognostic biomarkers, and predictive biomarkers, which can include molecular, histological, radiographic, and physiological indicators.27The quantitative imaging features captured from CT images may reflect potential tumor heterogeneity, angiogenesis, and cell density, which can reflect the biological behavior of tumors.28 29 Related studies have shown that quantitative imaging features extracted from CT images were correlated with immune microenvironment features, which can predict the pathological response of solid tumors.30 In previous studies, some research has predicted treatment response by extracting radiomics features before and after neoadjuvant immunochemotherapy. Liu et al included 89 lung cancer patients who underwent neoadjuvant immunochemotherapy, with 64 patients in the training group and 25 patients in the validation group. The radiomics model, combining clinical features, predicted major pathological response (MPR) with an AUC of 0.81 and an accuracy of 0.80 in the validation set.16 Han et al enrolled 206 lung cancer patients who underwent neoadjuvant immunochemotherapy and extracted radiomics features from contrast enhanced CT images taken before and after treatment to calculate the percentage of feature changes. These changes were integrated into a machine learning model for MPR prediction, resulting in an AUC of 0.732 and an accuracy of 0.660 in the testing group.31 Yang et al enrolled 110 lung cancer patients who received neoadjuvant immunochemotherapy and developed the NACIP model by extracting radiomics features from pretreatment and post-treatment contrast enhanced CT images. The NACIP model achieved an AUC of 0.85 and an accuracy of 0.81.4 She et al incorporated multi-center data and employed deep learning features to develop a predictive model for MPR in patients undergoing neoadjuvant immunochemotherapy, achieving an AUC of 0.75 in an external validation cohort.32 However, conventional deep learning feature extraction methods are typically based on models trained on natural images or limited datasets, limiting their applicability in medical research. To address the issue, our study employed a contrastive learning algorithm using MAE to train FM-LCT models on different datasets containing various types and stages of lung cancer as deep learning feature extractors.

Most research employed contrast enhanced CT to extract image features and construct models. However, no study has specifically investigated the disparities in constructing LUNAI models using both non-contrast enhanced and contrast enhanced CT. In our study, we independently extracted image features from both the non-contrast enhanced and contrast enhanced modalities of lung cancer patients before undergoing neoadjuvant immunochemotherapy. Subsequently, these features were fused to examine whether the fused model outperforms the single-modality prediction models. As a result, we constructed three models: LUNAI-uCT, LUNAI-eCT, and LUNAI-fCT. Significantly, LUNAI-fCT achieved the highest AUC value (0.866) among them and displayed superior performance compared with the other two models in additional evaluation metrics, including an accuracy of 0.800. In comparison to previous studies, LUNAI-fCT exhibited exceptional performance.

We used Grad-CAM to generate significant heatmaps, explaining how the FM-LCT foundation extracted deep learning features. The Grad-CAM saliency maps showed that the deep learning features were mainly extracted from intra-tumoral regions, but also involved information on tumor surrounding environments. Finally, two Immu_TR scores were generated, where patient A had a high Immu_TR score of 0.791 and patient B had a low Immu_TR score of 0.213. Pathological evaluation confirmed that patient A achieved pCR and patient B did not. This visualization of the model prediction process enhances the intuitiveness of clinical applications.

The study has several limitations and deficiencies. First, as a retrospective study, it may be subject to selection bias. Additionally, there are statistical differences in indicators such as gender, stage, and pathological type between the training group and the test group, potentially impacting model accuracy. Future prospective clinical studies should verify the model’s accuracy for improved performance. Second, although this is a multicenter study, the sample size in each center is small. Future research should aim for large sample sizes and multi-center validations. Third, the study did not include pretreatment biopsy specimen information, such as programmed cell death ligand 1 expression, tumor mutational burden, and sequencing data. Future research should integrate multimodal data such as pathology and sequencing to explore effective biomarkers for immunochemotherapy in lung cancer and achieve precise treatment for lung cancer immunochemotherapy.

In conclusion, the LUNAI-fCT, developed in this study, which combines deep learning features extracted from non-contrast enhanced and contrast-enhanced CT scans, demonstrating the highest predictive performance for pCR to neoadjuvant immunochemotherapy in patients with NSCLC. As an imaging biomarker, the LUNAI-fCT can non-invasively predict pCR to neoadjuvant immunochemotherapy in NSCLC. This capability has the potential to aid clinical decision-making in resectable NSCLC patients, prevent overtreatment, and facilitate personalized and precise cancer treatment.

supplementary material

10.1136/jitc-2024-009348 online supplemental file 1

Acknowledgements

The authors would like to express their gratitude for the support provided by MediAI Hub, an advanced medical image analysis software developed and maintained by MediaLab (non-commercial research group). Its powerful functions and efficient performance in medical image analysis have greatly contributed to the success of this research. We also extend our thanks to the R&D team of MediAI Hub for their continuous efforts and innovation in the field of medical image analysis.

Data availability statement

Data are available upon reasonable request.

Funding: This research was supported by the National Key Research and Development Program (No.2023YFC2508603), International Cooperation Project of Hubei Provincial Department of Science and Technology (No.2023EHA060), Wuhan Science and Technology Bureau Knowledge Innovation Special Project (No.2023020201010164), the National Science Foundation for Young Scientists of China (No. 82102034, 82202148, 62102103), the National Natural Science Foundation of China (No. 82472062, 82372044), the Natural Science Foundation of Guangdong Province of China (No.2024A1515011672), the Key-Area Research and Development Program of Guangdong Province, China (No.2021B0101420006), Regional Innovation and Development Joint Fund of National Natural Science Foundation of China (No.U22A20345).

Provenance and peer review: Not commissioned; externally peer reviewed.

Patient consent for publication: Not applicable.

Ethics approval: This study involves human participants and was approved by the Ethics Committee of Union Hospital, Tongji Medical College, Huazhong University of Science and Technology (No. UCT240116). The requirement for patient approval or written informed consent was waived due to the retrospective nature of this study.
==== Refs
References

1 Siegel RL Miller KD Fuchs HE et al Cancer statistics, 2022 CA Cancer J Clin 2022 72 7 33 10.3322/caac.21708 35020204
2 Lahiri A Maji A Potdar PD et al Lung cancer immunotherapy: progress, pitfalls, and promises Mol Cancer 2023 22 40 10.1186/s12943-023-01740-y 36810079
3 Wang C Chen K-N Chen Q et al Neoadjuvant nivolumab plus chemotherapy versus chemotherapy for resectable NSCLC: subpopulation analysis of Chinese patients in CheckMate 816 ESMO Open 2023 8 102040 10.1016/j.esmoop.2023.102040 37922691
4 Yang N Yue H Zhang B et al Predicting pathological response to neoadjuvant or conversion chemoimmunotherapy in stage IB–III non‐small cell lung cancer patients using radiomic features Thorac Cancer 2023 14 2869 76 10.1111/1759-7714.15052 37596822
5 Zhao J Hao S Li Y et al Comparative Efficacy and Safety of Neoadjuvant Immunotherapy with Chemotherapy versus Chemotherapy Alone in Non-Small Cell Lung Cancer: A Propensity Score and Inverse Probability Treatment Weighting Analysis Immunotargets Ther 2023 12 113 33 10.2147/ITT.S437911 38026088
6 Provencio M Nadal E Insa A et al Neoadjuvant chemotherapy and nivolumab in resectable non-small-cell lung cancer (NADIM): an open-label, multicentre, single-arm, phase 2 trial Lancet Oncol 2020 21 1413 22 10.1016/S1470-2045(20)30453-8 32979984
7 Reck M Remon J Hellmann MD First-Line Immunotherapy for Non-Small-Cell Lung Cancer J Clin Oncol 2022 40 586 97 10.1200/JCO.21.01497 34985920
8 Smith JJ Strombom P Chow OS et al Assessment of a Watch-and-Wait Strategy for Rectal Cancer in Patients With a Complete Response After Neoadjuvant Therapy JAMA Oncol 2019 5 e185896 10.1001/jamaoncol.2018.5896 30629084
9 de Gouw D Klarenbeek BR Driessen M et al Detecting Pathological Complete Response in Esophageal Cancer after Neoadjuvant Therapy Based on Imaging Techniques: A Diagnostic Systematic Review and Meta-Analysis J Thorac Oncol 2019 14 1156 71 10.1016/j.jtho.2019.04.004 30999111
10 Yap W-K Hsiao I-T Yap W-L et al A Radiotherapy Dose Map-Guided Deep Learning Method for Predicting Pathological Complete Response in Esophageal Cancer Patients after Neoadjuvant Chemoradiotherapy Followed by Surgery Biomedicines 2023 11 3072 10.3390/biomedicines11113072 38002072
11 Marijnen CAM Organ preservation in rectal cancer: have all questions been answered? Lancet Oncol 2015 16 e13 22 10.1016/S1470-2045(14)70398-5 25638548
12 Deng H Zhou J Chen H et al Impact of lymphadenectomy extent on immunotherapy efficacy in postresectional recurred non-small cell lung cancer: a multi-institutional retrospective cohort study Int J Surg 2024 110 238 52 10.1097/JS9.0000000000000774 37755384
13 Maitland ML Wilkerson J Karovic S et al Enhanced Detection of Treatment Effects on Metastatic Colorectal Cancer with Volumetric CT Measurements for Tumor Burden Growth Rate Evaluation Clin Cancer Res 2020 26 6464 74 10.1158/1078-0432.CCR-20-1493 32988968
14 Chen M Copley SJ Viola P et al Radiomics and artificial intelligence for precision medicine in lung cancer treatment Semin Cancer Biol 2023 93 97 113 10.1016/j.semcancer.2023.05.004 37211292
15 Wu G Jochems A Refaee T et al Structural and functional radiomics for lung cancer Eur J Nucl Med Mol Imaging 2021 48 3961 74 10.1007/s00259-021-05242-1 33693966
16 Liu C Zhao W Xie J et al Development and validation of a radiomics-based nomogram for predicting a major pathological response to neoadjuvant immunochemotherapy for patients with potentially resectable non-small cell lung cancer Front Immunol 2023 14 1115291 10.3389/fimmu.2023.1115291 36875128
17 Lin Q Wu HJ Song QS et al CT-based radiomics in predicting pathological response in non-small cell lung cancer patients receiving neoadjuvant immunotherapy Front Oncol 2022 12 937277 10.3389/fonc.2022.937277 36267975
18 Goldstraw P Chansky K Crowley J et al The IASLC Lung Cancer Staging Project: Proposals for Revision of the TNM Stage Groupings in the Forthcoming (Eighth) Edition of the TNM Classification for Lung Cancer J Thorac Oncol 2016 11 39 51 10.1016/j.jtho.2015.09.009 26762738
19 Moons KGM Altman DG Reitsma JB et al Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration Ann Intern Med 2015 162 W1 73 10.7326/M14-0698 25560730
20 Travis WD Dacic S Wistuba I et al IASLC Multidisciplinary Recommendations for Pathologic Assessment of Lung Cancer Resection Specimens After Neoadjuvant Therapy J Thorac Oncol 2020 15 709 40 10.1016/j.jtho.2020.01.005 32004713
21 He K Chen X Xie S Masked autoencoders are scalable vision learners In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2022
22 Xue B Li D Lu C et al Use of Machine Learning to Develop and Evaluate Models Using Preoperative and Intraoperative Data to Identify Risks of Postoperative Complications JAMA Netw Open 2021 4 e212240 10.1001/jamanetworkopen.2021.2240 33783520
23 Shi G-Y Wu H-P Luo S-H et al 1D Gradient-Weighted Class Activation Mapping, Visualizing Decision Process of Convolutional Neural Network-Based Models in Spectroscopy Analysis Anal Chem 2023 95 9959 66 10.1021/acs.analchem.3c01101 37351568
24 Wang M Herbst RS Boshoff C Toward personalized treatment approaches for non-small-cell lung cancer Nat Med 2021 27 1345 56 10.1038/s41591-021-01450-2 34385702
25 Wu Y Verma V Gay CM et al Neoadjuvant immunotherapy for advanced, resectable non-small cell lung cancer: A systematic review and meta-analysis Cancer 2023 129 1969 85 10.1002/cncr.34755 36994945
26 Deutsch JS Cimino-Mathews A Thompson E et al Association between pathologic response and survival after neoadjuvant therapy in lung cancer Nat Med 2024 30 218 28 10.1038/s41591-023-02660-6 37903504
27 Lotze MT Cottrell T Bifulco C et al SITC Clinical Immuno-Oncology Network (SCION) commentary on measurement and interpretation of essential biomarkers in early clinical trials J Immunother Cancer 2024 12 e008655 10.1136/jitc-2023-008655 38519056
28 Lee JY Lee K-S Seo BK et al Radiomic machine learning for predicting prognostic biomarkers and molecular subtypes of breast cancer using tumor heterogeneity and angiogenesis properties on MRI Eur Radiol 2022 32 650 60 10.1007/s00330-021-08146-8 34226990
29 Shi Z Huang X Cheng Z et al Erratum for: MRI-based Quantification of Intratumoral Heterogeneity for Predicting Treatment Response to Neoadjuvant Chemotherapy in Breast Cancer Radiology 2023 308 e239021 10.1148/radiol.239021 37489997
30 Holder AM Dedeilia A Sierra-Davidson K et al Defining clinically useful biomarkers of immune checkpoint inhibitors in solid tumours Nat Rev Cancer 2024 24 498 512 10.1038/s41568-024-00705-7 38867074
31 Han X Wang M Zheng Y et al Delta-radiomics features for predicting the major pathological response to neoadjuvant chemoimmunotherapy in non-small cell lung cancer Eur Radiol 2024 34 2716 26 10.1007/s00330-023-10241-x 37736804
32 She Y He B Wang F et al Deep learning for predicting major pathological response to neoadjuvant chemoimmunotherapy in non-small cell lung cancer: A multicentre study EBioMedicine 2022 86 104364 10.1016/j.ebiom.2022.104364 36395737
