
==== Front
NPJ Precis Oncol
NPJ Precis Oncol
NPJ Precision Oncology
2397-768X
Nature Publishing Group UK London

39251820
699
10.1038/s41698-024-00699-3
Review Article
Illuminating the future of precision cancer surgery with fluorescence imaging and artificial intelligence convergence
http://orcid.org/0000-0002-0081-267X
Cheng Han 12345
http://orcid.org/0009-0005-5382-4497
Xu Hongtao 12345
http://orcid.org/0009-0006-6971-0897
Peng Boyang 6
http://orcid.org/0000-0003-0981-4206
Huang Xiaojuan 12345
http://orcid.org/0000-0001-6181-8912
Hu Yongjie 12345
http://orcid.org/0009-0000-0142-7653
Zheng Chongyang cy.zheng0412@gmail.com

12345
http://orcid.org/0000-0002-4388-880X
Zhang Zhiyuan zhzhy0502@126.com

12345
1 grid.16821.3c 0000 0004 0368 8293 Department of Oral and Maxillofacial-Head & Neck Oncology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200011 P. R. China
2 https://ror.org/0220qvk04 grid.16821.3c 0000 0004 0368 8293 College of Stomatology, Shanghai Jiao Tong University & National Center for Stomatology, Shanghai, 200011 P. R. China
3 grid.412523.3 0000 0004 0386 9086 National Clinical Research Center for Oral Diseases & Shanghai Key Laboratory of Stomatology, Shanghai, 200011 P. R. China
4 Shanghai Research Institute of Stomatology, Shanghai, 200011 P. R. China
5 https://ror.org/02drdmm93 grid.506261.6 0000 0001 0706 7839 Research Unit of Oral and Maxillofacial Regenerative Medicine, Chinese Academy of Medical Sciences, Shanghai, 200011 P. R. China
6 https://ror.org/03r8z3t63 grid.1005.4 0000 0004 4902 0432 School of Computer Science and Engineering, University of New South Wales, Sydney, Australia
9 9 2024
9 9 2024
2024
8 1968 4 2024
29 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Real-time and accurate guidance for tumor resection has long been anticipated by surgeons. In the past decade, the flourishing material science has made impressive progress in near-infrared fluorophores that may fulfill this purpose. Fluorescence imaging-guided surgery shows great promise for clinical application and has undergone widespread evaluations, though it still requires continuous improvements to transition this technique from bench to bedside. Concurrently, the rapid progress of artificial intelligence (AI) has revolutionized medicine, aiding in the screening, diagnosis, and treatment of human doctors. Incorporating AI helps enhance fluorescence imaging and is poised to bring major innovations to surgical guidance, thereby realizing precision cancer surgery. This review provides an overview of the principles and clinical evaluations of fluorescence-guided surgery. Furthermore, recent endeavors to synergize AI with fluorescence imaging were presented, and the benefits of this interdisciplinary convergence were discussed. Finally, several implementation strategies to overcome technical hurdles were proposed to encourage and inspire future research to expedite the clinical application of these revolutionary technologies.

Subject terms

Surgical oncology
Mathematics and computing
https://doi.org/10.13039/501100001809 National Natural Science Foundation of China (National Science Foundation of China) 82002853 Zhang Zhiyuan Shanghai Clinical Research Center for Oral Diseases (19MC1910600), Shanghai Municipal Key Clinical Specialty (shslczdzk01601), Shanghai's Top Priority Research Center (2022ZZ01017), CAMS Innovation Fund for Medical Sciences (CIFMS) (2019-I2M-5-037)Zhejiang Medical and Health Technology Plan (2020KY939)issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Surgeons have long relied on physical, radiological, and pathological examinations to guide tumor resection1, but these approaches fall short of delivering real-time, comprehensive information on tumors to guide complete excision. This limitation can lead to potential oversight of residual tumors, which may culminate in cancer recurrence and increased mortality2,3. For instance, initial positive margins were found in about 20% of head and neck cancer surgery4–6; and a similar prevalence exists in breast conservation surgery, where approximately 10% of patients experience involved surgical margin, strongly correlating with recurrence risk7,8. Such a challenge is typically addressed by expanding the extent of resection. However, surgeries today are evolving towards the emphasis on the preservation of form, function, and quality of life, meanwhile maximizing surgical outcomes9. This “conservative” form of surgery requires a precise treatment design based on patient-specific lesion characterization. Therefore, there is an urgent imperative for the development of robust decision-making assistance to facilitate the realization of precision cancer surgery.

It is believed by the majority that image-guided surgery, particularly under fluorescence imaging guidance, could largely improve surgical outcomes and minimize the potential overtreatment10. Thanks to the thriving development in material science, various fluorescence imaging techniques were developed to improve surgical vision and provide intraoperative guidance. The near-infrared (NIR) fluorescence imaging provides excellent spatial resolution and convenient real-time visualization of cancer2,11,12. Relying on the fluorophores that specifically accumulate within tumors, the different fluorescence signals between cancerous and normal tissues could be highlighted, thereby realizing the accurate delineation of tumor extent13,14. The accurate identification of cancerous tissues offered by fluorescence guidance may facilitate the complete and conservative resection of tumors to minimize postoperative recurrence, reduce functional impairment caused by undue tissue loss, and substantially improve the long-term quality of life of patients, thus introducing major innovation to cancer surgery15. Nevertheless, there remain several technical issues, such as the specific accumulation of fluorophores within the tumor, light penetration, and the detection and analysis of signals, which are the major aspects demanding additional refinement16.

Artificial intelligence (AI) technology has gradually permeated into the healthcare field over the past few years, with intelligent technology added to the commonplace clinical practice17. Since these computational methods excel at processing complex, massive, and abstract data that would require human intelligence, AI-related techniques have been employed to develop various intelligent medical applications. Examples of such technological convergence include intelligent radiologic and pathologic diagnosis18,19, intraoperative surgical guidance20, and autonomous surgical robotics21, all of which leave a strong impression on healthcare professionals. AI has also been utilized to enhance fluorescence imaging, assisting in data computation, imaging analysis, and pattern recognition, which significantly improved the imaging quality and compensated for the drawbacks of fluorophores22. This combinatorial strategy is likely to overcome the technical hurdles and accelerate the clinical translation of fluorescence imaging, fulfilling the requirements of precision cancer surgery.

This review aims to promote the integration of fluorescence imaging and AI to further enhance intraoperative decision-making support, thereby assisting the development of precision cancer surgery and improving the surgical outcomes of patients. In addition, we proposed several approaches to realize AI-enhanced fluorescence imaging to inspire further research. In the future, one can envision this synergistic strategy to bring major innovation to the practice of surgery, by strengthening human-AI collaboration, as briefly demonstrated in Fig. 1.Fig. 1 The workflow of surgical decision-making and AI-enhanced fluorescence-guided precision surgery.

Briefly, current surgical decision-making is mostly supported by the intraoperative frozen section inspection and postoperative pathologic examination of the specimen. However, these techniques fall short of providing real-time guidance for surgeons and may result in positive surgical margins. This failure to completely remove the tumor brings about the increased risk of tumor recurrence, which is commonly addressed by adjuvant therapy or second-stage tumor resection. Fluorescence imaging guidance helps to delineate the extent of the primary tumor and could discover occult or satellite lesions that may evade the human eye in real time. Currently, AI techniques have been explored to further enhance the quality of the imaging modality and may robustly support the decision-making process, fulfilling the requirements of precision cancer surgery. In the future, this AI-enhance fluorescence imaging, along with other cutting-edge technology such as augmented reality, may realize the seamless human-AI collaboration and bring major innovation to precision surgery. This figure is created with Adobe Illustrator 2022.

Fundamentals of cancer fluorescence imaging

The realization of fluorescence-guided surgery (FGS) involves the administration of fluorophores that accumulate abundantly within tumors and are swiftly cleared in the peripheral normal tissues. After external light irradiation and energy transfer, the electrons of the fluorophore molecules transition from the ground state to the excited state, which is shortly followed by the reverse process and is accompanied by the emission of the received energy, thereby emitting fluorescence that is captured by a detection instrument12,23. In comparison, autofluorescence imaging detects the light-induced inherent fluorescence of human tissues, which typically yields weak contrast between normal and cancerous tissues and, therefore, is less commonly employed to guide surgery24.

To obtain optimal highlight of tumors, the NIR fluorescence is commonly adopted, which covers an emission spectrum (700-1700 nm) with reduced background noise stemming from photon scattering, light absorption, and tissue autofluorescence, achieving a high tumor-to-background ratio (TBR)25. Moreover, the NIR window can be further divided into the NIR-I region (700-1000 nm) and the NIR-II region (1000-1700 nm). Compared with the NIR-I fluorescence, emission in the NIR-II spectrum exhibits further reduced noise, enhanced imaging resolution, and better tissue penetration to increase imaging depth26, as depicted in Fig. 2. Therefore, current research efforts are largely attracted to developing fluorophores with peak emission in the NIR-II range but are generally hindered from clinical use due to biosafety problems27. We here present several milestone fluorophores that have undergone clinical evaluation in cancer surgery, and the discovered technical issues that require further optimization.Fig. 2 The basic mechanism of NIR fluorescence imaging.

Briefly, the external laser irradiation induces the fluorophores to emit light in the NIR spectrum (700-1700 nm), which is later captured by NIR sensors (often accompanied by white light sensor), followed by data processing to analyze the signal intensity and distribution. Moreover, the NIR spectrum can be further divided into NIR-I (700-900 nm) and NIR-II (900–1700 nm), with NIR-II imaging demonstrating reduced absorbance, light scattering, and tissue autofluorescence, as well as increased tissue penetration, enabling a more accurate, localized, and sharp highlight of the tumor. This figure is created with Adobe Illustrator 2022.

Fluorophores are a key element in fluorescence-guided surgery. Until now, indocyanine green (ICG), methylene blue (MB), and 5-aminolevulinic acid (5-ALA) are the only three agents with fluorescent properties authenticated to be applied in clinical settings. These agents have been used for tumor resection guidance28, metastatic disease identification29, and sentinel lymph node mapping30. While demonstrating significant clinical benefits, the further application of these agents was hampered by their low cancer-specificity and poor photophysical properties, such as small Stokes shift, low signal intensity, and poor photostability. This has led to a less satisfactory contrast of tumor31.

To meet the demand for fluorescence-guided cancer surgery, research efforts in the past few years have mainly focused on improving the cancer selectivity of fluorophores32. The conjugation of targeting molecules to fluorophores represents the most common strategy to realize cancer-specific fluorescence imaging, as summarized in Supplementary Table 1. A notable example is the combination of anti-EGFR antibodies (such as cetuximab and panitumumab) with IRDye800CW, which enabled the targeted fluorescence imaging of squamous cell carcinoma and glioblastoma33,34. Additionally, other affinity molecule-labeled fluorophores, represented by ABY-029 (anti-EGFR affibody linked to IRDye800CW), also exhibit preferential distribution into tumors while rapidly clearing from normal tissues, thereby achieving higher tumor contrast35. Beyond the surface receptors, intracellular overexpressed molecules have also been selected as appropriate targets for fluorescence imaging. For example, PARP1 inhibitors conjugated with BODIPY-FL (PARPi-FL) have been utilized in imaging various tumors36. Another strategy involves developing tumor-specific fluorophores by leveraging abundant molecules within the tumor microenvironment (TME). For instance, ONM-100, a pH-sensitive probe based on ICG, exhibits fluorescence selectively in the acidic TME, demonstrating an “on-off” capability that enhances TBR and facilitates precise tumor delineation37 (please refer to Supplementary Content 1 for more technical details).

Clinical applications of fluorescence-guided cancer surgery

With the aid of burgeoning material science, current cancer fluorescence imaging is advancing both in sensitivity and specificity38, thus offering satisfactory delineation of tumors. The enhanced vision provided by intraoperative fluorescence guidance serves various functions, such as the delineation of cancer margin for complete resection28, the identification of occult and satellite lesions present within or outside the primary tumor39, the depth indication for maximal tumor debulking40, mapping sentinel lymph nodes41,42 and the visualization of critical structures43, covering the majority of cancer surgical practice. We here summarize the clinical evaluations of the imaging agents, and discuss existing challenges that necessitate further optimization for the advancement of precision cancer surgery.

Workflow of intraoperative cancer fluorescence imaging

During the clinical evaluation of the targeted fluorophores mentioned above, various imaging and analysis approaches were explored44,45. The typical workflow of intraoperative fluorescence guidance involves several steps. First, the tumor is imaged in situ with a wide-field imaging device to illuminate the extent of the primary tumor and identify potential satellite lesions. After tumor resection, the surgical cavity (wound bed) is imaged to check for positive margins, which may necessitate re-resection. Subsequently, the excised specimen is processed in the surgical back table to perform ex vivo close-field fluorescence imaging, which may also inform on the margin status and support real-time surgical decision-making, as demonstrated in Fig. 3. Comparatively, the current in situ fluorescence imaging excels at providing simultaneous surgical guidance and identifying sub-clinical lesions, while ex vivo specimen imaging exhibits better diagnostic efficacy due to the removal of various confounding factors of in situ imaging. Both methods support intraoperative decision-making13,46,47.Fig. 3 Clinical workflow of fluorescence-guided surgery.

a The preoperative bright field and b fluorescence images of the primary lesion. c The bright-field and d open-field fluorescence images of the deep surface of a resected tumor specimen. Positive areas of fluorescence were detected on e the superficial surface, but not on f the deep surface of the additional resection margin, informing the margin status. g H&E staining of a sectioned tissue specimen from a primary tumor. The outlined areas indicate the presence of a tumor. h Fluorescence scan of the adjacent cut section, with outlined areas displaying bright fluorescence intensity. Reproduced with permission from Zhang et al.33. Copyright 2017, Springer Nature.

Challenges in the analysis of fluorescence imaging

Several targeted fluorophores have been evaluated in phase I and II clinical trials, during which the imaging quality of both in situ fluorescence guidance and ex vivo specimen examination were also investigated, as summarized in Tables 1 and 2. These results showcased several indications for further improvement of this technology, identifying the quantitative analysis of fluorescence signals as the most notable challenge. As the major parameter of diagnostic performance of in situ fluorescence imaging, the TBR can be influenced by various factors, such as the dose of the fluorophore, the ambient light in the operating room, camera orientation, distance, confounding signals from adjacent normal tissue, etc48. These confounding factors resulted in the highly heterogeneous signal fluctuation, as well as lowering the calculated TBR, reducing imaging quality. Also, in situ margin delineation could be influenced by previous radiotherapy, exhibiting larger deviation of fluorescence borders from pathologic borders, indicating the altered TME structure and fluorophore distribution profile49. These variables, together with the heterogeneity among individual patients and lesions, will ultimately lead to the impaired validity of quantitative signal analysis. The limitations of in situ imaging can be partly mitigated by ex vivo specimen inspection using black-box imaging devices. This method provides a more homogeneous examination and retains the capability to discern close surgical margin50 though it is not comparably intuitive to that of in situ imaging.Table 1 Clinical evaluation of in situ fluorescence imaging

Fluorophore	Trial type	Population size	Cancer type	Time of administration	Study dose	Signal calibration	Tumor-to-background ratio	Positive resection margin	Occult lesion	Decision-making assistance	Reference	
Cetuximab-IRDye800	Phase-I	12	HNSCC	3–4 d pre-surgery	2.5 mg/m2 (n = 3)

25 mg/m2 (n = 6)

67.5 mg/m2 (n = 3)

	Cervical skin and forearm skin	25 mg/m2 TBR = 4.3 (2.1–7.8)

62.5 mg/m2 TBR = 5.2 (4.8–6)

	/	/	/	44	
Cetuximab-IRDye800	Phase-I	6	HNSCC	3–4 d pre-surgery	10* + 25 mg/m2 (n = 3)

100* + 25 mg/m2 (n = 3)

	Background tissue was defined as 3-4 cm from the edge of the gross tumor.	10* + 25 mg/m2 TBR = 1.7 ± 0.7

100* + 25 mg/m2 TBR = 5.5 ± 2.6

	/	/	/	97	
Cetuximab-IRDye800	Phase-I	3	GBM	2–5 d pre-surgery	50 mg (n = 2)

100 mg (n = 1)

	TBR > 1						
Cetuximab-IRDye800	Phase-II	65	OSCC	2 d pre-surgery	75* + 15 mg	Selecting normal mucosa as background tissue	TBR = 3.1 (2.0–5.4)	Positive margin (0/1)	3	/	49,52	
Panitumumab-IRDye800CW	Phase-I	18	HNSCC	1–5 d pre-surgery	100* + 0.5 mg/kg (n = 5)

100*1.0 mg/kg (n = 7)

50 mg (n = 6)

	Peritumoral tissue was chosen as the background fluorescence	100* + 0.5 mg/kg TBR = 2.4 ± 0.4

100* + 1.0 mg/kg TBR = 2.6 ± 0.4

50 mg TBR = = 2.5 ± 0.4

	/	/	/	13	
Panitumumab-IRDye800CW	Phase-I	14	HNSCC	1–5 d pre-surgery	/	Selecting normal tissue with the most constant signal and coefficient of variance as background.	TBR = 1.8–2.7	1	1	3	98	
ONM-100	Phase-I	13	HNSCC	24 h pre-surgery	0.1 mg/kg (n = 3)

0.5 mg/kg (n = 2)

0.8 mg/kg (n = 1)

1.2 mg/kg (n = 7)

	/	1.2 mg/kg TBR = 2.6	6	1	/	99,100	
ONM-100	Phase-I	11	BC	24 h pre-surgery	0.3 mg/kg (n = 3)

0.5 mg/kg (n = 1)

0.8 mg/kg (n = 2)

1.2 mg/kg (n = 5)

	/	/	Positive margin (3/3)	4	1	99	
ONM-100	Phase II	11	HNSCC	24 h pre-surgery	1 mg/kg	/	2.608 (2.5936)	/	/	/	NCT03735680	
ONM-100	Phase II	40	PC	2–3 d pre-surgery	1 mg/kg	/	/	/	20	/	101	
PARPi-FL	Phase-I/II	12	OSCC	60 s mouthwash, 60 s clearing	15 mL mouthwash

100 nM (n = 3)

250 nM (n = 3)

500 nM (n = 3)

1000 nM (n = 3)

	Tumor to non-tumor ratio was calculated by dividing each tumor ROI by the average of non-tumor ROIs in its respective frame.	PARPi-FL post wash

100 mM TBR = 2.1

250 mM TBR = 2.1

500 mM TBR = 2.8

1000 mM TBR = 3.3

	/	1	/	102	
ICG	/	20	OSCC	6–8 h pre-surgery	0.75 mg/kg	Fluorescence signal of the surgical margin was defined as peritumoral signal. Fluorescence signal of tissues away from the tumor was defined as normal tissue signal.	Tumor vs. normal TBR = 1.56 ± 0.41

Tumor vs. peritumor TBR = 1.45 ± 0.36

	Positive margin (2/4)	2	/	103	
The asterisk (*) in the “study dose” column indicates the pre-dose of unbound monoclonal antibody prior to infusion of conjugated fluorophores. Signal calibration refers to the process of choosing the comparison tissue as the background to calculate TBR. Positive resection margin identification is performed by in situ fluorescence imaging of the surgical cavity after tumor resection. Decision-making assistance is defined as surgical plan alteration based on clinically undetectable lesions or positive margins identified with fluorescence imaging. HNSCC head and neck squamous cell carcinoma, OSCC oral squamous cell carcinoma, BC breast cancer, PC peritoneal carcinomatosis, TBR tumor-to-background ratio, ICG indocyanine green.

Table 2 Clinical evaluation of ex vivo surgical specimen fluorescence imaging

Fluorophore	Trial type	Sample volume	Cancer type	Time of administration	Study dose	Signal calibration	ex vivo specimen TBR	Slices TBR	Histology conformity	Positive margin diagnostic performance	Decision-making assistance	Reference	
Cetuximab-IRDye800	Phase-I	12	HNSCC	3–4 d pre-surgery	2.5 mg/m2 (n = 3)

25 mg/m2 (n = 6)

67.5 mg/m2 (n = 3)

	Skin and muscle tissue	2.5 mg/m2 TBR = 5.9

25 mg/m2 TBR = 9.5

62.5 mg/m2 TBR = 5.7

	2.5 mg/m2 TBR = 2

25 mg/m2 TBR = 2.9

62.5 mg/m2 TBR = 1.7

	Yes	Sensitivity 92.0%

specificity 74.5%

	1	44,51	
Cetuximab-IRDye800	Phase-I	15	HNSCC	4 d pre-surgery	10 mg (n = 3)

25 mg (n = 3)

50 mg (n = 3)

75* + 15 mg (n = 3)

75* + 25 mg (n = 3)

	The specimen was used as an internal control	10 mg TBR = 1.61 ± 0.93

25 mg TBR = 2.02 ± 0.55

50 mg TBR = 1.81 ± 0.32

75* + 15 mg TBR = 3.06 ± 0.43

75* + 25 mg TBR = 3.10 ± 2.53

	/	Yes	Sensitivity 100% (4/4)

specificity 91% (10/11)

	/	28	
Cetuximab-IRDye800	Phase-I	3	GBM	2–5 d pre-surgery	50 mg (n = 2)

100 mg (n = 1)

	The threshold of the weight of a “detectable” tumor (TBR > 1) was determined	50 mg TBR = 1.58 (70 mg GBM)

100 mg TBR = 2.56 (10 mg GBM)

	50 mg TBR = 3.6

100 mg TBR = 4.3

	Yes	/	/	104	
Cetuximab-IRDye800	Phase-II	16	OSCC	2 d pre-surgery	75* + 15 mg (n = 16)	Adjacent tissue without fluorescent signal.	Optimal cut-off for positive margin: TBR ≥ 2

Optimal cut-off for close margin: TBR ≥ 1.5

	/	Yes	Positive margin:

sensitivity 100%

specificity 85.9%

Close margin:

sensitivity 70.3%

specificity 76.1%

	/	49,52	
Panitumumab-IRDye800	Phase-I	18	HNSCC	1–5 d pre-surgery	100* + 0.5 mg/kg (n = 5)

100*1.0 mg/kg (n = 7)

50 mg (n = 6)

	Muscle tissue.	100* + 0.5 mg/kg (n = 5) TBR = 5.40 ± 0.6

100* + 1.0 mg/kg (n = 7) TBR = 5.44 ± 0.7

50 mg (n = 6) TBR = = 6.53 ± 1.2

	/	Yes	Sensitivity 91%

specificity 88%

	/	13	
Panitumumab-IRDye800	Phase-I	8	HNSCC	1–4 d pre-surgery	25 mg (n = 4)

50 mg (n = 4)

	Signal thresholding	/	/	Yes	5 mm margin

sensitivity 95%

specificity 89%

2 mm margin

sensitivity 100%

	/	47	
ONM-100	Phase-I	13	HNSCC	24 h pre-surgery	0.1 mg/kg (n = 3)

0.5 mg/kg (n = 2)

0.8 mg/kg (n = 1)

1.2 mg/kg (n = 7)

	Non-fluorescent resection planes.	1.2 mg/kg TBR = 4.5 (median)	3.36 ± 1.62	Yes	Sensitivity 100% (6/6)

specificity 57%

	/	99,100	
ONM-100	Phase 2			2–3 d pre-surgery	1 mg/kg								
PARPi-FL	Phase-I/II	12	OSCC	60 s mouthwash, 60 s clearing	15 mL mouthwash

100 nM (n = 3)

250 nM (n = 3)

500 nM (n = 3)

1000 nM (n = 3)

	/	/	/	PARPI-FL uptake: 45.14% tumor cells, 3.89% normal cells	/	/	102	
ICG	/	20	OSCC	6–8 h pre-surgery	0.75 mg/kg	Fluorescence signal of the surgical margin was defined as peritumoral signal. Fluorescence signal of tissues away from the tumor was defined as normal tissue signal.	Tumor vs. normal TBR = 1.43 ± 0.27

Tumor vs. peritumor TBR = 1.38 ± 0.22

	/	/	/	/	103	
The asterisk (*) in the “study dose” column indicates the pre-dose of unbound monoclonal antibody prior to infusion of conjugated fluorophores. Signal calibration refers to the process of choosing the comparison tissue as the background to calculate TBR. Slices TBR was acquired from imaging performed on pathologic slices. Histopathological conformity is represented by the correlation of wide-field/close-field fluorescence signal distribution or intensity with that of histopathological examination. Decision-making assistance is defined as surgical plan alteration based on clinically undetectable lesions or positive margins identified with fluorescence imaging. HNSCC head and neck squamous cell carcinoma, GBM glioblastoma, OSCC oral squamous cell carcinoma, TBR tumor-to-background ratio, ICG indocyanine green.

To enhance the consistency of quantitative fluorescence analysis, superior computational methods can be employed. For instance, a ratiometric threshold based on fluorescence signal intensity was introduced to determine the border of tumor and normal tissue in situ imaging51. In a study evaluating the diagnostic accuracy, this was done by choosing four ratios (25%, 50%, 75%, and 100%) of the maximal fluorescence intensity, imaging border was then compared with the corresponding pathologic border, and the most suitable threshold was selected49. Similar analyses were also conducted in ex vivo specimen fluorescence imaging and evaluated the diagnostic performance of different ratios, suggesting a TBR ≥ 2 holds the advantage in identifying positive surgical margin, whereas a TBR ≥ 1.5 outperforms in delineating close surgical margin52. These results suggest that advanced computational methods could enhance the quantitative analysis of fluorescence intensity and discern the patterns underlying variations, thereby enabling fine-tuning of diagnostic performance.

To recapitulate briefly, clinical evaluations revealed promising advantages of cancer fluorescence imaging to provide intraoperative guidance and support decision-making, as well as the challenges in the analysis and optimization of this modality. It is advisable to develop more advanced data interpretation methods alongside the progression of imaging techniques.

AI-enabled precision fluorescence guidance

In the past decade, the world has witnessed the galloping advancement of AI techniques that have been broadly applied to healthcare, aiding in radiologic and pathologic diagnosis, drug development and usage, providing surgical guidance, medical robotics, and so forth20,53,54. AI-based techniques excel at processing abstract, complex, and large volumes of data, as well as accurately and consistently performing tasks like learning information, recognizing patterns, and aiding in decision-making, leading the way of healthcare into the era of precision medicine. In the following section, we introduce the commonly adopted AI technologies, their contributions to fluorescence imaging, and the complementary effects that could realize precision cancer surgery.

Common models for fluorescence imaging analysis

Neural networks (NN) are an integral subset of machine learning (ML), relying on the algorithmic structure mimicking human neural networks, composed of multiple layers of neurons that calculate the input and pass on the output to the next neuron, extracting features from data in an accurate, flexible, and efficient manner, and is capable of learning and improving over time55,56. Therefore, the NNs excel at performing pattern recognition and calibration activities in comparison to conventional statistical methods, which is a prerequisite to reducing the heterogeneity of fluorescence characteristics among individuals22. Taking the convolutional neural networks (CNNs) as a typical example of medical image processing, the workflow entails pre-processing of the image, followed by image segmentation that divides the pictures into multiple segments according to the color, texture, and brightness, etc., and performing a feature extraction process to identify and distinguish key features of the image, and ends with a classification layer based on reasoning the weight acquired from the previous stages and conclude on the identification of cancerous tissues57. These structures meet the need to process the grid-like topology of the image well, making it an ideal structure for the data processing of fluorescence imaging58,59, as demonstrated in Fig. 4.Fig. 4 The typical structure of a convolutional neural network.

As the figure demonstrated, data augmentation techniques such as rotation, scaling, and flipping are applied to increase the diversity of the training data and improve the model’s robustness before feeding the image into the neural network. Then a 5 × 5 kernel of the first convolution layer extracts from the original 500 × 500 grayscale image into a 496 × 496 array of features; subsampling (max pooling) then halves the size into a 248 × 248 array of features. The second Convolution layer implemented another 5 × 5 kernel to extract a 244 × 244 array of features, which the second subsampling (max pooling) layer further reduces into a 122 × 122 array of features. Later, these neurons get flattened and then pass through two fully connected layers into outputs of ‘1’ and ‘0’, which means ‘tested positive for tumor’ and ‘tested negative for tumor’. This figure is created with Adobe Illustrator 2022.

To date, various machine learning models have been designed, such as recurrent neural networks (RNNs), generative adversarial networks (GANs), and autoencoders (AEs), each specializes in a unique function and has been applied to healthcare purposes17. Each model has unique capabilities, and combining different neural network (NN) models often results in complex, multi-functional systems to solve a particular set of problems. For example, the integration of CNN and RNN to analyze the sequential data of fluorescence lifetime imaging (FLIm). In addition, the synergistic combination of CNN and GAN to construct CycleGAN enables the conversion of images to forms in different patterns meanwhile preserving their original visual integrity (please refer to Supplementary Content 2 and Supplementary Fig. 1 for more technical details).

Such a combination of AI technologies brings several positive technical contributions to fluorescence imaging. For a start, since the fluorescent levels of cancerous and normal tissues vary significantly, there exists the need to process and analyze the data and present the contrast to differentiate tumors from normal tissues, such as the process of aforementioned ratiometric thresholding51 and various fluorescence spectra-related parametric data. Also, it is important to note the inherent heterogeneity in fluorescence imaging, which arises from factors such as subtle histopathologic features, variations in lesion sites, differing imaging environments, fluctuating levels of fluorescence intensity, and individual patient differences. The integration of AI can help manage this complexity by identifying patterns, generalizing diverse characteristics, and adapting to a range of scenarios, thereby enhancing the applicability and effectiveness of this imaging technique60. Moreover, current clinically available fluorophores are not able to provide a satisfactory TBR to delineate tumors sensitively and accurately from their surroundings22; and the further reduction in resolution caused by light scattering61, which requires additional enhancement during image processing, and is made possible by AI-related techniques. Finally, AI can quantify and analyze the subtle differences, as well as multimodal medical information, that could evade human perception, thereby enhancing the sensitivity, accuracy, and comprehensiveness of this imaging modality11. The current AI-enhanced cancer fluorescence imaging studies are briefly summarized in Table 3 and are discussed in detail in the coming sections.Table 3 Neural models applied in clinical and preclinical studies of fluorescence imaging

Disease (model)	Model used	Sensitive	Specificity	Accuracy	ROC–AUC	Reference	
SCC subcutaneous xenograft	BPNN	85.50%	92.86%	90.00%	–	62	
Prostate cancer	S3 Spectroscopy and PCA-SVM	PC1 vs. PC2	100%	100%	100%	1	63	
S3 Spectroscopy and NMF-SVM	NC1 vs. NC2	86.7%	100%	100%	1	
NC1 vs. NC3	100%	100%	100%	1	
NC2 vs. NC3	100%	100%	100%	1	
NB subcutaneous xenograft	PCA-KNN	–	–	97.5%	1	64	
Oral and Oropharyngeal Cancer	1-D CNN	79%	76%	–	0.78	66	
SVM	78%	75%	–	0.77	
RF	86%	87%	–	0.88	
colorectal liver metastases	KNN Two-way Classification (cancer vs. Not cancer)	50 s	–	–	–	0.805	67	
100 s	0.881	
200 s	0.888	
300 s	0.912	
400 s	0.920	
KNN Three-way Classification (cancer vs. Benign tumor vs. Healthy control)	50 s	0.812	
100 s	0.899	
200 s	0.907	
300 s	0.910	
400 s	0.931	
colorectal cancer	Ensemble of a gradient-boosting tree model	100%	92%	95%	–	68	
Oral cancer	DT (Base of Tongue)	87%	94%	–	0.94	70	
DT + SVM (Oral Tongue)	91%	83%	–	0.95	
DT + SVM (Palatine Tonsil)	90%	88%	–	0.92	
Oral Cancer	ResNet + FCN (Wavelet-Based Level 3)	93.32%	67.92%	–	–	71	
ResNet + FPN (Wavelet-Based Level 3)	96.87%	70.93%	–	–	
ResNet + FCN (Gabor Filter-Based Level 3)	81.01%	83.28%	–	–	
ResNet + FPN (Gabor Filter-Based Level 3)	79.35%	89.72%	–	–	
Inception + FCN (Wavelet-Based Level 3)	94.26%	73.12%	–	–	
Inception + FPN (Wavelet-Based Level 3)	96.33%	71.58%	–	–	
Inception + FCN (Gabor Filter-Based Level 3)	77.78%	89.41%	–	–	
Inception + FPN (Gabor Filter-Based Level 3)	72.72%	91.49%	–	–	
SCC subcutaneous xenograft	CycleGAN	–	–	–	–	22	
Quantum dot phantom study	MLP	–	–	–	–	61	
Prostate cancer orthotopic xenograft	Hopfield Neural Network	–	–	–	–	74	
DLGG and GBM	FL-CNN	82.2%	93.8%	–	0.945	78	
WL-CNN	80.3%	82.1%		0.873	
Hepatocellular carcinoma	Random Forest	–	–	–	-	79	
Glioma	DLS-DARTS	–	–	–	0.843	80	
MMNAS	–	–	–	0.832	
PyramidNet-SD	–	–	–	0.828	
EfficientNet-B0 (pre-trained)	–	–	–	0.792	
EfficientNet-B0 (no pre-trained)	–	–	–	0.829	
ResNet-18	–	–	–	0.817	
Breast cancer	GoogLeNet	–	–	88.61%	0.9708	5	
BPNN	Collagen	–	–	95.33%	0.9746	
Lipid	–	–	98.67%	0.9871	
Oral Cancer	GAN + ResNet50	–	–	89.2%	–	86	
PDAC mouse model	KNN	–	–	–	–	76	
Prostate cancer cell in vitro study	AdderNet + ResNet	–	–	–	–	87	
Oral mucosal lesion and oral cancer	Joint Neural Network	87.5%	67.6%	77.6%	–	60	
SVM SFS	81.0%	67.3%	74.0%	–	
SVM L1	79.2%	73.3%	76.4%	–	
The sensitivity of the model represents the percentage of true positive cases correctly identified. The specificity of the model represents the percentage of true negative cases correctly identified. The accuracy of the model represents the overall percentage of correct predictions. ROC-AUC area under the receiver operating characteristic curve represents the model’s ability to distinguish between classes. SCC squamous cell carcinoma, NB neuroblastoma, DLGG diffuse lower-grade glioma, GBM glioblastoma multiforme, PDAC pancreatic ductal adenocarcinoma, BPNN backpropagation neural network, PCA principal component analysis, NMF nonnegative matrix factorization, SVM support vector machine, KNN K-nearest neighbors, CNN convolutional neural network, RF Random Forest, DLS-DARTS double-learnable-stem differentiable ARchiTecture search, MMNAS multi-modality neural architecture search, FL-CNN fluorescence imaging of ICG—convolutional neural network, WL-CNN convolutional neural network that takes WL images as input, GAN generative adversarial network, ResNet, residual network, FCN fully convolutional network, FPN feature pyramid network, MLP multilayer perceptron, DT decision tree.

Assisting data processing and classification

When evaluating light-induced tissue auto-fluorescence, identifying optimal spectral features to differentiate tumors from normal tissue is a complex and challenging task that can be resolved by using artificial intelligence (AI). For example, in a study on skin squamous cell auto-fluorescence, researchers trained a backpropagation neural network (BPNN) with principal component analysis (PCA) results derived from fluorescence spectra. This approach achieved a 90% accuracy rate, showcasing the model’s strong classification capability62. Similarly, by analyzing the Stokes shift spectra of normal and cancerous prostate tissues with ML models like PCA, non-negative matrix factorization (NMF), and support vector machines (SVMs), the inherent key molecules (tryptophan, collagen, and NADH) were identified, thereby enabling the “fingerprints analysis” using label-free auto-fluorescence imaging63. Such a strategy can be further extended to fluorophore-based imaging. For example, the signals of neuroblastoma-targeted multispectral fluorescence imaging (850-1450 nm) were produced into spectral image cubes, which were later trained with multiple ML methods, including PCA, linear discriminant analysis (LDA), K-nearest neighbors (KNN) and NN, for pixel-by-pixel classification of tumor and non-tumor tissue. The trained ML model achieved a 97.5% per-pixel classification accuracy, as well as discovered the conserved multispectral differences between tumor and non-tumor among individuals, indicating the feasibility for ML algorithms to utilize the differences and discern tumor64. These results well demonstrate the utility of AI algorithms in handling complex spectral data to discern cancer and also showcase the augmentation of AI to existing imaging techniques.

Recognizing patterns and reducing heterogeneity

Another example of utilizing auto-fluorescence to differentiate cancerous tissue is fluorescence lifetime imaging (FLIm), by analyzing parameters like fluorescence spectra, average lifetime, decay dynamics, and intensity ratio. These parametric data are often massive, complex, and heterogeneous among tumor types, anatomic structures, and patients, hence may require generalization to identify the patterns and offset the heterogeneity65. Under this notion, Marsden et al. employed ML models like SVM, random forests (RFs), and CNNs to classify the data of various FLIm-derived parameters of oral and oropharyngeal cancer and explored the most proper FLIm parameter (or combination set) to overcome the inter-patient heterogeneity. The approach realized the discrimination of healthy and cancerous tissues with an AUC of 0.88, demonstrating the potential to intraoperatively determine margin status, and identify the time-resolved parameters that contribute the most towards discrimination among all 42 FLIm parameters66. Based on this property, researchers have also utilized ICG perfusion videos to investigate the time-series quantification of the fluorescence signal of rectal tumor and employed a KNN algorithm to classify extracted data, with an achieved AUC–ROC over 0.967, as well as utilizing an ensemble of a gradient-boosting tree model for a similar scenario68. These approaches are pending clinical evaluation. A recently registered clinical trial set out to evaluate the efficacy of AI to intraoperatively classify rectal polyps and tumors based on ICG-perfusion videos (NCT05793554), exploiting the different perfusion characteristics of the lesions69.

The heterogeneity derived from tumor anatomic sites can also be resolved by using ML models, for instance, through respectively training anatomy-specific feature-based classification models, including decision tree (DT), SVM, multilayer perception (MLP), CNN, and optimal transport (OT), to classify oral and oropharyngeal cancer FLIm signals and discriminating cancer and healthy tissues. The training dataset was further supplemented with paired hand-crafted FLIm features and histological results. The models for specific tumor sites (tongue, base of tongue, and palatine tonsil) were able to achieve a region-level prediction of cancerous, healthy, and dysplastic tissues, thereby contributing to the intraoperative delineation of tumor boundary70. The problem can also be tackled with another strategy. Chan and colleagues constructed a texture-map-based branch-collaborative network model to classify the auto-fluorescence images of oral cancer and normal tissues. The neural network comprises a cancer detection branch, consisting of ResNet and Inception models; and a segmentation branch with a fully convolutional neural network (FCN) and feature pyramid network (FPN). By generating texture maps from the input image, the problem of insufficient visual characteristics and the high morphological variations of oral cancer fluorescence images were overcome, making it easier to train deep neural networks, achieving 96% sensitivity and 94% specificity71.

Enhancing imaging quality and tumor demarcation

Deep learning approaches can be expedited to overcome the drawbacks of NIR-I and NIR-IIa fluorescence, such as shallow imaging depth, insufficient TBR, and low resolution, enabling the refinement of currently available imaging fluorophores and equipments72. Generative Adversarial Networks (GANs), a type of deep-learning algorithm designed to generate new data matching the distribution of the training examples, are particularly effective for generating enhanced images. Stemming from this premise, Ma et al. first utilized deep learning techniques to enhance the quality of NIR fluorescence imaging, by employing a U-Net architecture as the generator and a PatchGAN structure as the discriminator and trained the constructed CycleGAN model with both NIR-IIa and NIR-IIb images. Their results demonstrated that the trained model could generate NIR-IIb-like images based on NIR-IIa ones, with the original structures preserved and sharpened, image resolution and contrast-enhanced, as well as improved tumor demarcation and lymph node mapping. The trained model was also verified with images from different emission spectrums, cancer models, fluorophores, and conjugated antibodies, with results suggesting compatibility in different imaging scenarios22.

Later, Xiong et al. employed deep learning techniques to enhance the imaging quality of ICG, a fluorophore with improvable optical properties and inadequate performance in the NIR-I spectrum, and achieved results comparable to that of imaging in the NIR-II window, as demonstrated in Fig. 5. A CycleGAN model was trained with NIR-I fluorescent images to generate NIR-II-like images, the discriminator was modified to ensure faithful image-to-image conversions, thereby retaining the structural details while enhancing the imaging resolution. The constructed model significantly improved the contrast of the original NIR-I images, with TBR augmented by 130% and tumor edges sharpened, thus increasing the tumor margin delineation capability73. In addition to ICG, other fluorescence imaging modalities may as well be enhanced by AI approaches. Utilizing an NN model, the imaging resolution of a quantum dot-based NIR fluorescence imaging system was enhanced twofold, realizing an imaging depth exceeding 10 mm61. These works are typical examples of overcoming hardware shortcomings with software approaches, which may be extended to optimizing real-time fluorescence imaging.Fig. 5 ICG fluorescence imaging enhanced by CycleGAN.

a, c ICG-based fluorescence images of a mouse were captured simultaneously from NIR-I (a) and NIR-IIa (c) windows, respectively. b, d The images generated from the images a and c using a CycleGAN, respectively. The generated images show significantly improved spatial resolution and contrast of mice vasculature, as marked with yellow dashes P1–P4. e Depicts a fluorescence image taken from a mouse tumor xenograft model in the NIR-I window. f Shows the generated image derived from e via the neural network. Green and red curves outlined the tumor and background regions to calculate the TBR, with the generated image f yielding significantly increased TBR. Reproduced with permission from Xiong et al.73. Copyright 2023, Wiley.

Deep learning algorithms can also be applied to segment the entire extent of the tumor and its potential satellite lesions based on fluorescence imaging, thereby assisting the precision surgical resection of tumors. Sammouda and colleagues constructed a prostate cancer mice model and developed prostate-specific membrane antigen (PSMA)-targeted fluorophores, accompanied by whole-body NIR fluorescence imaging. The acquired NIR fluorescence images were used to train a modified Hopfield neural network classifier to segment cancer signals. The processing and segmentation results provided by the model exhibited low signal variation among pixels of the cancerous tissue, which smoothened tumor signal and enhanced the contrast with normal tissue by sharpening the tumor edges74, demonstrating impressive potential to assist in intraoperative guidance and decision support.

Integrating multimodal information

Integrating fluorescence imaging with other medical information, such as radiologic imaging, pathological features, and omics data, may generate additional benefits to the decision-making support of AI, as well as achieving a more holistic view of the disease75–77. Applying this concept by integrating NIR-II fluorescence imaging and deep CNNs through training with fluorescent and white light images of surgical samples with various clinical and pathological parameters. The constructed deep CNN model effectively discriminated tumors from non-tumor samples, with an AUC of 0.945, as well as achieved real-time pathological grading (II–III vs. IV) and Ki-67 levels (<10% vs. ≥10%) estimations of glioma samples, demonstrating outstanding ability to provide intraoperative decision-support78. Similarly, in a study correlating hepatic cell carcinoma fluorescence characteristics with gene mutations and various pathological features extracted with the help of an RF algorithm, the rim-like emission feature and signal heterogeneity were found to be associated with tumor differentiation levels, mutations in TP53, CTNNB1, and others, as well as the expression level of PD-1 in inflammatory cells. Though suffering from a limited number of training datasets, this strategy represents a potential means for intraoperative pathological and molecular characterization of the tumor79.

In addition, integrating multi-modal image data to train NNs helps to offset drawbacks such as small-scale, noise, and low-resolution fluorescent images. By employing a double-learnable-stem differentiable architecture search (DLS-DARTS) method, the appropriate existing architectures were used to process different forms of pictures, as well as realize the multimodal low-level feature fusion. After training with white-light, NIR-I, and NIR-II images of human glioma, the model achieved an AUC of 0.843 and an accuracy of 0.634, exhibiting the potential for intraoperative glioma grading80. Likewise, in another study constructing a multi-modal breast cancer diagnostic DL model, the data of both fluorescence imaging and Raman spectroscope were separately processed and used to train different NN models for tumor classification. The discriminant results of both imaging were integrated and analyzed with a partial least squares algorithm, which resulted in outstanding accuracy of tumor prediction81.

These studies well characterize the potential advantages of employing AI methods for fluorescence and multimodal data analysis to provide intraoperative guidance, which is not only capable of overcoming error but also provides additional integrated information that is beyond the perception of humans. Recently, foundation models for medical application, or generalist medical AI, are on the horizon, capable of integrating multiple medical modalities and performing various tasks82, which are bound to extend the utility of fluorescence imaging once they are deployed. Nevertheless, technical advancement is required for fluorescence imaging to identify genetic alterations, which was achieved in an AI-enabled stimulated Raman histology imaging supplemented with genomic data, intraoperatively determining the molecular alterations (IDH mutation, 1p19q co-deletion, and ATRX mutation) of glioma77. Since certain genomic changes could influence tumor biology and determine its response to therapies, the identification of these determinants will inform the personalized formulation of treatment83. Moreover, efforts should be made to amalgamate fluorescence imaging with other imaging modalities to achieve a more complementary and holistic characterization of malignancies, as demonstrated by the results of image-based multimodal fusion models84,85. Additionally, the use of image-based machine learning models is encouraged for further benefits3. One can expect this extension of surgeons’ insights into tumors from multimodal information to significantly improve the treatment and prognosis of patients.

Mitigating hurdles in AI model deployment

One prominent problem among the several technical considerations when developing an AI model for intraoperative application is the limited volumes of training datasets, due to the difficulty of acquiring standardized imaging data, which may lead to overfitting and underfitting. This can also be partly offset by employing deep learning approaches. Fujimoto and colleagues collected fluorescent oral images based on 5-ALA and constructed a diagnostic model by combining ResNet50 and styleGAN2 with differentiable augmentation (DiffAugment). The original dataset was used to train styleGAN2 to yield two additional generated datasets and was later used to train ResNet50 separately. Their results from training with synthetic datasets demonstrated good classification performance of oral cancer images86. Also, the strategy of transfer learning should also be considered when developing AI models for fluorescence imaging by exploiting the potential of models trained with other data modalities, which could significantly alleviate the hindrance of limited fluorescence image dataset76.

Meanwhile, the intraoperative application of AI requires simultaneous responsiveness. The demand for real-time surgical guidance emphasizes the necessity of developing a compact model with high efficiency while maintaining model performance, exemplified by the use of EfficientNet-B0 in the above glioma case, which reached a balance between performance and speed78. Also, integrating models into hardware can achieve simultaneous data analysis and support decision-making in real time. By simplifying the forward propagation process, a quantized, lightweight CNN model was developed with a 37.8% compression while maintaining its performance for the analysis of FLIm parameters and was integrated into hardware to achieve swift FLIM data analysis87. These results emphasized the importance of meticulous designs of models to overcome hurdles that may exist in research and clinical translations.

Future challenges and translation

AI-based computational methods are a satisfactory complement to existing fluorescence imaging techniques, offering a strategy to overcome hardware drawbacks with software. This approach effectively reaches the middle ground between utilizing “mature” and “immature” technologies in clinical practice, thus is conducive to the iteration of technology and the accumulation of practical experience. We here demonstrate several research gaps in the development and translation of AI-enabled fluorescence-guided precision surgery to achieve human-AI collaboration and propose potential solutions to the problem.

Several approaches can be employed to develop and enhance the performance of AI-driven fluorescence imaging, aligning with the demands of precision cancer surgery. Firstly, the credibility of fluorescence imaging requires it to be validated with phase III clinical trials and evaluate its effect on patient survival88. Secondly, the gap between fluorescence imaging-defined lesions and pathologic results (gold standard) still exists, which may be resolved by computational methods that align fluorescence signals to pathologic results and enhance the validity of imaging-based evaluation. Moreover, to construct AI models that can formulate surgical plans, human surgeon expertise is required to form a supervised learning paradigm, or through gathering valid surgical videos for self-supervised or reinforcement learning approaches89,90. This is similar to the development of highly autonomous robotics, where human-performed surgeries are recorded to train robots that can perform surgical subtasks accordingly21,91. In the context of fluorescence-guided surgery, multimodal information, such as surgical cut design, radiologic imaging, histopathologic results, and disease outcomes, even genetic alterations, can be incorporated to “teach” AI to devise optimal surgical plans based on multi-dimensional disease information. This AI-enabled, fluorescence imaging-based surgical planning holds the promise of transferring superior surgical expertise across institutions and even extending to surgical robotics90. Hence, gathering comprehensive, precise, and sufficient clinical data is imperative to bolster the dependability of AI in surgical decision-making. For instance, training predictive models for optimal margin width of liver tumor resection required a huge dataset from multicenter cohorts92, emphasizing the need for abundant and comprehensive training data from real-world clinical records. Establishing a comprehensive cancer fluorescence image database while respecting patient privacy, will undoubtedly boost the development of task-specific AI models. Finally, advanced display technology such as augmented reality (AR) could be integrated into the imaging systems and realize seamless visualization of information93, thereby increasing maneuverability and user inclination (please refer to Supplementary Content 3 for more technical details).

Nevertheless, various challenges and obstacles remain on the path to clinical implementation. One major challenge in developing AI models for fluorescence imaging is the limited accessibility to large-scale, continuously updated, and real-world datasets. Various preclinical studies, as summarized in this article, have verified the feasibility of using AI to enhance fluorescence imaging. Thus, the logical next step is to prepare training datasets for developing clinical-oriented models. To achieve this, regional and international collaborations between surgeons and data scientists are essential to generate such eligible and accessible datasets. Establishing related technical associations or communities to initiate these efforts is crucial. Moreover, like other medical-AI applications, clinical evaluations are necessary to formulate development and implementation guidelines, demonstrating the impact and efficacy in medical practice, as suggested by consensus-based trial guidelines (SPIRIT-AI, CONSORT-AI)94. Additionally, other long-term applicational hurdles, such as ethical and privacy issues, explainability problems of neural networks, legal oversights, etc., still exist in the development of reliable AI-enabled technologies for medical use17,95,96. Thus, ongoing debates and discussions to balance the benefits and risks of medical AI will undoubtedly lead to a deeper understanding of this technology. Ultimately, future applications of AI-enhanced fluorescence imaging should adhere to these consensus-based standards.

In summary, integrating fluorescence imaging with AI offers ideal intraoperative decision-making support and fulfills the requirements of precision cancer surgery. To expedite this AI-enhanced fluorescence guidance, an interdisciplinary collaboration between the medical, chemical, industrial, and computer science communities is requisite, with surgeons playing the captain in navigating technological innovations. We can anticipate this synergistic approach will lead to significant innovations in surgery and enhance outcomes for cancer patients in the foreseeable future, by illuminating tumors and providing intelligent surgical assistance.

Supplementary information

Supplementary Information

Proof of permission for reprint-Figure 3

Proof of permission for reprint-Figure 5

Supplementary information

The online version contains supplementary material available at 10.1038/s41698-024-00699-3.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (82002853), Shanghai Clinical Research Center for Oral Diseases (19MC1910600), Shanghai Municipal Key Clinical Specialty (shslczdzk01601), Shanghai’s Top Priority Research Center (2022ZZ01017), CAMS Innovation Fund for Medical Sciences (CIFMS) (2019-I2M-5-037), Zhejiang Medical and Health Technology Plan (2020KY939).

Author contributions

H.C. and H.X. drafted the paper. B.P. assisted in the design of illustrations and critically reviewed the paper. X.H. and Y.H. critically reviewed the paper. C.Z. and Z.Z. contributed to the conceptualization and critically reviewed the paper. All authors read and approved the final paper.

Data availability

All data is available by contacting the corresponding author, Z.Z. and C.Z.

Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

These authors contributed equally: Han Cheng, Hongtao Xu.
==== Refs
References

1. Teixeira J One hundred years of evolution in surgery: from asepsis to artificial intelligence Surg. Clin. North Am. 2020 100 xv xvi 10.1016/j.suc.2020.01.001 32169192
Teixeira, J. One hundred years of evolution in surgery: from asepsis to artificial intelligence. Surg. Clin. North Am. 100, xv–xvi (2020).32169192 10.1016/j.suc.2020.01.001
2. Ogrinc N Saudemont P Takats Z Salzet M Fournier I Cancer surgery 2.0: guidance by real-time molecular technologies Trends Mol. Med. 2021 27 602 615 10.1016/j.molmed.2021.04.001 33965341
Ogrinc, N., Saudemont, P., Takats, Z., Salzet, M. & Fournier, I. Cancer surgery 2.0: guidance by real-time molecular technologies. Trends Mol. Med. 27, 602–615 (2021).33965341 10.1016/j.molmed.2021.04.001
3. Urken ML Frozen section analysis in head and neck surgical pathology: a narrative review of the past, present, and future of intraoperative pathologic consultation Oral. Oncol. 2023 143 106445 10.1016/j.oraloncology.2023.106445 37285683
Urken, M. L. et al. Frozen section analysis in head and neck surgical pathology: a narrative review of the past, present, and future of intraoperative pathologic consultation. Oral. Oncol. 143, 106445 (2023).37285683 10.1016/j.oraloncology.2023.106445
4. Coutu B Positive margins matter regardless of subsequent resection findings Oral. Oncol. 2022 128 105850 10.1016/j.oraloncology.2022.105850 35436713
Coutu, B. et al. Positive margins matter regardless of subsequent resection findings. Oral. Oncol. 128, 105850 (2022).35436713 10.1016/j.oraloncology.2022.105850
5. Nocon CC Ajmani GS Bhayani MK Association of facility volume with positive margin rate in the surgical treatment of head and neck cancer JAMA Otolaryngol. Head Neck Surg. 2018 144 1090 1097 10.1001/jamaoto.2018.2421 30347018
Nocon, C. C., Ajmani, G. S. & Bhayani, M. K. Association of facility volume with positive margin rate in the surgical treatment of head and neck cancer. JAMA Otolaryngol. Head Neck Surg. 144, 1090–1097 (2018).30347018 10.1001/jamaoto.2018.2421
6. Zhang L Immediate resection of positive margins improves local control in oral tongue cancer Oral. Oncol. 2023 141 106402 10.1016/j.oraloncology.2023.106402 37094500
Zhang, L. et al. Immediate resection of positive margins improves local control in oral tongue cancer. Oral. Oncol. 141, 106402 (2023).37094500 10.1016/j.oraloncology.2023.106402
7. Bundred JR Margin status and survival outcomes after breast cancer conservation surgery: prospectively registered systematic review and meta-analysis Br. Med. J. 2022 378 e070346 10.1136/bmj-2022-070346 36130770
Bundred, J. R. et al. Margin status and survival outcomes after breast cancer conservation surgery: prospectively registered systematic review and meta-analysis. Br. Med. J. 378, e070346 (2022).36130770 10.1136/bmj-2022-070346
8. Chow LQM Head and neck cancer N. Engl. J. Med. 2020 382 60 72 10.1056/NEJMra1715715 31893516
Chow, L. Q. M. Head and neck cancer. N. Engl. J. Med. 382, 60–72 (2020).31893516 10.1056/NEJMra1715715
9. Wyld L Audisio RA Poston GJ The evolution of cancer surgery and future perspectives Nat. Rev. Clin. Oncol. 2015 12 115 124 10.1038/nrclinonc.2014.191 25384943
Wyld, L., Audisio, R. A. & Poston, G. J. The evolution of cancer surgery and future perspectives. Nat. Rev. Clin. Oncol. 12, 115–124 (2015).25384943 10.1038/nrclinonc.2014.191
10. Dell’Oglio P Precision surgery: the role of intra-operative real-time image guidance—outcomes from a multidisciplinary European consensus conference Am. J. Nucl. Med. Mol. Imaging 2022 12 74 80 35535122
Dell’Oglio, P. et al. Precision surgery: the role of intra-operative real-time image guidance—outcomes from a multidisciplinary European consensus conference. Am. J. Nucl. Med. Mol. Imaging 12, 74–80 (2022).35535122
11. de Boer E Optical innovations in surgery Br. J. Surg. 2015 102 e56 e72 10.1002/bjs.9713 25627136
de Boer, E. et al. Optical innovations in surgery. Br. J. Surg. 102, e56–e72 (2015).25627136 10.1002/bjs.9713
12. Koch M Ntziachristos V Advancing surgical vision with fluorescence imaging Annu. Rev. Med. 2016 67 153 164 10.1146/annurev-med-051914-022043 26768238
Koch, M. & Ntziachristos, V. Advancing surgical vision with fluorescence imaging. Annu. Rev. Med. 67, 153–164 (2016).26768238 10.1146/annurev-med-051914-022043
13. Gao RW Determination of tumor margins with surgical specimen mapping using near-infrared fluorescence Cancer Res. 2018 78 5144 5154 10.1158/0008-5472.CAN-18-0878 29967260
Gao, R. W. et al. Determination of tumor margins with surgical specimen mapping using near-infrared fluorescence. Cancer Res. 78, 5144–5154 (2018).29967260 10.1158/0008-5472.CAN-18-0878
14. Li M Zhang J He Q Yan W Zhang L Research progress of fluorescence imaging in intraoperative navigation based on VOSviewer bibliometric analysis Microchem. J. 2024 196 109709 10.1016/j.microc.2023.109709
Li, M., Zhang, J., He, Q., Yan, W. & Zhang, L. Research progress of fluorescence imaging in intraoperative navigation based on VOSviewer bibliometric analysis. Microchem. J. 196, 109709 (2024).10.1016/j.microc.2023.109709
15. Wendler T van Leeuwen FWB Navab N van Oosterom MN How molecular imaging will enable robotic precision surgery: the role of artificial intelligence, augmented reality, and navigation Eur. J. Nucl. Med. Mol. Imaging 2021 48 4201 4224 10.1007/s00259-021-05445-6 34185136
Wendler, T., van Leeuwen, F. W. B., Navab, N. & van Oosterom, M. N. How molecular imaging will enable robotic precision surgery: the role of artificial intelligence, augmented reality, and navigation. Eur. J. Nucl. Med. Mol. Imaging 48, 4201–4224 (2021).34185136 10.1007/s00259-021-05445-6
16. Zhu S Near-infrared-II (NIR-II) bioimaging via off-peak NIR-I fluorescence emission Theranostics 2018 8 4141 4151 10.7150/thno.27995 30128042
Zhu, S. et al. Near-infrared-II (NIR-II) bioimaging via off-peak NIR-I fluorescence emission. Theranostics 8, 4141–4151 (2018).30128042 10.7150/thno.27995
17. Rajpurkar P Chen E Banerjee O Topol EJ AI in health and medicine Nat. Med. 2022 28 31 38 10.1038/s41591-021-01614-0 35058619
Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. AI in health and medicine. Nat. Med. 28, 31–38 (2022).35058619 10.1038/s41591-021-01614-0
18. Niazi MKK Parwani AV Gurcan MN Digital pathology and artificial intelligence Lancet Oncol. 2019 20 e253 e261 10.1016/S1470-2045(19)30154-8 31044723
Niazi, M. K. K., Parwani, A. V. & Gurcan, M. N. Digital pathology and artificial intelligence. Lancet Oncol. 20, e253–e261 (2019).31044723 10.1016/S1470-2045(19)30154-8
19. Hosny A Parmar C Quackenbush J Schwartz LH Aerts HJWL Artificial intelligence in radiology Nat. Rev. Cancer 2018 18 500 510 10.1038/s41568-018-0016-5 29777175
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. J. W. L. Artificial intelligence in radiology. Nat. Rev. Cancer 18, 500–510 (2018).29777175 10.1038/s41568-018-0016-5
20. Zhou X-Y Guo Y Shen M Yang G-Z Application of artificial intelligence in surgery Front. Med. 2020 14 417 430 10.1007/s11684-020-0770-0 32705406
Zhou, X.-Y., Guo, Y., Shen, M. & Yang, G.-Z. Application of artificial intelligence in surgery. Front. Med. 14, 417–430 (2020).32705406 10.1007/s11684-020-0770-0
21. Panesar S Artificial intelligence and the future of surgical robotics Ann. Surg. 2019 270 223 10.1097/SLA.0000000000003262 30907754
Panesar, S. et al. Artificial intelligence and the future of surgical robotics. Ann. Surg. 270, 223 (2019).30907754 10.1097/SLA.0000000000003262
22. Ma Z Wang F Wang W Zhong Y Dai H Deep learning for in vivo near-infrared imaging Proc. Natl Acad. Sci. USA 2021 118 e2021446118 10.1073/pnas.2021446118 33372162
Ma, Z., Wang, F., Wang, W., Zhong, Y. & Dai, H. Deep learning for in vivo near-infrared imaging. Proc. Natl Acad. Sci. USA 118, e2021446118 (2021).33372162 10.1073/pnas.2021446118
23. Mieog JSD Fundamentals and developments in fluorescence-guided cancer surgery Nat. Rev. Clin. Oncol. 2022 19 9 22 10.1038/s41571-021-00548-3 34493858
Mieog, J. S. D. et al. Fundamentals and developments in fluorescence-guided cancer surgery. Nat. Rev. Clin. Oncol. 19, 9–22 (2022).34493858 10.1038/s41571-021-00548-3
24. Pavlova I Williams M El-Naggar A Richards-Kortum R Gillenwater A Understanding the biological basis of autofluorescence imaging for oral cancer detection: high-resolution fluorescence microscopy in viable tissue Clin. Cancer Res. 2008 14 2396 2404 10.1158/1078-0432.CCR-07-1609 18413830
Pavlova, I., Williams, M., El-Naggar, A., Richards-Kortum, R. & Gillenwater, A. Understanding the biological basis of autofluorescence imaging for oral cancer detection: high-resolution fluorescence microscopy in viable tissue. Clin. Cancer Res. 14, 2396–2404 (2008).18413830 10.1158/1078-0432.CCR-07-1609
25. Hong G Antaris AL Dai H Near-infrared fluorophores for biomedical imaging Nat. Biomed. Eng. 2017 1 1 22 10.1038/s41551-016-0010
Hong, G., Antaris, A. L. & Dai, H. Near-infrared fluorophores for biomedical imaging. Nat. Biomed. Eng. 1, 1–22 (2017).10.1038/s41551-016-0010
26. Zhu S Tian R Antaris AL Chen X Dai H Near-infrared-II molecular dyes for cancer imaging and surgery Adv. Mater. 2019 31 1900321 10.1002/adma.201900321
Zhu, S., Tian, R., Antaris, A. L., Chen, X. & Dai, H. Near-infrared-II molecular dyes for cancer imaging and surgery. Adv. Mater. 31, 1900321 (2019).10.1002/adma.201900321
27. Zhu S Repurposing cyanine NIR-I dyes accelerates clinical translation of near-infrared-II (NIR-II) bioimaging Adv. Mater. 2018 30 1802546 10.1002/adma.201802546
Zhu, S. et al. Repurposing cyanine NIR-I dyes accelerates clinical translation of near-infrared-II (NIR-II) bioimaging. Adv. Mater. 30, 1802546 (2018).10.1002/adma.201802546
28. Voskuil FJ Fluorescence-guided imaging for resection margin evaluation in head and neck cancer patients using cetuximab-800CW: a quantitative dose-escalation study Theranostics 2020 10 3994 4005 10.7150/thno.43227 32226534
Voskuil, F. J. et al. Fluorescence-guided imaging for resection margin evaluation in head and neck cancer patients using cetuximab-800CW: a quantitative dose-escalation study. Theranostics 10, 3994–4005 (2020).32226534 10.7150/thno.43227
29. Hardy NP Real-time administration of indocyanine green in combination with computer vision and artificial intelligence for the identification and delineation of colorectal liver metastases Surg. Open Sci. 2023 12 48 54 10.1016/j.sopen.2023.03.004 36936453
Hardy, N. P. et al. Real-time administration of indocyanine green in combination with computer vision and artificial intelligence for the identification and delineation of colorectal liver metastases. Surg. Open Sci. 12, 48–54 (2023).36936453 10.1016/j.sopen.2023.03.004
30. Kim J-H Ku M Yang J Byeon HK Recent developments of ICG-guided sentinel lymph node mapping in oral cancer Diagnostics (Basel) 2021 11 891 10.3390/diagnostics11050891 34067713
Kim, J.-H., Ku, M., Yang, J. & Byeon, H. K. Recent developments of ICG-guided sentinel lymph node mapping in oral cancer. Diagnostics (Basel) 11, 891 (2021).34067713 10.3390/diagnostics11050891
31. Yang R-Q Surgical navigation for malignancies guided by near-infrared-II fluorescence imaging Small Methods 2021 5 2001066 10.1002/smtd.202001066
Yang, R.-Q. et al. Surgical navigation for malignancies guided by near-infrared-II fluorescence imaging. Small Methods 5, 2001066 (2021).10.1002/smtd.202001066
32. Bhandari C What NIR photodynamic activation offers molecular targeted nanomedicines: perspectives into the conundrum of tumor specificity and selectivity Nano Today 2021 36 101052 10.1016/j.nantod.2020.101052 33552231
Bhandari, C. et al. What NIR photodynamic activation offers molecular targeted nanomedicines: perspectives into the conundrum of tumor specificity and selectivity. Nano Today 36, 101052 (2021).33552231 10.1016/j.nantod.2020.101052
33. Zhang RR Beyond the margins: real-time detection of cancer using targeted fluorophores Nat. Rev. Clin. Oncol. 2017 14 347 364 10.1038/nrclinonc.2016.212 28094261
Zhang, R. R. et al. Beyond the margins: real-time detection of cancer using targeted fluorophores. Nat. Rev. Clin. Oncol. 14, 347–364 (2017).28094261 10.1038/nrclinonc.2016.212
34. Zhou Q EGFR-targeted intraoperative fluorescence imaging detects high-grade glioma with panitumumab-IRDye800 in a phase 1 clinical trial Theranostics 2021 11 7130 7143 10.7150/thno.60582 34158840
Zhou, Q. et al. EGFR-targeted intraoperative fluorescence imaging detects high-grade glioma with panitumumab-IRDye800 in a phase 1 clinical trial. Theranostics 11, 7130–7143 (2021).34158840 10.7150/thno.60582
35. Samkoe, K. S. et al. Measuring microdose ABY-029 fluorescence signal in a primary human soft-tissue sarcoma resection. Proc. SPIE Int. Soc. Opt. Eng. 10862, 1086212 (2019).
36. Kossatz S Validation of the use of a fluorescent PARP1 inhibitor for the detection of oral, oropharyngeal and oesophageal epithelial cancers Nat. Biomed. Eng. 2020 4 272 285 10.1038/s41551-020-0526-9 32165735
Kossatz, S. et al. Validation of the use of a fluorescent PARP1 inhibitor for the detection of oral, oropharyngeal and oesophageal epithelial cancers. Nat. Biomed. Eng. 4, 272–285 (2020).32165735 10.1038/s41551-020-0526-9
37. Zhao T A transistor-like pH nanoprobe for tumour detection and image-guided surgery Nat. Biomed. Eng. 2016 1 0006 10.1038/s41551-016-0006 28966871
Zhao, T. et al. A transistor-like pH nanoprobe for tumour detection and image-guided surgery. Nat. Biomed. Eng. 1, 0006 (2016).28966871 10.1038/s41551-016-0006
38. Ho D Wang P Kee T Artificial intelligence in nanomedicine Nanoscale Horiz. 2019 4 365 377 10.1039/C8NH00233A 32254089
Ho, D., Wang, P. & Kee, T. Artificial intelligence in nanomedicine. Nanoscale Horiz. 4, 365–377 (2019).32254089 10.1039/C8NH00233A
39. Poh CF Fluorescence visualization-guided surgery for early-stage oral cancer JAMA Otolaryngol. Head Neck Surg. 2016 142 209 216 10.1001/jamaoto.2015.3211 26769431
Poh, C. F. et al. Fluorescence visualization-guided surgery for early-stage oral cancer. JAMA Otolaryngol. Head Neck Surg. 142, 209–216 (2016).26769431 10.1001/jamaoto.2015.3211
40. van Dam GM Intraoperative tumor-specific fluorescence imaging in ovarian cancer by folate receptor-α targeting: first in-human results Nat. Med. 2011 17 1315 1319 10.1038/nm.2472 21926976
van Dam, G. M. et al. Intraoperative tumor-specific fluorescence imaging in ovarian cancer by folate receptor-α targeting: first in-human results. Nat. Med. 17, 1315–1319 (2011).21926976 10.1038/nm.2472
41. Krishnan G Metastatic and sentinel lymph node mapping using intravenously delivered Panitumumab-IRDye800CW Theranostics 2021 11 7188 7198 10.7150/thno.55389 34158844
Krishnan, G. et al. Metastatic and sentinel lymph node mapping using intravenously delivered Panitumumab-IRDye800CW. Theranostics 11, 7188–7198 (2021).34158844 10.7150/thno.55389
42. Nishio N Optical molecular imaging can differentiate metastatic from benign lymph nodes in head and neck cancer Nat. Commun. 2019 10 5044 10.1038/s41467-019-13076-7 31695030
Nishio, N. et al. Optical molecular imaging can differentiate metastatic from benign lymph nodes in head and neck cancer. Nat. Commun. 10, 5044 (2019).31695030 10.1038/s41467-019-13076-7
43. Lauwerends LJ Real-time fluorescence imaging in intraoperative decision making for cancer surgery Lancet Oncol. 2021 22 e186 e195 10.1016/S1470-2045(20)30600-8 33765422
Lauwerends, L. J. et al. Real-time fluorescence imaging in intraoperative decision making for cancer surgery. Lancet Oncol. 22, e186–e195 (2021).33765422 10.1016/S1470-2045(20)30600-8
44. Rosenthal EL Safety and tumor specificity of cetuximab-IRDye800 for surgical navigation in head and neck cancer Clin. Cancer Res. 2015 21 3658 3666 10.1158/1078-0432.CCR-14-3284 25904751
Rosenthal, E. L. et al. Safety and tumor specificity of cetuximab-IRDye800 for surgical navigation in head and neck cancer. Clin. Cancer Res. 21, 3658–3666 (2015).25904751 10.1158/1078-0432.CCR-14-3284
45. Gao RW Safety of panitumumab-IRDye800CW and cetuximab-IRDye800CW for fluorescence-guided surgical navigation in head and neck cancers Theranostics 2018 8 2488 2495 10.7150/thno.24487 29721094
Gao, R. W. et al. Safety of panitumumab-IRDye800CW and cetuximab-IRDye800CW for fluorescence-guided surgical navigation in head and neck cancers. Theranostics 8, 2488–2495 (2018).29721094 10.7150/thno.24487
46. Krishnan G Fluorescent molecular imaging can improve intraoperative sentinel margin detection in oral squamous cell carcinoma J. Nucl. Med. 2022 63 1162 1168 10.2967/jnumed.121.262235 35027369
Krishnan, G. et al. Fluorescent molecular imaging can improve intraoperative sentinel margin detection in oral squamous cell carcinoma. J. Nucl. Med. 63, 1162–1168 (2022).35027369 10.2967/jnumed.121.262235
47. van Keulen S Rapid, non-invasive fluorescence margin assessment: optical specimen mapping in oral squamous cell carcinoma Oral. Oncol. 2019 88 58 65 10.1016/j.oraloncology.2018.11.012 30616798
van Keulen, S. et al. Rapid, non-invasive fluorescence margin assessment: optical specimen mapping in oral squamous cell carcinoma. Oral. Oncol. 88, 58–65 (2019).30616798 10.1016/j.oraloncology.2018.11.012
48. Moore LS Characterizing the utility and limitations of repurposing an open-field optical imaging device for fluorescence-guided surgery in head and neck cancer patients J. Nucl. Med. 2017 58 246 251 10.2967/jnumed.115.171413 27587708
Moore, L. S. et al. Characterizing the utility and limitations of repurposing an open-field optical imaging device for fluorescence-guided surgery in head and neck cancer patients. J. Nucl. Med. 58, 246–251 (2017).27587708 10.2967/jnumed.115.171413
49. de Wit JG Comparison of narrow band and fluorescence molecular imaging to improve intraoperative tumour margin assessment in oral cancer surgery Oral Oncol. 2022 134 106099 10.1016/j.oraloncology.2022.106099 36096045
de Wit, J. G. et al. Comparison of narrow band and fluorescence molecular imaging to improve intraoperative tumour margin assessment in oral cancer surgery. Oral Oncol. 134, 106099 (2022).36096045 10.1016/j.oraloncology.2022.106099
50. Lu G Rosenthal EL Translating a fluorescent DNA-repair inhibitor Nat. Biomed. Eng. 2020 4 247 249 10.1038/s41551-020-0537-6 32165734
Lu, G. & Rosenthal, E. L. Translating a fluorescent DNA-repair inhibitor. Nat. Biomed. Eng. 4, 247–249 (2020).32165734 10.1038/s41551-020-0537-6
51. Warram JM A ratiometric threshold for determining presence of cancer during fluorescence-guided surgery J. Surg. Oncol. 2015 112 2 8 10.1002/jso.23946 26074273
Warram, J. M. et al. A ratiometric threshold for determining presence of cancer during fluorescence-guided surgery. J. Surg. Oncol. 112, 2–8 (2015).26074273 10.1002/jso.23946
52. de Wit JG EGFR-targeted fluorescence molecular imaging for intraoperative margin assessment in oral cancer patients: a phase II trial Nat. Commun. 2023 14 4952 10.1038/s41467-023-40324-8 37587149
de Wit, J. G. et al. EGFR-targeted fluorescence molecular imaging for intraoperative margin assessment in oral cancer patients: a phase II trial. Nat. Commun. 14, 4952 (2023).37587149 10.1038/s41467-023-40324-8
53. Adir O Integrating artificial intelligence and nanotechnology for precision cancer medicine Adv. Mater. 2020 32 1901989 10.1002/adma.201901989
Adir, O. et al. Integrating artificial intelligence and nanotechnology for precision cancer medicine. Adv. Mater. 32, 1901989 (2020).10.1002/adma.201901989
54. Xu M Chen Z Zheng J Zhao Q Yuan Z Artificial intelligence-aided optical imaging for cancer theranostics Semin. Cancer Biol. 2023 94 62 80 10.1016/j.semcancer.2023.06.003 37302519
Xu, M., Chen, Z., Zheng, J., Zhao, Q. & Yuan, Z. Artificial intelligence-aided optical imaging for cancer theranostics. Semin. Cancer Biol. 94, 62–80 (2023).37302519 10.1016/j.semcancer.2023.06.003
55. Vasey B Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI Nat. Med. 2022 28 924 933 10.1038/s41591-022-01772-9 35585198
Vasey, B. et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat. Med. 28, 924–933 (2022).35585198 10.1038/s41591-022-01772-9
56. Heidari AE The use of optical coherence tomography and convolutional neural networks to distinguish normal and abnormal oral mucosa J. Biophotonics 2020 13 e201900221 10.1002/jbio.201900221 31710775
Heidari, A. E. et al. The use of optical coherence tomography and convolutional neural networks to distinguish normal and abnormal oral mucosa. J. Biophotonics 13, e201900221 (2020).31710775 10.1002/jbio.201900221
57. Sarkar, P. & Saha, A. Review of artificial intelligence methods for detecting cancer in medical image processing. In 2023 Third International Conference on Artificial Intelligence and Smart Energy (ICAIS) 636–640. 10.1109/ICAIS56108.2023.10073817. (2023).
58. Shmatko A Ghaffari Laleh N Gerstung M Kather JN Artificial intelligence in histopathology: enhancing cancer research and clinical oncology Nat. Cancer 2022 3 1026 1038 10.1038/s43018-022-00436-4 36138135
Shmatko, A., Ghaffari Laleh, N., Gerstung, M. & Kather, J. N. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nat. Cancer 3, 1026–1038 (2022).36138135 10.1038/s43018-022-00436-4
59. Ward TM Computer vision in surgery Surgery 2021 169 1253 1256 10.1016/j.surg.2020.10.039 33272610
Ward, T. M. et al. Computer vision in surgery. Surgery 169, 1253–1256 (2021).33272610 10.1016/j.surg.2020.10.039
60. Caughlin, K. et al. End-to-end neural network for feature extraction and cancer diagnosis of in vivo fluorescence lifetime images of oral lesions. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) 3894–3897. 10.1109/EMBC46164.2021.9629739. (2021).
61. Darwan D Deep fluorescence imaging by laser-scanning excitation and artificial neural network processing Adv. Optical Mater. 2020 8 2000390 10.1002/adom.202000390
Darwan, D. et al. Deep fluorescence imaging by laser-scanning excitation and artificial neural network processing. Adv. Optical Mater. 8, 2000390 (2020).10.1002/adom.202000390
62. Nogueira, J. M. et al. Backpropagation neural network for analysis and classification of fluorescence spectroscopy of squamous cell carcinoma in animal model. In 2021 SBFoton International Optics and Photonics Conference (SBFoton IOPC) 1–4. 10.1109/SBFotonIOPC50774.2021.9461949. (2021).
63. Pu Y Wu B Mo H Alfano RR Stokes shift spectroscopy and machine learning for label-free human prostate cancer detection Opt. Lett. 2023 48 936 939 10.1364/OL.483076 36790979
Pu, Y., Wu, B., Mo, H. & Alfano, R. R. Stokes shift spectroscopy and machine learning for label-free human prostate cancer detection. Opt. Lett. 48, 936–939 (2023).36790979 10.1364/OL.483076
64. Waterhouse DJ Privitera L Anderson J Stoyanov D Giuliani S Enhancing intraoperative tumor delineation with multispectral short-wave infrared fluorescence imaging and machine learning JBO 2023 28 094804 36993142
Waterhouse, D. J., Privitera, L., Anderson, J., Stoyanov, D. & Giuliani, S. Enhancing intraoperative tumor delineation with multispectral short-wave infrared fluorescence imaging and machine learning. JBO 28, 094804 (2023).36993142
65. Vu CQ Arai S Quantitative imaging of genetically encoded fluorescence lifetime biosensors Biosensors 2023 13 939 10.3390/bios13100939 37887132
Vu, C. Q. & Arai, S. Quantitative imaging of genetically encoded fluorescence lifetime biosensors. Biosensors 13, 939 (2023).37887132 10.3390/bios13100939
66. Marsden M Intraoperative margin assessment in oral and oropharyngeal cancer using label-free fluorescence lifetime imaging and machine learning IEEE Trans. Biomed. Eng. 2021 68 857 868 10.1109/TBME.2020.3010480 32746066
Marsden, M. et al. Intraoperative margin assessment in oral and oropharyngeal cancer using label-free fluorescence lifetime imaging and machine learning. IEEE Trans. Biomed. Eng. 68, 857–868 (2021).32746066 10.1109/TBME.2020.3010480
67. Hardy NPP Clinical application of machine learning and computer vision to indocyanine green quantification for dynamic intraoperative tissue characterisation: how to do it Surg. Endosc. 2023 37 6361 6370 10.1007/s00464-023-09963-2 36894810
Hardy, N. P. P. et al. Clinical application of machine learning and computer vision to indocyanine green quantification for dynamic intraoperative tissue characterisation: how to do it. Surg. Endosc. 37, 6361–6370 (2023).36894810 10.1007/s00464-023-09963-2
68. Cahill RA Artificial intelligence indocyanine green (ICG) perfusion for colorectal cancer intra-operative tissue classification Br. J. Surg. 2021 108 5 9 10.1093/bjs/znaa004 33640921
Cahill, R. A. et al. Artificial intelligence indocyanine green (ICG) perfusion for colorectal cancer intra-operative tissue classification. Br. J. Surg. 108, 5–9 (2021).33640921 10.1093/bjs/znaa004
69. Moynihan A CLASSICA: validating artificial intelligence in classifying cancer in real time during surgery Colorectal Dis. 2023 25 2392 2402 10.1111/codi.16769 37932915
Moynihan, A. et al. CLASSICA: validating artificial intelligence in classifying cancer in real time during surgery. Colorectal Dis. 25, 2392–2402 (2023).37932915 10.1111/codi.16769
70. Hassan MA Anatomy-specific classification model using label-free FLIm to aid intraoperative surgical guidance of head and neck cancer IEEE Trans. Biomed. Eng. 2023 70 2863 2873 10.1109/TBME.2023.3266678 37043314
Hassan, M. A. et al. Anatomy-specific classification model using label-free FLIm to aid intraoperative surgical guidance of head and neck cancer. IEEE Trans. Biomed. Eng. 70, 2863–2873 (2023).37043314 10.1109/TBME.2023.3266678
71. Chan C-H Texture-map-based branch-collaborative network for oral cancer detection IEEE Trans. Biomed. Circuits Syst. 2019 13 766 780 10.1109/TBCAS.2019.2918244 31135368
Chan, C.-H. et al. Texture-map-based branch-collaborative network for oral cancer detection. IEEE Trans. Biomed. Circuits Syst. 13, 766–780 (2019).31135368 10.1109/TBCAS.2019.2918244
72. Liu Y Versatile types of inorganic/organic nir-iia/iib fluorophores: from strategic design toward molecular imaging and theranostics Chem. Rev. 2022 122 209 268 10.1021/acs.chemrev.1c00553 34664951
Liu, Y. et al. Versatile types of inorganic/organic nir-iia/iib fluorophores: from strategic design toward molecular imaging and theranostics. Chem. Rev. 122, 209–268 (2022).34664951 10.1021/acs.chemrev.1c00553
73. Xiong X Indocyanine green-based fluorescence imaging improved by deep learning J. Biophotonics 2023 16 e202300066 10.1002/jbio.202300066 37556710
Xiong, X. et al. Indocyanine green-based fluorescence imaging improved by deep learning. J. Biophotonics 16, e202300066 (2023).37556710 10.1002/jbio.202300066
74. Sammouda, R. S., Wang, X. & Basilion, J. P. Hopfield neural network for the segmentation of near infrared fluorescent images for diagnosing prostate cancer. In 2015 6th International Conference on Information and Communication Systems (ICICS) 111–118. 10.1109/IACS.2015.7103212. (2015).
75. Theek B Automation of data analysis in molecular cancer imaging and its potential impact on future clinical practice Methods 2021 188 30 36 10.1016/j.ymeth.2020.06.019 32615232
Theek, B. et al. Automation of data analysis in molecular cancer imaging and its potential impact on future clinical practice. Methods 188, 30–36 (2021).32615232 10.1016/j.ymeth.2020.06.019
76. Race AM Deep learning-based annotation transfer between molecular imaging modalities: an automated workflow for multimodal data integration Anal. Chem. 2021 93 3061 3071 10.1021/acs.analchem.0c02726 33534548
Race, A. M. et al. Deep learning-based annotation transfer between molecular imaging modalities: an automated workflow for multimodal data integration. Anal. Chem. 93, 3061–3071 (2021).33534548 10.1021/acs.analchem.0c02726
77. Hollon T Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging Nat. Med. 2023 29 828 832 10.1038/s41591-023-02252-4 36959422
Hollon, T. et al. Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging. Nat. Med. 29, 828–832 (2023).36959422 10.1038/s41591-023-02252-4
78. Shen B Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks Eur. J. Nucl. Med. Mol. Imaging 2021 48 3482 3492 10.1007/s00259-021-05326-y 33904984
Shen, B. et al. Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks. Eur. J. Nucl. Med. Mol. Imaging 48, 3482–3492 (2021).33904984 10.1007/s00259-021-05326-y
79. Sebagh M Indocyanine green fluorescence patterns of hepatocellular carcinoma correlate with pathological and molecular features HPB 2023 25 198 209 10.1016/j.hpb.2022.10.008 36411232
Sebagh, M. et al. Indocyanine green fluorescence patterns of hepatocellular carcinoma correlate with pathological and molecular features. HPB 25, 198–209 (2023).36411232 10.1016/j.hpb.2022.10.008
80. Xiao A Intraoperative glioma grading using neural architecture search and multi-modal imaging IEEE Trans. Med. Imaging 2022 41 2570 2581 10.1109/TMI.2022.3166129 35404810
Xiao, A. et al. Intraoperative glioma grading using neural architecture search and multi-modal imaging. IEEE Trans. Med. Imaging 41, 2570–2581 (2022).35404810 10.1109/TMI.2022.3166129
81. Shang L-W Fluorescence imaging and Raman spectroscopy applied for the accurate diagnosis of breast cancer with deep learning algorithms Biomed. Opt. Express 2020 11 3673 3683 10.1364/BOE.394772 33014559
Shang, L.-W. et al. Fluorescence imaging and Raman spectroscopy applied for the accurate diagnosis of breast cancer with deep learning algorithms. Biomed. Opt. Express 11, 3673–3683 (2020).33014559 10.1364/BOE.394772
82. Moor M Foundation models for generalist medical artificial intelligence Nature 2023 616 259 265 10.1038/s41586-023-05881-4 37045921
Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).37045921 10.1038/s41586-023-05881-4
83. Galmiche A Genomics and precision surgery for head and neck squamous cell carcinoma Cancer Lett. 2020 481 45 54 10.1016/j.canlet.2020.04.004 32272147
Galmiche, A. et al. Genomics and precision surgery for head and neck squamous cell carcinoma. Cancer Lett. 481, 45–54 (2020).32272147 10.1016/j.canlet.2020.04.004
84. Inoue A Usefulness of an image fusion model using three-dimensional CT and MRI with indocyanine green fluorescence endoscopy as a multimodal assistant system in endoscopic transsphenoidal surgery Int. J. Endocrinol. 2015 2015 1 10 10.1155/2015/694273
Inoue, A. et al. Usefulness of an image fusion model using three-dimensional CT and MRI with indocyanine green fluorescence endoscopy as a multimodal assistant system in endoscopic transsphenoidal surgery. Int. J. Endocrinol. 2015, 1–10 (2015).10.1155/2015/694273
85. Zhang P Real-time navigation for laparoscopic hepatectomy using image fusion of preoperative 3D surgical plan and intraoperative indocyanine green fluorescence imaging Surg. Endosc. 2020 34 3449 3459 10.1007/s00464-019-07121-1 31705286
Zhang, P. et al. Real-time navigation for laparoscopic hepatectomy using image fusion of preoperative 3D surgical plan and intraoperative indocyanine green fluorescence imaging. Surg. Endosc. 34, 3449–3459 (2020).31705286 10.1007/s00464-019-07121-1
86. Fujimoto, T., Fukuzawa, E., Tatehara, S., Satomura, K. & Ohya, J. Automatic diagnosis of early-stage oral cancer and precancerous lesions from ALA-PDD images using GAN and CNN. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) 2161–2164. 10.1109/EMBC48229.2022.9871868. (2022).
87. Zang, Z. et al. Hardware inspired neural network for efficient time-resolved biomedical imaging. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) 1883–1886. 10.1109/EMBC48229.2022.9871214. (2022).
88. Refaat A In vivo fluorescence imaging: success in preclinical imaging paves the way for clinical applications J. Nanobiotechnol. 2022 20 450 10.1186/s12951-022-01648-7
Refaat, A. et al. In vivo fluorescence imaging: success in preclinical imaging paves the way for clinical applications. J. Nanobiotechnol. 20, 450 (2022).10.1186/s12951-022-01648-7
89. Ramesh S Dissecting self-supervised learning methods for surgical computer vision Med. Image Anal. 2023 88 102844 10.1016/j.media.2023.102844 37270898
Ramesh, S. et al. Dissecting self-supervised learning methods for surgical computer vision. Med. Image Anal. 88, 102844 (2023).37270898 10.1016/j.media.2023.102844
90. Yip M Artificial intelligence meets medical robotics Science 2023 381 141 146 10.1126/science.adj3312 37440630
Yip, M. et al. Artificial intelligence meets medical robotics. Science 381, 141–146 (2023).37440630 10.1126/science.adj3312
91. Murali, A. et al. Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D orthotropic tissue phantoms. In 2015 IEEE International Conference on Robotics and Automation (ICRA) 1202–1209. 10.1109/ICRA.2015.7139344. (2015).
92. Bertsimas D Using artificial intelligence to find the optimal margin width in hepatectomy for colorectal cancer liver metastases JAMA Surg. 2022 157 e221819 10.1001/jamasurg.2022.1819 35648428
Bertsimas, D. et al. Using artificial intelligence to find the optimal margin width in hepatectomy for colorectal cancer liver metastases. JAMA Surg. 157, e221819 (2022).35648428 10.1001/jamasurg.2022.1819
93. McKnight RR Virtual reality and augmented reality-translating surgical training into surgical technique Curr. Rev. Musculoskelet. Med 2020 13 663 674 10.1007/s12178-020-09667-3 32779019
McKnight, R. R. et al. Virtual reality and augmented reality-translating surgical training into surgical technique. Curr. Rev. Musculoskelet. Med 13, 663–674 (2020).32779019 10.1007/s12178-020-09667-3
94. Cruz Rivera S Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension Nat. Med 2020 26 1351 1363 10.1038/s41591-020-1037-7 32908284
Cruz Rivera, S. et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat. Med 26, 1351–1363 (2020).32908284 10.1038/s41591-020-1037-7
95. Yu K-H Beam AL Kohane IS Artificial intelligence in healthcare Nat. Biomed. Eng. 2018 2 719 731 10.1038/s41551-018-0305-z 31015651
Yu, K.-H., Beam, A. L. & Kohane, I. S. Artificial intelligence in healthcare. Nat. Biomed. Eng. 2, 719–731 (2018).31015651 10.1038/s41551-018-0305-z
96. Aung YYM Wong DCS Ting DSW The promise of artificial intelligence: a review of the opportunities and challenges of artificial intelligence in healthcare Br. Med Bull. 2021 139 4 15 10.1093/bmb/ldab016 34405854
Aung, Y. Y. M., Wong, D. C. S. & Ting, D. S. W. The promise of artificial intelligence: a review of the opportunities and challenges of artificial intelligence in healthcare. Br. Med Bull. 139, 4–15 (2021).34405854 10.1093/bmb/ldab016
97. Moore LS Effects of an unlabeled loading dose on tumor-specific uptake of a fluorescently labeled antibody for optical surgical navigation Mol. Imaging Biol. 2017 19 610 616 10.1007/s11307-016-1022-1 27830425
Moore, L. S. et al. Effects of an unlabeled loading dose on tumor-specific uptake of a fluorescently labeled antibody for optical surgical navigation. Mol. Imaging Biol. 19, 610–616 (2017).27830425 10.1007/s11307-016-1022-1
98. van Keulen S The clinical application of fluorescence-guided surgery in head and neck cancer J. Nucl. Med 2019 60 758 763 10.2967/jnumed.118.222810 30733319
van Keulen, S. et al. The clinical application of fluorescence-guided surgery in head and neck cancer. J. Nucl. Med 60, 758–763 (2019).30733319 10.2967/jnumed.118.222810
99. Voskuil FJ Exploiting metabolic acidosis in solid cancers using a tumor-agnostic pH-activatable nanoprobe for fluorescence-guided surgery Nat. Commun. 2020 11 3257 10.1038/s41467-020-16814-4 32591522
Voskuil, F. J. et al. Exploiting metabolic acidosis in solid cancers using a tumor-agnostic pH-activatable nanoprobe for fluorescence-guided surgery. Nat. Commun. 11, 3257 (2020).32591522 10.1038/s41467-020-16814-4
100. Steinkamp PJ A standardized framework for fluorescence-guided margin assessment for head and neck cancer using a tumor acidosis sensitive optical imaging agent Mol. Imaging Biol. 2021 23 809 817 10.1007/s11307-021-01614-z 34031845
Steinkamp, P. J. et al. A standardized framework for fluorescence-guided margin assessment for head and neck cancer using a tumor acidosis sensitive optical imaging agent. Mol. Imaging Biol. 23, 809–817 (2021).34031845 10.1007/s11307-021-01614-z
101. Wagner P Detection of peritoneal metastases during cytoreductive surgery using pegsitacianine, a pH sensitive imaging agent: Final results from a phase 2 study JCO 2023 41 3003 3003 10.1200/JCO.2023.41.16_suppl.3003
Wagner, P. et al. Detection of peritoneal metastases during cytoreductive surgery using pegsitacianine, a pH sensitive imaging agent: Final results from a phase 2 study. JCO 41, 3003–3003 (2023).10.1200/JCO.2023.41.16_suppl.3003
102. de Souza França PD A phase I study of a PARP1-targeted topical fluorophore for the detection of oral cancer Eur. J. Nucl. Med. Mol. Imaging 2021 48 3618 3630 10.1007/s00259-021-05372-6 33954826
de Souza França, P. D. et al. A phase I study of a PARP1-targeted topical fluorophore for the detection of oral cancer. Eur. J. Nucl. Med. Mol. Imaging 48, 3618–3630 (2021).33954826 10.1007/s00259-021-05372-6
103. Pan J Real-time surveillance of surgical margins via ICG-based near-infrared fluorescence imaging in patients with OSCC World J. Surg. Oncol. 2020 18 96 10.1186/s12957-020-01874-z 32414418
Pan, J. et al. Real-time surveillance of surgical margins via ICG-based near-infrared fluorescence imaging in patients with OSCC. World J. Surg. Oncol. 18, 96 (2020).32414418 10.1186/s12957-020-01874-z
104. Miller SE First-in-human intraoperative near-infrared fluorescence imaging of glioblastoma using cetuximab-IRDye800 J. Neurooncol. 2018 139 135 143 10.1007/s11060-018-2854-0 29623552
Miller, S. E. et al. First-in-human intraoperative near-infrared fluorescence imaging of glioblastoma using cetuximab-IRDye800. J. Neurooncol. 139, 135–143 (2018).29623552 10.1007/s11060-018-2854-0
