
==== Front
Skin Res Technol
Skin Res Technol
10.1111/(ISSN)1600-0846
SRT
Skin Research and Technology
0909-752X
1600-0846
John Wiley and Sons Inc. Hoboken

10.1111/srt.70060
SRT70060
Research Letter
Research Letter
Enhancing Melanoma Diagnosis: Integration of Zero‐Shot and Few‐Shot Learning With Large Language Models
Nagaoka Takashi https://orcid.org/0000-0002-7460-5008
1 nagaoka@waka.kindai.ac.jp

1 Faculty of Biology‐Oriented Science and Technology Kindai University Kinokawa Japan
* Correspondence: Takashi Nagaoka (nagaoka@waka.kindai.ac.jp)

23 9 2024
9 2024
30 9 10.1111/srt.v30.9 e7006028 8 2024
02 9 2024
© 2024 The Author(s). Skin Research and Technology published by John Wiley & Sons Ltd.
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the terms of the http://creativecommons.org/licenses/by-nc-nd/4.0/ License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non‐commercial and no modifications or adaptations are made.

Japan Society for the Promotion of Science 10.13039/501100001691 20K12698 24K15777 source-schema-version-number2.0
cover-dateSeptember 2024
details-of-publishers-convertorConverter:WILEY_ML3GV2_TO_JATSPMC version:6.4.8 mode:remove_FC converted:23.09.2024
Funding: The authors received no specific funding for this work.
==== Body
pmcMelanoma is one of the most aggressive skin cancers, underscoring the critical need for early diagnosis to improve patient outcomes. Despite significant advancements in dermoscopic techniques, variability in lesion presentation continues to challenge diagnostic accuracy. Earlier phases of this research demonstrated that a large language model could be leveraged for feature extraction, achieving a classification accuracy of 80.3% [1]. Building on that foundation, this study integrates zero‐shot and few‐shot learning approaches with ResNet‐50 [2] and Vision Transformer [3] architectures to address the diverse and complex nature of dermoscopic images. The system described here comprises four main components, each contributing to enhanced diagnostic accuracy.

First, we employed a large language model to generate characteristics for melanoma and melanocytic nevi, as shown in Figure 1. The specific text prompt used was: “You are a dermatologist and diagnose two types of lesions by dermoscopy: melanocytic nevi and melanoma. For each lesion, write a list of 20 characteristics that would help you diagnose the lesion by dermoscopy.” Based on this prompt, the model generated 20 features for each lesion type. These features, including criteria such as asymmetry, border irregularity, and color variation, closely mirror the clinical ABCDE guidelines used in melanoma diagnosis [4], thus validating the model's ability to capture critical diagnostic information. These features formed the basis for subsequent classification tasks.

FIGURE 1 Text prompt for generating characteristics of melanocytic nevi and melanoma using a large language model. This figure shows the specific text prompt used to generate a list of 20 characteristics each for melanocytic nevi and melanoma using a large language model. The generated features serve as the foundation for the subsequent classification tasks in the study.

Second, the zero‐shot learning system, depicted in Figure 2, utilized the features generated by the large language model. For this system and the subsequent systems, we utilized the HAM10000 dataset [5], a well‐known collection of dermoscopic images that includes various pigmented skin lesions, such as melanoma and melanocytic nevi. From this dataset, we randomly selected 1000 images of melanoma and 1000 images of melanocytic nevi. All images were resized to a uniform size of 224 × 224 pixels to standardize the input for our models. The HAM10000 dataset is widely recognized for its diversity and quality, providing a robust foundation for training and evaluating machine learning models in dermatology. The zero‐shot learning system employed a pretrained CLIP model, adapted from the implementation developed by Radford et al. [6], applied through both RN50 and ViT‐B32 architectures using a text and image encoder. It calculated 40 feature values, representing the probabilities that each of the 20 melanoma and 20 nonmelanoma characteristics applied to the image. These values were then used to classify dermoscopic images using a support vector machine (SVM), determining whether an image was more likely to represent melanoma or nonmelanoma. This approach allowed the system to achieve high classification accuracy without requiring prelabeled training data specific to the task.

FIGURE 2 Overview of the zero‐shot learning system using a pre‐trained CLIP model. This figure provides an overview of the zero‐shot learning system, where a pre‐trained CLIP model is used to analyze dermoscopic images. The model, which has not been specifically trained on dermoscopic images, calculates 40 feature values representing the likelihood that each characteristic applies to the image. These values are then used by a SVM for classification. SVM, support vector machine.

Third, the few‐shot learning system, shown in Figure 3(a), aimed to further improve the model's performance by leveraging a small amount of labeled data. We implemented a five‐part cross‐validation, using 1600 dermoscopic images for training and 800 for evaluation in each fold. The CLIP model was fine‐tuned on these 1600 images to better capture the nuances of dermoscopic images. The model, again utilizing RN50 and ViT‐B32 architectures, processed the images through both the text and image encoders and calculated a single feature value representing the probability of melanoma. This value was used to enhance the classification process, improving the model's ability to accurately distinguish between melanoma and nonmelanoma images. In the subsequent step, shown in Figure 3(b), the model trained on these 1600 images was applied to the 800 evaluation images to diagnose melanoma or nonmelanoma. Few‐shot learning is particularly advantageous in medical imaging contexts, where obtaining large, annotated datasets can be challenging. This method leverages limited data to achieve results that approach those of fully supervised learning, making it a practical solution for real‐world applications.

FIGURE 3 Application of few‐shot learning for melanoma classification. Here it shows the process of applying few‐shot learning for melanoma classification. (a) The CLIP model is fine‐tuned on 1600 dermoscopic images using five‐part cross‐validation. (b) The fine‐tuned model is then used to classify the 800 evaluation dermoscopic images, determining whether each image represents melanoma or non‐melanoma, leveraging the knowledge gained from the training data.

Fourth, the SVM system integrated the outputs from the zero‐shot and few‐shot learning systems. The SVM system received the feature values generated by both learning approaches and used them to make the final classification decisions. A radial basis function kernel was employed, and class weights were adjusted to address dataset imbalance. The SVM system was trained and tested on the dataset of 2000 images, with performance evaluated using confusion matrices to ensure robustness. By combining the strengths of both zero‐shot and few‐shot learning, the SVM system demonstrates a balanced approach that leverages the generalization capabilities of zero‐shot learning with the specificity of few‐shot learning, resulting in a robust and accurate classification system.

The results demonstrate the effectiveness of our approach. The zero‐shot learning model achieved an accuracy of 81.6% with the RN50 model and 80.1% with the ViT‐B32 model. When few‐shot learning was applied, accuracy improved to 85.7% with RN50 and 85.5% with ViT‐B32. The integration of these approaches using the SVM classifier further enhanced accuracy to 86.4% for RN50 and 85.4% for ViT‐B32. This improvement underscores how the broad generalization capabilities of zero‐shot learning complement the specificity provided by few‐shot learning, resulting in a system that is both robust and clinically applicable.

The features identified through these processes, as presented in Table 1, offer valuable insights into the distinguishing characteristics of melanoma and melanocytic nevi. Notably, the features selected by the model—such as asymmetry, irregular borders, and varied colors—align closely with the clinical ABCDE criteria for melanoma diagnosis, highlighting the model's relevance and reliability in clinical settings. These features align with clinical observations, reinforcing the models' focus on key aspects of lesion analysis, which contributes to the observed high accuracy. The identification of these features also underscores the importance of feature selection in developing AI‐based diagnostic tools, as the choice of features directly impacts model performance and reliability.

TABLE 1 Significant differences in features between melanoma and melanocytic nevi.

Type	Feature	Significance	
Mel	A pigmented skin lesion asymmetrical in shape and structure.	*	
Mel	A pigmented skin lesion with irregular, scalloped, or poorly defined borders.	n.s.	
Mel	A pigmented skin lesion with multiple colors, including black, brown, tan, white, red, or blue.	*	
Mel	A pigmented skin lesion often larger than 6 mm, but can be smaller.	*	
Mel	A pigmented skin lesion changing in size, shape, or color over time.	n.s.	
Mel	A pigmented skin lesion with irregular, uneven shapes.	*	
Mel	A pigmented skin lesion with a rough, scaly, or ulcerated surface.	n.s.	
Mel	A pigmented skin lesion showing an irregular pigment network.	n.s.	
Mel	A pigmented skin lesion with irregularly distributed dots and globules.	*	
Mel	A pigmented skin lesion with varied structure within the lesion.	*	
Mel	A pigmented skin lesion with presence of atypical or irregular blood vessels.	*	
Mel	A pigmented skin lesion with areas of regression or scar‐like tissue.	*	
Mel	A pigmented skin lesion with presence of blue‐white veil indicating deep dermal involvement.	*	
Mel	A pigmented skin lesion with radial streaming or streaks at the periphery.	*	
Mel	A pigmented skin lesion with milky‐red globules and areas indicating inflammation.	*	
Mel	A pigmented skin lesion with irregular blotches of different colors.	*	
Mel	A pigmented skin lesion with small lesions around the main tumor.	*	
Mel	A pigmented skin lesion with nodular areas within the lesion.	*	
Mel	A pigmented skin lesion may show ulceration or bleeding.	*	
Mel	A pigmented skin lesion with rapid increase in size over weeks to months.	*	
Nevi	A pigmented skin lesion typically symmetrical in shape and color.	*	
Nevi	A pigmented skin lesion usually has smooth, well‐defined borders.	*	
Nevi	A pigmented skin lesion with uniform color, often tan, brown, or flesh‐colored.	*	
Nevi	A pigmented skin lesion usually smaller than 6 mm in diameter.	n.s.	
Nevi	A pigmented skin lesion generally stable, not changing significantly over time.	*	
Nevi	A pigmented skin lesion round or oval in shape.	n.s.	
Nevi	A pigmented skin lesion with a smooth surface, sometimes slightly raised.	*	
Nevi	A pigmented skin lesion showing a regular pigment network with even distribution.	*	
Nevi	A pigmented skin lesion may have regular, evenly distributed dots and globules.	n.s.	
Nevi	A pigmented skin lesion with uniform structure throughout.	n.s.	
Nevi	A pigmented skin lesion generally absent or not prominent blood vessels.	*	
Nevi	A pigmented skin lesion may have hair growing from them.	n.s.	
Nevi	A pigmented skin lesion with presence of nests of melanocytes in the dermis.	*	
Nevi	A pigmented skin lesion commonly shows a reticular (net‐like) pattern.	*	
Nevi	A pigmented skin lesion typically has various shades of brown.	*	
Nevi	A pigmented skin lesion usually one uniform color.	*	
Nevi	A pigmented skin lesion can be flat or slightly elevated.	*	
Nevi	A pigmented skin lesion present since early childhood and stable through adulthood.	*	
Nevi	A pigmented skin lesion common on sun‐exposed areas.	*	
Nevi	A pigmented skin lesion may run in families, indicating a genetic predisposition.	n.s.	
Note: This table presents the features of pigmented skin lesions that distinguish melanoma (Mel) from melanocytic nevi (Nevi). Features marked with an asterisk indicate a significant difference between the two types of lesions, while “n.s.” denotes no significant difference.

John Wiley & Sons, Ltd.

In conclusion, this study demonstrates significant improvements in diagnostic accuracy through the integration of zero‐shot and few‐shot learning approaches using RN50 and ViT‐B32 models. These findings enhance the reliability of automated melanoma diagnosis systems and suggest potential for broader applications in other skin lesion diagnostics. However, challenges such as variability in clinical environments and the availability of diverse datasets must be addressed to fully harness these systems' potential in clinical practice. Future research should explore the application of this methodology to other medical imaging tasks and further refine the models to achieve even higher levels of accuracy and reliability in clinical settings.

Ethics Statement

This research was conducted in accordance with the ethical standards set forth by the institutional review board of Kindai University. All procedures followed were in accordance with the ethical standards of the responsible committee for human experimentation and with the Helsinki Declaration of 1975, as revised in 2000.

Conflicts of Interest

The author declares no conflicts of interest.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.
==== Refs
References

1 T. Nagaoka , “Development of an Automated Melanoma Diagnosis System Using a Large‐Scale Language Model,” in Proceedings of the 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Orlando, FL: IEEE, 2024), https://epapers2.org/embc2024‐posters/ESR/paper_details.php?paper_id=8273.
2 K. He , X. Zhang , S. Ren , and J Sun , “Deep Residual Learning for Image Recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Las Vegas, NV: IEEE, 2016), 770–778, https://arxiv.org/abs/1512.03385.
3 A. Dosovitskiy , L. Beyer , A. Kolesnikov , et al., “An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale,” in Proceedings of the Ninth International Conference on Learning Representations (ICLR) (Vienna, Austria: ICLR, 2021), https://arxiv.org/abs/2010.11929.
4 R. J. Friedman , D. S. Rigel , and A. W. Kopf , “Early Detection of Malignant Melanoma: The Role of Physician Examination and Self‐Examination of the Skin,” CA: A Cancer Journal for Clinicians 35 , no. 3 (1985): 130–151.3921200
5 P. Tschandl , C. Rosendahl , and H. Kittler , “The HAM10000 Dataset: A Large Collection of Multi‐Source Dermoscopic Images of Common Pigmented Skin Lesions,” Scientific Data 5 (2018): 180161.30106392
6 A. Radford , J. W. Kim , C. Hallacy , et al., “Learning Transferable Visual Models From Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine Learning (PMLR, 2021), 8748–8763, https://proceedings.mlr.press/v139/radford21a.html.
