
==== Front
Arch Dermatol Res
Arch Dermatol Res
Archives of Dermatological Research
0340-3696
1432-069X
Springer Berlin Heidelberg Berlin/Heidelberg

39240381
3344
10.1007/s00403-024-03344-y
Original Paper
Interobserver agreement on line-field confocal optical coherence tomography image markers in keratinocyte carcinomas and precursor lesions
Jacobsen Kevin kevin.jacobsen@regionh.dk

1
Ortner Vinzent Kevin 1
Wenande Emily 1
Fredman Gabriella 1
Untracht Gavrielle R. 16
Wolswijk Tom 23
Cruts Emmy 23
Mosterd Klara 23
Nielsen Kari 4
Philipsen Peter Alshede 1
Wiegell Stine Regin 15
Haedersdal Merete 15
1 https://ror.org/05bpbnx46 grid.4973.9 0000 0004 0646 7373 Department of Dermatology, Copenhagen University Hospital – Bispebjerg and Frederiksberg, Copenhagen, Denmark
2 https://ror.org/02jz4aj89 grid.5012.6 0000 0001 0481 6099 Department of Dermatology, Maastricht University Medical Center+, Maastricht, The Netherlands
3 https://ror.org/02jz4aj89 grid.5012.6 0000 0001 0481 6099 GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
4 https://ror.org/012a77v79 grid.4514.4 0000 0001 0930 2361 Department of Dermatology, Clinical Sciences Lund, Lund University and Skaane University Hospital, Lund, Sweden
5 https://ror.org/035b05819 grid.5254.6 0000 0001 0674 042X Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
6 https://ror.org/04qtj9h94 grid.5170.3 0000 0001 2181 8870 Department of Health Technology, Technical University of Denmark, Kongens Lyngby, Denmark
6 9 2024
6 9 2024
2024
316 8 60821 6 2024
6 8 2024
16 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by/4.0/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Line-field confocal optical coherence tomography (LC-OCT) is a new technology for skin cancer diagnostics. However, the interobserver agreement (IOA) of known image markers of keratinocyte carcinomas (KC), including basal cell carcinoma (BCC) and squamous cell carcinoma (SCC), as well as precursors, SCC in situ (CIS) and actinic keratosis (AK), remains unexplored. This study determined IOA on the presence or absence of 10 key LC-OCT image markers of KC and precursors, among evaluators new to LC-OCT with different levels of dermatologic imaging experience. Secondly, the frequency and association between reported image markers and lesion types, was determined. Six evaluators blinded to histopathologic diagnoses, assessed 75 LC-OCT images of KC (21 SCC; 21 BCC), CIS (12), and AK (21). For each image, evaluators independently reported the presence or absence of 10 predefined key image markers of KCs and precursors described in an LC-OCT literature review. Evaluators were stratified by experience-level as experienced (3) or novices (3) based on previous OCT and reflectance confocal microscopy usage. IOA was tested for all groups, using Conger’s kappa coefficient (κ). The frequency of reported image marker and their association with lesion-types, were calculated as proportions and odds ratios (OR), respectively. Overall IOA was highest for the image markers lobules (κ = 0.68, 95% confidence interval (CI) 0.57;0.78) and clefting (κ = 0.63, CI 0.52;0.74), typically seen in BCC (94%;OR 143.2 and 158.7, respectively, p < 0.001), followed by severe dysplasia (κ = 0.42, CI 0.31;0.53), observed primarily in CIS (79%;OR 7.1, p < 0.001). The remaining seven image-markers had lower IOA (κ = 0.06–0.32) and were more evenly observed across lesion types. The lowest IOA was noted for a well-defined (κ = 0.07, CI 0;0.15) and interrupted dermal-epidermal junction (DEJ) (κ = 0.06, CI -0.002;0.13). IOA was higher for all image markers among experienced evaluators versus novices. This study shows varying IOA for 10 key image markers of KC and precursors in LC-OCT images among evaluators new to the technology. IOA was highest for the assessments of lobules, clefting, and severe dysplasia while lowest for the assessment of the DEJ integrity.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00403-024-03344-y.

Keywords

Line-field confocal optical coherence tomography
Basal cell carcinoma
Squamous cell carcinoma
Carcinoma in situ
Actinic keratosis
Keratinocyte carcinoma
Interobserver agreement
Copenhagen UniversityOpen access funding provided by Copenhagen University

issue-copyright-statement© Springer-Verlag GmbH Germany, part of Springer Nature 2024
==== Body
pmcIntroduction

Skin cancer burden and the need for non-invasive imaging tools

Skin cancer is the most frequent cancer worldwide [1] and keratinocyte carcinomas (KC), including basal cell carcinoma (BCC) and squamous cell carcinoma (SCC), are the most common subtypes [2, 3]. The past decades have seen a marked increase in rates of KC and its precursors, SCC in situ (CIS) and actinic keratosis (AK) [4]. Consequently, new, less time- and resource-consuming non-invasive methods are needed for skin cancer detection. Such advances are expected to enhance the patient experience, and mitigate healthcare costs, particularly in high-risk populations that often require intensified KC management, including organ transplant recipients (OTR) [5]. Non-invasive imaging techniques, such as optical coherence tomography (OCT) and reflectance confocal microscopy (RCM), are increasingly being implemented in the field of dermato-oncology, due to their ability to visualize skin layers below the skin surface in real-time [6–9]. These techniques are reported to improve the diagnostic accuracy of KC lesion evaluation, especially when used in combination with dermoscopy [7, 10]. Moreover, imaging enables monitoring of KC treatment outcomes [11, 12], and enhances preoperative surgical planning [6, 13], underscoring its broad potential in skin cancer management.

Line-field confocal optical coherence tomography

Line-field confocal optical coherence tomography (LC-OCT) was introduced in 2018 as a non-invasive imaging technique based on the principles of OCT and RCM [14, 15]. As such, LC-OCT combines high image resolution, depth penetration, and imaging speed to generate real-time three-dimensional (3D) multi-planar imaging (cross-section and en-face) of the epidermis and superficial dermis at the cellular level. The modality is further enhanced by a dermoscopy-like camera, enabling simultaneous visualization of the skin surface to target the precise region of interest [16]. Since its introduction, numerous studies have explored the use of LC-OCT to characterize KC and precursor lesions in vivo [17–27].

LC-OCT image markers and clinical implementation

Implementing LC-OCT into the clinic, includes identification of recognizable image markers within lesions, understanding the level of interobserver agreement (IOA) among individuals reporting on these markers, and understanding their utility for diagnosis. In a recent review, we compiled a comprehensive glossary of LC-OCT markers for KCs and precursors, identifying 10 key image markers for KC, CIS, and AK lesions that were integrated with conventional histopathology [28]. The IOA in which these predefined LC-OCT markers are reported by different evaluators without prior LC-OCT experience remains to be tested.

Objectives

The primary objective of this study was to determine IOA on the presence or absence of 10 key LC-OCT image markers of BCC, SCC, CIS, and AK among evaluators new to LC-OCT with different experience-levels in dermatologic imaging. For a more comprehensive understanding of factors that may contribute to varying IOAs, the study additionally assessed evaluators’ confidence and opinion on whether image-artefacts impacted individual image marker evaluations. A secondary objective was to determine the frequency of image markers reported by evaluators for each lesion-type and test the association between image marker and lesion-type.

Methods

Patients, lesion-types, and LC-OCT images

For this study, LC-OCT images of various BCC, SCC, CIS, and AK lesions were required. Included patients were identified and corresponding images registered in the dedicated OTR clinic at Dermatology Department, Bispebjerg Hospital, Copenhagen, Denmark, between September 2022 and June 2023. Images were collected by a single physician with two years of LC-OCT experience (K.J.) at the time of study initiation. All patients signed informed consent and preapproval for the collection of LC-OCT images was obtained by local ethical committee (H-21038387). The study was conducted in accordance with the Declaration of Helsinki.

LC-OCT image acquisition

Imaging was performed using a commercially available, hand-held LC-OCT scanner (deepLive™, DAMAE Medical, Paris, France, 800 nm, axial and lateral resolution 1.3 μm; field of view 1.2 mm x 0.5 mm x 0.5 mm). Data was prepared as high-resolution videos using Open Broadcaster Software, version 30 (OBS®, Corunna, USA) enabling presentation of 3D image-cases to evaluators on a computer screen i.e., front-to-back in the cross-section-view and top-to-bottom in the en-face-view (Figure S1).

Key image markers

Ten predefined key LC-OCT image markers identified in a previously published review and correlated to diagnostically important histopathological markers for KC and precursor lesions [28], were evaluated in the study. The markers included epidermal markers (severe dysplasia (i.e., full-thickness epidermal dysplasia), mild-moderate dysplasia, and tumor budding), dermal-epidermal markers (well-defined dermal-epidermal junction (DEJ) and interrupted DEJ), and dermal markers (broad strands, keratin pearls, clefting, and collagen alterations) and lobules located in the epidermis or dermis (Fig. 1).

Evaluators, experience and training

Six evaluators, all new to LC-OCT but with varying experience in conventional OCT and/or RCM skin cancer imaging, participated in the study. Evaluators originated from three university hospitals: Maastricht University Medical Center + in the Netherlands, Skaane University Hospital in Sweden, and The Bispebjerg Hospital (BBH) in Denmark. Three evaluators who had completed an e-learning on OCT with continuous monitoring of the ability to differentiate between BCC and non-BCC by cumulative sum analysis [29] were categorized as ‘experienced’ imaging device users; the remaining three were categorized as ‘novices’ (Table S1). To ensure that evaluators were familiar with all 10 image markers, a pictorial overview with brief descriptions for each marker, as illustrated in Fig. 1, was sent to each evaluator a week prior to assessments. On the day of the evaluation, an approximately 2 hour in-person session was held to clarify study procedures and review specific image features, examples of healthy skin and image artifacts (e.g., shadows caused by hyperkeratosis). When referring to the extent of disease, multiple grades could not co-exist within the same image and only one grade was given (i.e., ‘severe’ vs. ‘mild-moderate’ dysplasia, or ‘well-defined vs. interrupted DEJ’); rather, the highest disease level was reported.

Fig. 1 Line-field confocal optical coherence tomography images of 10 key markers and their definition. Illustration and description of 10 predefined and previously reported key image markers evaluated in the study [28]. Epidermal markers A-C and H (top image). Dermal-epidermal junction markers D-E. Dermal markers F-J as well as H (bottom image). Images are resized for illustrative purposes. A. Severe dysplasia (i.e., full-thickness epidermal dysplasia), with large, atypical cells (red arrows) in the stratum granulosum layer. B. Mild-moderate dysplasia, with large, atypical cells in the stratum spinosum and basal layer (blue arrows) but not in stratum granulosum layer (yellow arrows). C. Tumor budding, epidermal protrusion into the dermis (blue stars) composed of disarranged atypical cells with an intact dermal-epidermal junction. D. Well-defined dermal-epidermal junction, a clear line separating the epidermis from the dermis (blue arrows). E. Interrupted dermal-epidermal junction, the line separating the epidermis is blurry/not visible (blue arrows). F. Broad strands, dermal compartment (red dashed line) presenting with a core of disarranged atypical cells. G. Keratin pearl, circular bright structure in the dermis with no connection to the epidermis (red circle), surrounded by a layer of atypical cells (blue arrow). H. Lobule, hemispheric or circular structures (blue arrows) with an inner millefeuille pattern that is either connected to the epidermis (top) or unconnected to the epidermis (bottom). I. Clefting, dark space immediately surrounding the lobule (blue arrow). J. Collagen alterations, bright dense structure in the dermis (within green lines)

Image evaluation

Image-cases were presented by a moderator (K.J.). Evaluators assessed each image independently without the possibility of discussion. Evaluators were blinded to the histopathology diagnosis and clinical photos related to the specific lesion. Assessments were conducted in one live meeting (35 cases), and three virtual meetings (40 cases). For each case, both images of the lesion and healthy skin of the same patient were shown. First, a single cross-section image of the healthy skin was shown, whereafter the 3D image of the lesion was displayed. For each case and image marker, evaluators responded to the following three questions: (1) ‘is the marker present?’ (yes/no), (2) ‘are you confident in your decision?’ (yes/no), (3) ‘did artefacts impact your decision?’ (yes/no). Additional written comments other than yes or no (such as ‘not applicable’) were allowed, if evaluators were unable to make a decision about the presence or absence of an image marker. In the final dataset, missing data was omitted and evaluator responses other than yes/no were recoded.

Outcome measures

The primary outcome measure was IOA in the assessment of 10 LC-OCT markers (Fig. 1), expressed as the Conger’s kappa (κ) coefficient (a generalization of Cohen’s kappa for multiple evaluators) [30]. Reported for all evaluators (n = 6) and stratified by experience level (n = 3), κ values were interpreted as poor (≤ 0), slight (0.01–0.2), fair (0.21–0.4), moderate (0.41–0.6), substantial (0.61–0.8) or almost perfect (> 0.81) as previously described in the literature [31]. To identify κ paradoxes where high agreement but low κ values coexist [32, 33], other complementary agreement measures were also calculated, including overall agreement (Po), agreement on the presence (Ppos) or absence (Pneg) of a marker, and reported prevalence on marker presence (f1) or absence (f2) (described in detail in Table S2).

Evaluators’ responses on confidence levels (i.e., observer’ confidence) and influence of image-artefacts (i.e., resistance to image-artefacts) for each marker, were expressed as proportions of yes/no responses, and the responses of experienced (n = 3) and novice users (n = 3) were compared.

Finally, the frequency of reported image markers for each lesion-type for all evaluators (n = 6) were expressed as proportions, and the association between the presence of independent image markers and lesion-type were expressed as odds ratios (OR) with 95% confidence intervals (CI).

Sample size calculation, statistical analyses, and data quality assurance

A sample size calculation for the κ statistics for two evaluators was performed using a validated online tool (https://wnarifin.github.io/ssc/sskappa.html). The minimal accepted κ value was prespecified at 0.4, and the expected κ value to 0.8, with a power of 0.80 and level of significance of 0.05 [34–36]. To estimate the anticipated prevalence rate of each image marker, an experienced LC-OCT user (K.J.), assessed 156 images of KC, CIS, and AK lesion types from a LC-OCT imaging database at BBH. Although the ideal prevalence rate is 50% for a κ statistics sample size calculation [32], the prevalence rate in that dataset was found to be 4-69%. We chose 15% as the prevalence rate for the sample size calculation, resulting in a recommended sample size of 75 images. Included images (n = 75) aimed to ensure a prevalence rate as close as possible to 50% for all image markers, as well as acceptable image quality and a balanced distribution of available lesion-types: SCC (8 cases, 21 images), BCC (7 cases, including 6 images of superficial BCC and 15 images of nodular BCC), CIS (4 cases, 12 images) and AK (7 cases, including 6 images of AK clinical grade I, 9 images of AK clinical grade II, 6 images of AK clinical grade III). Six evaluators were later included in the study, exceeding the recommended minimum of two evaluators.

κ with 95% CIs were calculated using ‘simpleagree’ (https://svanbelle.shinyapps.io/simpleagree/) and was chosen over Fleiss’ kappa since the same set of evaluators evaluated all images [37]. Table S2 shows how Po, Ppos, Pneg, f1, and f2 were calculated. The proportion of observer’ confidence and resistance to image-artefacts responses between experienced and novice groups for each image marker were compared using the McNemar’s exact test. We used a binary logistic regression model to test whether any of the image markers were associated with specific lesion-types, estimating OR with 95% CI. The model consisted of one binary dependent variable (‘image marker’) and two categorical independent variables (‘single lesion-type’ compared to all other lesion-types, and ‘evaluators’). The dataset for the binary logistic regression analysis included the responses on image markers from all six evaluators for each assessment, coded as 1 if the image marker was present and 0 if absent. Evaluators were included as an independent variable to adjust for differences in their responses. We performed the regression for each of the ten image markers for each combination of the four specific lesion-types. The two latter statistical analyses were performed using SPSS, version 29 (IBM Corp, Chicago, USA). P values < 0.05 were considered statistically significant.

Results

Overall interobserver agreement

κ values among the six evaluators in the assessment of 10 different image markers ranged from slight to substantial (0.06–0.68, Fig. 2 and Table S2). The highest κ values were achieved for lobules (0.68, 95% CI 0.57;0.78) and clefting (0.63, CI 0.52;0.74), followed by severe dysplasia (0.42, CI 0.31;0.53). In contrast, the lowest κ values were observed for a well-defined DEJ (0.07, CI 0;0.15) and an interrupted DEJ (0.06, CI -0.002;0.13).

Fig. 2 Interobserver agreement as measured by Conger’s kappa of 10 key LC-OCT image markers of keratinocytic lesions and precursors categorized by level of evaluator experience. Bars with 95% confidence intervals (CI) show Conger’s kappa values for all six evaluators with conventional OCT and RCM background experience (green), of which three were categorized as experienced (blue) and three were categorized as novices in non-invasive imaging (yellow)

Interobserver agreement by experience-level

For all image markers, κ values were higher for experienced evaluators than for novices, most evident for markers related to DEJ integrity (Fig. 2 and Table S2). For both experience groups, the highest κ values were found for lobules (experienced 0.77, CI 0.64;0.90 vs. novices 0.63, CI 0.50;0.77), clefting (experienced 0.74, CI 0.61;0.87 vs. novices 0.57, CI 0.43;0.71), and severe dysplasia (experienced 0.49, CI 0.35;0.64 vs. novices 0.41, CI 0.26;0.55). κ values for a well-defined DEJ and interrupted DEJ were particularly low among novices (-0.11, CI -0.23;0.004 and κ = -0.10, CI -0.23;0.02, respectively).

Observer confidence and resistance to artefacts stratified by experience

Experienced evaluators were more confident compared to novices when evaluating six out of 10 image markers (p < 0.001), while no difference was observed for the remainder: tumor budding (p = 0.33), well-defined DEJ (p = 0.11), interrupted DEJ (p = 0.19) and broad strands (p = 0.12) (Fig. 3A). Consistent with IOA findings, experienced evaluators confidence was highest for severe-dysplasia (81%), clefting (80%), and lobules (79%) and lowest for an interrupted DEJ (52%) and well-defined DEJ (55%).

Fig. 3 Reported observer confidence and resistance to artefacts for 10 key LC-OCT image markers in images of basal cell carcinoma, squamous cell carcinoma, squamous cell carcinoma in situ/carcinoma in situ, and actinic keratosis. (A) Observer confidence (i.e., the proportion of ‘yes confident’ responses) for conventional optical coherence tomography and reflectance confocal microscopy experienced (n = 3) and novice (n = 3) evaluators, respectively. (B) Resistance to image-artefacts (i.e., the proportion of ‘no, image artefacts did not impact decision’ responses) for conventional optical coherence tomography and reflectance confocal microscopy experienced (n = 3) and novice (n = 3) evaluators, respectively

For all image markers, experienced evaluators were less influenced by artefacts compared to novices (p < 0.001) as illustrated in Fig. 3B. Notably, both groups’ assessments of the epidermal markers, severe dysplasia, mild-moderate dysplasia, and tumor budding were reportedly less impacted by artefacts, in contrast to assessments of the dermal markers, broad strands, keratin pearls, clefting, and collagen alterations, and the epidermal/dermal marker lobules. The image markers most often reported to be influenced by artefacts were an interrupted DEJ for the experienced group (resistance to image artefacts, 81.7%) and broad strands for the novice group (66.8%).

Reported image markers by lesion-type

Certain markers appeared to have greater lesion-specificity than others (Fig. 4). The image markers, lobules and clefting, were frequently observed in BCCs (94%, 94%, respectively) and less in images of SCC (17%, 20%), CIS (3%, 5%), and AK (6%, 10%). Moreover, the image marker, severe dysplasia, was frequently reported in CIS (79%) and more rarely in BCC (10%), SCC (44%), and AK (54%). The remaining image markers were more evenly distributed across all lesion types. For example, an interrupted DEJ were reported with similar frequencies for BCC (37%), SCC (51%), CIS (46%), and AK (33%).

Fig. 4 Reported image markers in line-field confocal optical coherence tomography images of keratinocyte carcinomas and precursor lesions. For each image marker, a radar chart is plotted for basal cell carcinoma (BCC), squamous cell carcinoma (SCC), squamous cell carcinoma in situ/carcinoma in situ (CIS), and actinic keratosis (AK). Green shaded areas within radar charts represents the number independent evaluators reported a given image marker in images of BCC, SCC, CIS, and AK, with each of the six corners representing a single evaluator (n = 6). The outermost ring represents 100% with 5 lines extending inwards in increments of 20%. Percentages are the mean response for all six evaluators

Based on evaluator reports, some image markers were also found to be associated with a particular lesion type (Fig. 5). Lobules (OR 143.2, CI 62.9;326.1, p < 0.001), clefting (OR 158.7, CI 64.5; 390.7, p < 0.001) and collagen alterations (OR 12, CI 6.1; 23.3, p < 0.001) were strongly associated with BCC, as was the non-presence of severe dysplasia (OR 0.08, CI 0.04;0.2, p < 0.001) and tumor budding (OR 0.4, CI 0.3;0.7, p < 0.001). Meanwhile, interrupted DEJ (OR 2.7, CI 1.6; 4.4, p < 0.001), keratin pearls (OR 2.5, CI 1.3;4.5, p < 0.004), and broad strands (OR 1.9, CI 1.2; 3.1, p = 0.009), were associated with SCC, in addition to the non-presence of clefting (OR 0.3, CI 0.2;0.6, p < 0.001), lobules (OR 0.3, CI 0.2;0.5, p < 0.001), and well-defined DEJ (OR 0.4, CI 0.2;0.6, p < 0.001).

The presence of severe dysplasia (OR 7.1, CI 3.9; 13.2, p < 0.001) and surprisingly interrupted DEJ (OR 2.5, CI 1.3;4.9, p = 0.005) was associated with CIS, as was the non-presence of clefting (OR 0.08, CI 0.03;0.2, p < 0.001), lobules (OR 0.05, CI 0.01;0.2, p < 0.001), and mild-moderate dysplasia (OR 0.4, CI 0.2;0.8, p = 0.005). Finally, tumor budding (OR 2.3, CI 1.5;3.5, p < 0.001), well-defined DEJ (OR 2.1, CI 1.3;3.3, p = 0.003), and interestingly severe dysplasia (OR 1.9, CI 1.3;2.9, p = 0.002) were associated with AK; in contrast to lobules (OR 0.09, CI 0.04;0.2, p < 0.001), clefting (OR 0.1, CI 0.06;0.2, p < 0.001), collagen alterations (OR 0.4, CI 0.2;0.6, p < 0.001) and interrupted DEJ (OR 0.5, CI 0.3;0.8, p = 0.003) which were inversely associated with AK.

Fig. 5 Odds ratios for 10 key image markers reported by evaluators in line-field confocal optical coherence tomography images of basal cell carcinoma (BCC), squamous cell carcinoma (SCC), squamous cell carcinoma in situ/carcinoma in situ (CIS), and actinic keratosis (AK). The association of reported image markers with lesion types are presented as odds ratios (plotted as black dots on individual graphs) with 95% confidence intervals on a log-scale. The independent variable is a single lesion type compared to other lesion-types, adjusted for differences in evaluators’ responses. Odds ratios where the 95% confidence intervals do not intersect the line (OR = 1) indicate a statistically significant test

Discussion

This study examined IOA among six evaluators new to LC-OCT, based on assessment of 10 key LC-OCT image markers in BCC, SCC, CIS, and AK images. The study found varying IOA, ranging from overall κ values of 0.06 to 0.68 depending on image marker (Fig. 2). The highest IOAs were noted for lobules and clefting (substantial κ) as well as severe dysplasia (moderate κ). In contrast, the remaining seven image markers were observed with slight and fair κ values across all evaluators. Notably, for both experienced and less experienced evaluators, assessments of DEJ integrity had consistently low κ values. The tendency among evaluators to agree on the presence or absence of lobules, clefting, severe dysplasia, may suggest that these image markers are easily and consistently recognized by LC-OCT users irrespective of their past imaging experience, whereas markers related to the DEJ appears to be challenging.

The higher IOA observed for lobules, compared to other image markers in the study, could reflect its distinct appearance as well as factors that enable a strong image-signal, including often superficial skin location and occasional occurrence with an atrophic epidermis. Described by Suppa et al., lobules often exhibit the combination of a central and characteristic millefeuille LC-OCT pattern and prominent peripheral dark clefting surrounded by a bright rim indicating collagen alterations. The combination of these three markers are referred to as triad of colors [23]. Furthermore, lobules often occur in combination with an atrophic epidermis in nodular BCC [25]. This atrophy reduces the distance from the skin surface to the dermis, potentially making the signal and image definition surrounding lobules, stronger. Indeed, previous studies on LC-OCT imaging of skin cancer lesions report a decrease in image definition with increasing depth [23, 38, 39], with cellular markers visible up to 300 μm depth, and architectural markers visible up to 400 μm depth [38]. This could similarly explain why severe dysplasia, with its distinct appearance in the uppermost layer of the epidermis where image signal/cellular details often are most clear (see Fig. 1A), had the third highest κ; despite reported challenges associated with examining single cell atypia using LC-OCT [40].

Lower κ values could be due to certain image markers being inherently difficult to identify or challenges distinguishing between two markers. For example, the thin and subtle line of a well-defined DEJ (Fig. 1D) is not easy to detect, particularly in the presence of hyperkeratosis and acanthosis seen in KC lesions [19, 21, 22, 41]. A discrete break in the DEJ caused by invasive tumor strands, and their presence as broad strands in the dermis, is arguably also challenging to visualize (Fig. 1E and Fig. 1F). Tumor budding can resemble lobules connected to the epidermis, conceivably making it difficult for evaluators to distinguish the two markers (i.e., both image markers are visible as a protrusion of the epidermis (Fig. 1H, top image, vs Fig. 1C). Another reason for lower κ values may be related to differing understandings of image marker definitions, especially since the validity of κ values depend upon an detailed definition of concordant or discordant findings [42]. This study used predefined image markers described in a published review of LC-OCT literature [28], but expert consensus in the LC-OCT community on image marker definitions is lacking. Comparative studies between LC-OCT and histopathology, as is reported for atypical keratinocyte growth patterns in AK [20], may establish a shared understanding of what should, for example, be deemed an interrupted DEJ or a broad strand in LC-OCT images. Nevertheless, it appears that background experience in OCT/RCM imaging can somewhat increase κ values, observer’ confidence, and resistance to image-artefacts for most image markers.

Some κ values were affected by κ paradox [32, 33]. Although we aimed to include images with a 50% prevalence rate of image markers, a large imbalance between evaluator-reported prevalence was observed for mild-moderate dysplasia, broad-strands, and keratin pearls (Table S2). In addition, some markers displayed notable differences between their Ppos and Pneg, meaning evaluators had good agreement on presence but not absence, or vice versa. Therefore, the κ values of mild-moderate dysplasia, broad-strands, and keratin pearls should be interpreted with caution as the estimated κ may be lowered.

Importantly, this study did not assess the diagnostic utility of LC-OCT, but the frequency of image markers for each lesion-type reported by evaluators new to the technology. Therefore, the true prevalence of the image markers within KC and precursor lesions remains to be tested by experts in the field. Nevertheless, in line with previous reports, we found lobules, clefting, and collagen alterations significantly associated with BCC [23–25]; interrupted DEJ, broad strands and keratin pearls with SCC, and severe dysplasia with CIS [21, 22, 43], as well as well-defined DEJ and tumor budding with AK [20]. Unexpected associations, such as an interrupted DEJ in CIS, could be attributed to the smaller sample size of CIS images (n = 12 image cases) as compared to other lesion-types (n = 21 for AK, SCC, and BCC) or the aforementioned challenges related DEJ assessment. Another unexpected finding was the association between severe dysplasia and AK: we speculate that this could be due to the presence of single, large cells in the stratum granulosum, a feature that is infrequently observed in AK lesions that would not tip the histopathology diagnosis towards CIS, a lesion characterized by more extensive full-thickness epidermal dysplasia in traditional histopathology. While the reference standard was histopathology, it is also possible that an area of CIS was seen in the LC-OCT image but not captured in the biopsy specimen.

This study has some limitations. We used the poor-almost perfect scale for κ interpretation [31], although other approaches are available [44]. Our study was further based on a limited selection of KC and precursors images, which limits the generalizability of our results. Future studies should include an extensive database to ensure sufficient prevalence of rare image markers. Additionally, we did not take the impact of in-person versus virtual image-evaluation into account, although this aspect may be important to image quality. Study strengths, on the other hand, are related to the study’s real-world applicability. We assessed six evaluators from three different centers with varying degrees of imaging experience, a selection of evaluators that in many ways is representative of the current, fledgling LC-OCT field.

Conclusion and perspectives

In conclusion, our study highlights variations in IOA between evaluators without prior experience with LC-OCT in assessment of 10 key LC-OCT image markers in SCC, BCC, CIS, and AK lesions. For lobules and clefting, as well as, severe dysplasia, evaluators achieved higher IOA, while lowest IOA was shown for assessments of DEJ integrity. The study thus provides new insights into imaging-based lesion assessment in dermato-oncology and specifically, shows the degree of IOA in LC-OCT image marker assessment amongst new users of this increasingly implemented technology.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1

Acknowledgements

The study was executed as a part of the Danish Research Center for Skin Cancer (www.researchcenterforskincancer.org), a public-private research partnership between the Private Hospital Molholm, Aalborg University Hospital and Copenhagen University Hospital, Bispebjerg and Frederiksberg, as well as the Skin Cancer Innovation Clinical Academic Group (SCIN CAG), Greater Copenhagen Health Science Partners (GCHSP).

Author contributions

Study conception and design: KJ, VKO, PAP, MH. Acquisition of data: KJ, GU, GF, EC, TW, KM, KN, SW, MH. Analysis and interpretation of data: KJ, VKO, EW, PAP, MH. Drafting of manuscript: KJ. Critical revision: All authors.

Funding

Open access funding provided by Copenhagen University

Data availability

No datasets were generated or analysed during the current study.

Declarations

Competing interests

The authors declare no competing interests.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Perez M Abisaad JA Rojas KD Marchetti MA Jaimes N Skin cancer: primary, secondary, and tertiary prevention. Part I J Am Acad Dermatol 2022 87 2 255 268 10.1016/j.jaad.2021.12.066 35176397
Perez M, Abisaad JA, Rojas KD, Marchetti MA, Jaimes N (2022) Skin cancer: primary, secondary, and tertiary prevention. Part I. J Am Acad Dermatol 87(2):255–26835176397 10.1016/j.jaad.2021.12.066
2. Miller DL, Weinstock MA (1994) Nonmelanoma skin cancer in the United States: Incidence. J Am Acad Dermatol. ;30(5, Part 1):774–8
3. Pandeya N Olsen CM Whiteman DC The incidence and multiplicity rates of keratinocyte cancers in Australia Med J Aust 2017 207 8 339 343 10.5694/mja17.00284 29020905
Pandeya N, Olsen CM, Whiteman DC (2017) The incidence and multiplicity rates of keratinocyte cancers in Australia. Med J Aust 207(8):339–34329020905 10.5694/mja17.00284
4. Lomas A Leonardi-Bee J Bath-Hextall F A systematic review of worldwide incidence of nonmelanoma skin cancer Br J Dermatol 2012 166 5 1069 1080 10.1111/j.1365-2133.2012.10830.x 22251204
Lomas A, Leonardi-Bee J, Bath-Hextall F (2012) A systematic review of worldwide incidence of nonmelanoma skin cancer. Br J Dermatol 166(5):1069–108022251204 10.1111/j.1365-2133.2012.10830.x
5. Wenande E Togsverd-Bo K Hastrup A Lei U Philipsen PA Haedersdal M Skin Cancer Development is strongly Associated with Actinic Keratosis in Solid Organ Transplant recipients: a Danish cohort study Dermatology 2023 239 3 393 402 10.1159/000529369 36731451
Wenande E, Togsverd-Bo K, Hastrup A, Lei U, Philipsen PA, Haedersdal M (2023) Skin Cancer Development is strongly Associated with Actinic Keratosis in Solid Organ Transplant recipients: a Danish cohort study. Dermatology 239(3):393–40236731451 10.1159/000529369
6. Pomerantz R Zell D McKenzie G Siegel DM Optical coherence tomography used as a modality to delineate basal cell Carcinoma prior to Mohs Micrographic surgery Case Rep Dermatol 2011 3 3 212 218 10.1159/000333000 22110434
Pomerantz R, Zell D, McKenzie G, Siegel DM (2011) Optical coherence tomography used as a modality to delineate basal cell Carcinoma prior to Mohs Micrographic surgery. Case Rep Dermatol 3(3):212–21822110434 10.1159/000333000
7. Forsea AM Carstea EM Ghervase L Giurcaneanu C Pavelescu G Clinical application of optical coherence tomography for the imaging of non-melanocytic cutaneous tumors: a pilot multi-modal study J Med Life 2010 3 4 381 389 21254735
Forsea AM, Carstea EM, Ghervase L, Giurcaneanu C, Pavelescu G (2010) Clinical application of optical coherence tomography for the imaging of non-melanocytic cutaneous tumors: a pilot multi-modal study. J Med Life 3(4):381–38921254735
8. Rajadhyaksha M González S Zavislan JM Rox Anderson R Webb RH In vivo confocal scanning laser Microscopy of Human skin II: advances in Instrumentation and Comparison with Histology11The authors have declared conflict of interest J Invest Dermatol 1999 113 3 293 303 10.1046/j.1523-1747.1999.00690.x 10469324
Rajadhyaksha M, González S, Zavislan JM, Rox Anderson R, Webb RH (1999) In vivo confocal scanning laser Microscopy of Human skin II: advances in Instrumentation and Comparison with Histology11The authors have declared conflict of interest. J Invest Dermatol 113(3):293–30310469324 10.1046/j.1523-1747.1999.00690.x
9. Huang D, Swanson EA, Lin CP, Schuman JS, Stinson WG, Chang W et al (2015) Opt Coherence Tomography. ;12
10. Borsari S Pampena R Lallas A Kyrgidis A Moscarella E Benati E Clinical indications for Use of Reflectance Confocal Microscopy for skin Cancer diagnosis JAMA Dermatol 2016 152 10 1093 1098 10.1001/jamadermatol.2016.1188 27580185
Borsari S, Pampena R, Lallas A, Kyrgidis A, Moscarella E, Benati E et al (2016) Clinical indications for Use of Reflectance Confocal Microscopy for skin Cancer diagnosis. JAMA Dermatol 152(10):1093–109827580185 10.1001/jamadermatol.2016.1188
11. Banzhaf CA Themstrup L Ring HC Mogensen M Jemec GBE Optical coherence tomography imaging of non-melanoma skin cancer undergoing imiquimod therapy Skin Res Technol 2014 20 2 170 176 10.1111/srt.12102 24103017
Banzhaf CA, Themstrup L, Ring HC, Mogensen M, Jemec GBE (2014) Optical coherence tomography imaging of non-melanoma skin cancer undergoing imiquimod therapy. Skin Res Technol 20(2):170–17624103017 10.1111/srt.12102
12. Ahlgrimm-Siess V Horn M Koller S Ludwig R Gerger A Hofmann-Wellenhof R Monitoring efficacy of cryotherapy for superficial basal cell carcinomas with in vivo reflectance confocal microscopy: a preliminary study J Dermatol Sci 2009 53 1 60 64 10.1016/j.jdermsci.2008.08.005 18829267
Ahlgrimm-Siess V, Horn M, Koller S, Ludwig R, Gerger A, Hofmann-Wellenhof R (2009) Monitoring efficacy of cryotherapy for superficial basal cell carcinomas with in vivo reflectance confocal microscopy: a preliminary study. J Dermatol Sci 53(1):60–6418829267 10.1016/j.jdermsci.2008.08.005
13. Tannous Z Torres A González S In vivo real-time confocal reflectance Microscopy: a Noninvasive Guide for Mohs Micrographic surgery facilitated by Aluminum Chloride, an excellent contrast enhancer Dermatol Surg 2003 29 8 839 846 12859385
Tannous Z, Torres A, González S (2003) In vivo real-time confocal reflectance Microscopy: a Noninvasive Guide for Mohs Micrographic surgery facilitated by Aluminum Chloride, an excellent contrast enhancer. Dermatol Surg 29(8):839–84612859385
14. Dubois A Levecq O Azimani H Siret D Barut A Suppa M Line-field confocal optical coherence tomography for high-resolution noninvasive imaging of skin tumors J Biomed Opt 2018 23 10 106007 106007 10.1117/1.JBO.23.10.106007
Dubois A, Levecq O, Azimani H, Siret D, Barut A, Suppa M et al (2018) Line-field confocal optical coherence tomography for high-resolution noninvasive imaging of skin tumors. J Biomed Opt 23(10):106007–10600710.1117/1.JBO.23.10.106007
15. Latriglia F Ogien J Tavernier C Fischman S Suppa M Perrot JL Line-field confocal optical coherence tomography (LC-OCT) for skin imaging in Dermatology Life 2023 13 12 2268 10.3390/life13122268 38137869
Latriglia F, Ogien J, Tavernier C, Fischman S, Suppa M, Perrot JL et al (2023) Line-field confocal optical coherence tomography (LC-OCT) for skin imaging in Dermatology. Life 13(12):226838137869 10.3390/life13122268
16. Ogien J Tavernier C Fischman S Dubois A Line-field confocal optical coherence tomography (LC-OCT): principles and practical use Ital J Dermatol Venereol 2023 158 3 171 179 10.23736/S2784-8671.23.07613-2
Ogien J, Tavernier C, Fischman S, Dubois A (2023) Line-field confocal optical coherence tomography (LC-OCT): principles and practical use. Ital J Dermatol Venereol 158(3):171–17910.23736/S2784-8671.23.07613-2
17. Cinotti E Brunetti T Cartocci A Tognetti L Suppa M Malvehy J Diagnostic accuracy of line-field Confocal Optical Coherence Tomography for the diagnosis of skin carcinomas Diagnostics 2023 13 3 361 10.3390/diagnostics13030361 36766466
Cinotti E, Brunetti T, Cartocci A, Tognetti L, Suppa M, Malvehy J et al (2023) Diagnostic accuracy of line-field Confocal Optical Coherence Tomography for the diagnosis of skin carcinomas. Diagnostics 13(3):36136766466 10.3390/diagnostics13030361
18. Cinotti E Bertello M Dragotto M Cartocci A Tognetti L Cappilli S Comparison of reflectance confocal microscopy and line-field optical coherence tomography for the identification of basal cell carcinoma J Eur Acad Dermatol Venereol 2023 37 9 e1147 e1150 10.1111/jdv.19155 37165928
Cinotti E, Bertello M, Dragotto M, Cartocci A, Tognetti L, Cappilli S et al (2023) Comparison of reflectance confocal microscopy and line-field optical coherence tomography for the identification of basal cell carcinoma. J Eur Acad Dermatol Venereol 37(9):e1147–e115037165928 10.1111/jdv.19155
19. Razi S Khan S Truong TM Zia S Khan FF Uddin KM Cutaneous squamous cell carcinoma: an Up-to-date Comprehensive Review with a Focus on Contemporary Optical Imaging Diagnostic Modalities Dermato 2023 3 2 161 181 10.3390/dermato3020013
Razi S, Khan S, Truong TM, Zia S, Khan FF, Uddin KM et al (2023) Cutaneous squamous cell carcinoma: an Up-to-date Comprehensive Review with a Focus on Contemporary Optical Imaging Diagnostic Modalities. Dermato 3(2):161–18110.3390/dermato3020013
20. Ruini C Schuh S Gust C Hartmann D French LE Sattler EC In-Vivo LC-OCT evaluation of the Downward Proliferation Pattern of keratinocytes in Actinic Keratosis in comparison with histology: first impressions from a pilot study Cancers 2021 13 12 2856 10.3390/cancers13122856 34201052
Ruini C, Schuh S, Gust C, Hartmann D, French LE, Sattler EC et al (2021) In-Vivo LC-OCT evaluation of the Downward Proliferation Pattern of keratinocytes in Actinic Keratosis in comparison with histology: first impressions from a pilot study. Cancers 13(12):285634201052 10.3390/cancers13122856
21. Ruini C Schuh S Gust C Kendziora B Frommherz L French Le Line-field confocal optical coherence tomography for the in vivo real-time diagnosis of different stages of keratinocyte skin cancer: a preliminary study J Eur Acad Dermatol Venereol 2021 35 12 2388 2397 10.1111/jdv.17603 34415646
Ruini C, Schuh S, Gust C, Kendziora B, Frommherz L, French Le et al (2021) Line-field confocal optical coherence tomography for the in vivo real-time diagnosis of different stages of keratinocyte skin cancer: a preliminary study. J Eur Acad Dermatol Venereol 35(12):2388–239734415646 10.1111/jdv.17603
22. Cinotti E, Tognetti L, Cartocci A, Lamberti A, Gherbassi S, Orte Cano C et al (2021) Line-field confocal optical coherence tomography for actinic keratosis and squamous cell carcinoma: a descriptive study. Clin Exp Dermatol
23. Suppa M Fontaine M Dejonckheere G Cinotti E Yélamos O Diet G Line-field confocal optical coherence tomography of basal cell carcinoma: a descriptive study J Eur Acad Dermatol Venereol 2021 35 5 1099 1110 10.1111/jdv.17078 33398911
Suppa M, Fontaine M, Dejonckheere G, Cinotti E, Yélamos O, Diet G et al (2021) Line-field confocal optical coherence tomography of basal cell carcinoma: a descriptive study. J Eur Acad Dermatol Venereol 35(5):1099–111033398911 10.1111/jdv.17078
24. Gust C Schuh S Welzel J Daxenberger F Hartmann D French LE Line-field Confocal Optical Coherence Tomography increases the diagnostic accuracy and confidence for basal cell carcinoma in Equivocal lesions: a prospective study Cancers 2022 14 4 1082 10.3390/cancers14041082 35205830
Gust C, Schuh S, Welzel J, Daxenberger F, Hartmann D, French LE et al (2022) Line-field Confocal Optical Coherence Tomography increases the diagnostic accuracy and confidence for basal cell carcinoma in Equivocal lesions: a prospective study. Cancers 14(4):108235205830 10.3390/cancers14041082
25. Ruini C Schuh S Gust C Kendziora B Frommherz L French LE Line-field optical coherence tomography: in vivo diagnosis of basal cell carcinoma subtypes compared with histopathology Clin Exp Dermatol 2021 46 8 1471 1481 10.1111/ced.14762 34047380
Ruini C, Schuh S, Gust C, Kendziora B, Frommherz L, French LE et al (2021) Line-field optical coherence tomography: in vivo diagnosis of basal cell carcinoma subtypes compared with histopathology. Clin Exp Dermatol 46(8):1471–148134047380 10.1111/ced.14762
26. Donelli C Suppa M Tognetti L Perrot JL Calabrese L Pérez-Anker J Line-field Confocal Optical Coherence Tomography for the diagnosis of skin carcinomas: real-Life Data over three years Curr Oncol 2023 30 10 8853 8864 10.3390/curroncol30100639 37887539
Donelli C, Suppa M, Tognetti L, Perrot JL, Calabrese L, Pérez-Anker J et al (2023) Line-field Confocal Optical Coherence Tomography for the diagnosis of skin carcinomas: real-Life Data over three years. Curr Oncol 30(10):8853–886437887539 10.3390/curroncol30100639
27. Soglia S Pérez-Anker J Lobos Guede N Giavedoni P Puig S Malvehy J Diagnostics using Non-invasive technologies in Dermatological Oncology Cancers 2022 14 23 5886 10.3390/cancers14235886 36497368
Soglia S, Pérez-Anker J, Lobos Guede N, Giavedoni P, Puig S, Malvehy J (2022) Diagnostics using Non-invasive technologies in Dermatological Oncology. Cancers 14(23):588636497368 10.3390/cancers14235886
28. Jacobsen K Ortner VK Wenande E Sahu A Paasch U Haedersdal M Line-field confocal optical coherence tomography in dermato-oncology: a literature review towards harmonized histopathology-integrated terminology Exp Dermatol 2024 33 4 e15057 10.1111/exd.15057 38623958
Jacobsen K, Ortner VK, Wenande E, Sahu A, Paasch U, Haedersdal M (2024) Line-field confocal optical coherence tomography in dermato-oncology: a literature review towards harmonized histopathology-integrated terminology. Exp Dermatol 33(4):e1505738623958 10.1111/exd.15057
29. van Loo E Sinx KAE Welzel J Schuh S Kelleners-Smeets NWJ Mosterd K Cumulative Sum Analysis for the learning curve of Optical Coherence Tomography assisted diagnosis of basal cell carcinoma Acta Derm Venereol 2020 100 19 adv00343 10.2340/00015555-3696 33205824
van Loo E, Sinx KAE, Welzel J, Schuh S, Kelleners-Smeets NWJ, Mosterd K et al (2020) Cumulative Sum Analysis for the learning curve of Optical Coherence Tomography assisted diagnosis of basal cell carcinoma. Acta Derm Venereol 100(19):adv0034333205824 10.2340/00015555-3696
30. Conger AJ Integration and generalization of kappas for multiple raters Psychol Bull 1980 88 2 322 328 10.1037/0033-2909.88.2.322
Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull 88(2):322–32810.1037/0033-2909.88.2.322
31. Landis JR Koch GG The measurement of Observer Agreement for Categorical Data Biometrics 1977 33 1 159 10.2307/2529310 843571
Landis JR, Koch GG (1977) The measurement of Observer Agreement for Categorical Data. Biometrics 33(1):159843571 10.2307/2529310
32. Cicchetti DV Feinstein AR High agreement but low kappa: II. Resolving the paradoxes J Clin Epidemiol 1990 43 6 551 558 10.1016/0895-4356(90)90159-M 2189948
Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 43(6):551–5582189948 10.1016/0895-4356(90)90159-M
33. Feinstein AR Cicchetti DV High agreement but low kappa: I. The problems of two paradoxes J Clin Epidemiol 1990 43 6 543 549 10.1016/0895-4356(90)90158-L 2348207
Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 43(6):543–5492348207 10.1016/0895-4356(90)90158-L
34. Donner A, Rotondi MA Sample Size Requirements for Interval Estimation of the Kappa Statistic for Interobserver Agreement Studies with a Binary Outcome and Multiple Raters. Int J Biostat. 2010 Sep 22 [cited 2024 Jun 6];6(1). https://www.degruyter.com/document/doi/10.2202/1557-4679.1275/html
35. Gupta K Attri J Singh A Kaur H Kaur G Basic concepts for sample size calculation: critical step for any clinical trials! Saudi J Anaesth 2016 10 3 328 331 10.4103/1658-354X.174918 27375390
Gupta K, Attri J, Singh A, Kaur H, Kaur G (2016) Basic concepts for sample size calculation: critical step for any clinical trials! Saudi J Anaesth 10(3):328–33127375390 10.4103/1658-354X.174918
36. Serdar CC Cihan M Yücel D Serdar MA Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies Biochem Med 2021 31 1 010502 10.11613/BM.2021.010502
Serdar CC, Cihan M, Yücel D, Serdar MA (2021) Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochem Med 31(1):01050210.11613/BM.2021.010502
37. Vanbelle S Asymptotic variability of (multilevel) multirater kappa coefficients Stat Methods Med Res 2019 28 10–11 3012 3026 10.1177/0962280218794733 30132375
Vanbelle S (2019) Asymptotic variability of (multilevel) multirater kappa coefficients. Stat Methods Med Res 28(10–11):3012–302630132375 10.1177/0962280218794733
38. Perez-Anker J, Puig S, Alos L, García A, Alejo B, Cinotti E et al Morphologic evaluation of melanocytic lesions with three-dimensional line-field confocal optical coherence tomography: correlation with histopathology and reflectance confocal microscopy. A pilot study. Clin Exp Dermatol. [cited 2022 Sep 14];n/a(n/a). https://onlinelibrary.wiley.com/doi/abs/10.1111/ced.15383
39. Ruini C Schuh S Sattler E Welzel J Line-field confocal optical coherence tomography—practical applications in dermatology and comparison with established imaging methods Skin Res Technol 2021 27 3 340 352 10.1111/srt.12949 33085784
Ruini C, Schuh S, Sattler E, Welzel J (2021) Line-field confocal optical coherence tomography—practical applications in dermatology and comparison with established imaging methods. Skin Res Technol 27(3):340–35233085784 10.1111/srt.12949
40. Fischman S Pérez-Anker J Tognetti L Di Naro A Suppa M Cinotti E Non-invasive scoring of cellular atypia in keratinocyte cancers in 3D LC-OCT images using deep learning Sci Rep 2022 12 1 481 10.1038/s41598-021-04395-1 35013485
Fischman S, Pérez-Anker J, Tognetti L, Di Naro A, Suppa M, Cinotti E et al (2022) Non-invasive scoring of cellular atypia in keratinocyte cancers in 3D LC-OCT images using deep learning. Sci Rep 12(1):48135013485 10.1038/s41598-021-04395-1
41. Lenoir C Cinotti E Tognetti L Orte Cano C Diet G Miyamoto M Line-field confocal optical coherence tomography of actinic keratosis: a case series J Eur Acad Dermatol Venereol 2021 35 12 e900 e902 10.1111/jdv.17548 34310768
Lenoir C, Cinotti E, Tognetti L, Orte Cano C, Diet G, Miyamoto M et al (2021) Line-field confocal optical coherence tomography of actinic keratosis: a case series. J Eur Acad Dermatol Venereol 35(12):e900–e90234310768 10.1111/jdv.17548
42. Carpentier M Combescure C Merlini L Perneger TV Kappa statistic to measure agreement beyond chance in free-response assessments BMC Med Res Methodol 2017 17 62 10.1186/s12874-017-0340-6 28420347
Carpentier M, Combescure C, Merlini L, Perneger TV (2017) Kappa statistic to measure agreement beyond chance in free-response assessments. BMC Med Res Methodol 17:6228420347 10.1186/s12874-017-0340-6
43. Cappilli S Cinotti E Lenoir C Tognetti L Perez-Anker J Rubegni P Line-field confocal optical coherence tomography of basosquamous carcinoma: a case series with histopathological correlation J Eur Acad Dermatol Venereol 2022 36 8 1214 1218 10.1111/jdv.18038 35224784
Cappilli S, Cinotti E, Lenoir C, Tognetti L, Perez-Anker J, Rubegni P et al (2022) Line-field confocal optical coherence tomography of basosquamous carcinoma: a case series with histopathological correlation. J Eur Acad Dermatol Venereol 36(8):1214–121835224784 10.1111/jdv.18038
44. McHugh ML Interrater reliability: the kappa statistic Biochem Med 2012 22 3 276 282 10.11613/BM.2012.031
McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med 22(3):276–28210.11613/BM.2012.031
