
==== Front
BMC Oral Health
BMC Oral Health
BMC Oral Health
1472-6831
BioMed Central London

39285380
4696
10.1186/s12903-024-04696-7
Research
Assessing the usability and reliability of a web-based teledentistry tool for remote diagnosis of oral lesions: a cross-sectional study
Niknam Fatemeh 1
Mardani Maryam 2
Bastani Peivand 3
Bashiri Azadeh 1
Ha Diep 4
Sookhakian Asma 5
Akbari Reza 6
Sharifian Roxana sharifianroxana@gmail.com

1
1 https://ror.org/03w04rv71 grid.411746.1 0000 0004 4911 7066 Department of Health Information Management, Iran University of Medical Science, Tehran,, Iran
2 https://ror.org/01n3s4692 grid.412571.4 0000 0000 8819 4698 Department of Health Information Management, Shiraz University of Medical Sciences, Shiraz, Iran
3 https://ror.org/01kpzv902 grid.1014.4 0000 0004 0367 2697 College of Business, Government and Law, Flinders University, Adelaide, Bedford Park, SA 5042 Australia
4 https://ror.org/00rqy9422 grid.1003.2 0000 0000 9320 7537 School of Dentistry, UQ Oral Health Centre, The University of Queensland, Brisbane, QLD 4006 Australia
5 https://ror.org/01n3s4692 grid.412571.4 0000 0000 8819 4698 Student Research Committee, School of Dentistry, Shiraz University of Medical Sciences, Shiraz, Iran
6 https://ror.org/04bxa3v83 grid.444860.a 0000 0004 0600 0546 Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran
16 9 2024
16 9 2024
2024
24 109411 1 2024
1 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Background

Oral mucosa lesions are the third most prevalent oral pathology, following caries and periodontal diseases. Teledentistry offers an effective way to manage patients with these lesions. The accuracy of remote diagnoses and consultations relies heavily on the quality of the information and photos sent to remote specialists. This study aims to evaluate the usability and reliability of a teledentistry tool for the remote diagnosis of oral lesions.

Methods

The cross-sectional study included both usability evaluation and reliability assessment. The teledentistry platform, "OralMedTeledent", facilitated synchronous and asynchronous interactions, allowing for patient consultations, remote follow-ups, and doctor-to-doctor consultations. Usability was evaluated by 5 experts using the Nielsen heuristic checklist. Reliability was assessed from August 2022 to September 2023 with 109 patients, using Cohen's kappa coefficient to measure agreement between examiners and the gold standard in diagnosing oral lesions.

Results

The findings revealed 66 usability issues, most of which were related to helping users recognize, diagnose, and recover from errors, as well as issues with help and documentation. Among these, 11 issues were of minor severity. The reliability test, conducted with 109 participants (57.8% female, 42.2% male) showed that the web-based teleconsultation system performed significantly well. The system demonstrated significant substantial performance (0.81 ≤ κ < 1; P > 0.05).

Conclusion

Overall, the web-based teleconsultation system has proven to be reliable for the remote diagnosis of oral lesions, making it a valuable alternative during emergencies such as the COVID-19 pandemic. However, several usability issues have been identified and need to be addressed.

Keywords

Reliability
Remote diagnosis
Teleconsultation
Oral lesion
Teledentistry
Usability
issue-copyright-statement© BioMed Central Ltd., part of Springer Nature 2024
==== Body
pmcBackground

Oral mucosa lesions are the third most prevalent oral pathology, following caries and periodontal diseases [1]. These lesions refer to abnormalities or changes in the tissues lining the oral cavity [2, 3]. They can vary in size, appearance, and location, and may be caused by infection, inflammation, trauma, or neoplastic processes [4]. The overlapping signs and symptoms of these conditions present significant diagnostic challenges, which can only be resolved through clinicopathological tests and the expertise of an experienced oral medicine specialist [5]. Therefore, proper classification, diagnosis, and management of oral mucosal lesions are essential for providing appropriate care and ensuring positive patient outcomes [5, 6].

Teledentistry offers an effective approach to managing patients with oral lesions [7]. It involves the use of electronic health records, digital imaging, photography, and information and communications technology to transmit clinical information over the Internet both synchronously and asynchronously [8]. Teledentistry enables dental professionals to collaborate by exchanging insights and expertise regarding challenging cases, uncertainties, and patient care [9, 10]. This approach allows general practitioners to quickly access specialist opinions, ensuring optimal patient outcomes [8, 11]. Additionally, applying teledentistry in the field of oral medicine field has several advantages, including reducing unnecessary referrals [12–14], improving the quality of care [14, 15], facilitating patient-professional communication [14, 16, 17], enabling continuous virtual consultations and follow-ups [18–20], and monitoring oral medical emergencies [16, 19].

Telediagnosis and teleconsultation of oral mucosa lesions are among the most widely used applications of teledentistry in oral medicine. [9]. Several studies have highlighted teledentistry tools, including using smartphones to photograph oral lesions [21–26]. These photos, along with other clinical information, are shared through various means such as social media platforms [10, 23, 27, 28], emails [29], and cloud-based teledentistry platforms [14, 21, 30]. According to some studies, the accuracy of remote diagnoses and consultations depends on the quality of the information and photos sent to remote specialists [14, 31–33]. Therefore, testing the reliability of teledentistry tools is crucial, as emphasized in many studies [31, 34, 35].

In addition to testing the reliability of teledentistry platform, usability is a key consideration in the design and evaluation of software, websites, applications, and various other products to ensure they meet the needs and expectations of their users [36]. Usability refers to the extent to which a product, system, or interface can be used by a specified set of users to achieve specific goals with effectiveness, efficiency, and satisfaction in a particular context [37]. Assessing usability allows for the identification and resolution of difficulties, ensuring that telehealth systems are beneficial. Usability assessment covers aspects such as ease of learning, retention of task performance over time, speed of task performance, a low error rate, and subjective user satisfaction [38, 39]. This study aims to evaluate the usability and reliability of a web-based tele dentistry tool for diagnosing oral lesions through a cross-sectional investigation.

Methods

General description of teledentistry platform

The teledentistry platform, "OralMedTeledent," is a web-based teleconsultation system specifically designed for the remote diagnosis of patients with oral lesions. This system supports both synchronous and asynchronous interactions, allowing for consultations between patients and oral medicine specialists, remote follow-ups of patient conditions, doctor-to-doctor consultations, recording and sending photographs of oral cavity lesions to remote consultants, and the transmission of demographic and clinical information for remote consultations.

Evaluation method

This cross-sectional study included two phases: usability evaluation and reliability assessment. The Overview of research method is illustrated in Fig. 1.Fig. 1 The overview of research method

Phase one: usability evaluation

In this phase, usability was assessed using the Nielsen heuristic checklist, which includes ten principles: 1) visibility of system status; 2) match between system and the real world; 3) user control and freedom; 4) consistency and standards; 5) error prevention; 6) recognition rather than recall; 7) flexibility and efficiency of use; 8) aesthetic and minimalist design; 9) help users recognize, diagnose, and recover from errors; and 10) help and documentation. Heuristic evaluation is a widely used method for assessing usability, where 3 to 5 usability experts review the user interface based on these established principles [40].

A team of five specialists with expertise in heuristic evaluation conducted the usability assessment. The team included a PhD in Health Information Management, a PhD in Medical Informatics, a software engineer with experience in various health information systems, an M.Sc. student in Health Information Technology familiar with health information systems, and a professor in the Oral Medicine field. Each evaluator had previous experience with usability evaluation studies, ensuring their familiarity with heuristic evaluation methods.

The evaluators individually reviewed the teleconsultation system's user interface. They marked 'yes' for usability items that were met and 'no' for those that were not. For items marked "no", they rated the severity of the problem on a scale from 0 to 4 using Nielson`s scale (Table 1). The collected data were then analyzed using descriptive statistics in an Excel sheet. Table 1 Nielsen’s severity rating scale for usability problem

Problem	Severity	Description	
No problem	0	I do not agree that this is a usability problem at all	
Cosmetic	1	This need not be fixed unless extra time is available on the project	
Minor	2	Fixing this should be given low priority	
Major	3	Important to fix, so should be given high priority	
Catastrophic	4	Imperative to fix this problem	

Phase two: reliability assessment

Ethical consideration

A cross‐sectional study was conducted from August 2022 to September 2023 with a convenience sample of patients who visited the Oral, Maxillofacial and Dental Diseases Diagnosis Department at Shiraz Faculty of Dentistry in Iran. The study protocol was approved by the Research Ethics Committee of Shiraz University of Medical Science, Iran, under code IR.SUMS.DENTAL.REC.1401.072.

Before commencing the study, participants were thoroughly informed about the study`s objectives and conditions to ensure ethical considerations and awareness. They were made aware that participation was entirely voluntary and would not affect their treatment process. After being fully informed, participants provided written consent.

Patient confidentiality was strictly maintained throughout the study. Only the treatment team had access to patient information. To protect patients' privacy, participants were enrolled in the teledentistry platform anonymously, identified only by unique codes like 'patient 38'. During oral lesion photography, only the oral cavity and lesions were captured to avoid including any identifiable features such as the face. Images of the forehead or eyes were deliberately excluded.

To safeguard patient digital data, security measures were implemented, including user access controls, authentication procedures with usernames and passwords, and the use of secure servers managed by Shiraz University of Medical Sciences. Data encryption protocols were applied during both transmission and storage.

Patients did not participate directly in teledentistry interventions; only photographs of their oral lesions were used to assess the agreement between in-person and remote diagnoses.

Sample size and inclusion criteria

The required sample size was estimated to be 97 participants based on statistical data from similar studies [21, 34]. The calculation was performed using the Kappa coefficient method with the online Sample Size Calculator tool (https://wnarifin.github.io/ssc_web.html) [41, 42]. Details of the sample size calculation method and formula are given below. To account for potential dropouts or loss to follow-up, a total of 109 samples were ultimately collected for the study.

The method of calculating the sample size based on the kappa coefficient:Kappa (2 raters)—estimation2Expected kappa (K): 0.66

Precision (± expected): 0.15

Proportion of outcome (p): 0.5

Confidence level 100(1-α): 95%

Expected drop-out rate: 10%

Sample size: n = 97

Sample size (with 10% drop-out), n drop = 108

The inclusion criteria for this study required patients to have been diagnosed with oral lesions, including white and red lesions, ulcerative, vesicular, and bullous lesions, oral exophytic lesions, hard tissue lesions, and pigmented lesions of the oral mucosa. Patients with conditions unrelated to oral lesions, such as orofacial pain disorders, were excluded from this study.

Study design

This study comprised four steps:I. Step 1: Physical examination

Eligible patients were examined by a professor (examiner 1) from the Oral, Maxillofacial, and Dental Diseases Department. Each patient was then assigned to a code, and their clinical information and final diagnosis were documented.II. Step 2: Photography of oral lesions in patients

After the patient was examined, two residents trained in oral cavity photography took pictures of the patient's mouth, focusing on the oral lesions. The photos were taken with an iPhone featuring a 10-megapixel camera and a display resolution of 720 × 1280 pixels with autofocus (Fig. 2). In total, 262 photos were captured by two residents, who then reviewed the photographs for clarity and to ensure all relevant areas were covered.III. Step 3: Data Entry into the web-based teleconsultation system

Fig. 2 Example of an oral lesion with anterior and lateral views captured using an iPhone 13 Pro Max. Case number 38, final diagnosis: Oral lichen planus

Initially, two residents and two professors were trained on how to use the web-based teleconsultation system. The residents then submitted patient information, including photographs of oral lesions, demographic data, and clinical details, as separate teleconsultation requests through the application, without including the final diagnosis. These requests were sent to two professors in the department, who served as examiners (Examiner 1 and Examiner 2). The professors had profiles on this web-based teleconsultation system, which allowed them to access and respond to the consultation requests.IV. Step 4: Compare in-person diagnosis vs. remote diagnosis

At least four months after each patient`s initial examination, their information and lesion photographs were independently reviewed by two examiners using a web-based teleconsultation system. Both examiners analyzed the photos and information on their computers, as provided by the Department of Oral, Maxillofacial, and Dental Diseases for its professors. After reviewing the data, the examiners documented a final diagnosis for each patient. The level of agreement between the two examiners and between the examiners and the gold standard (in-person clinical oral examination) was assessed using Cohen's kappa coefficient. The interpretation of Cohen’s kappa coefficient is as follows: Perfect agreement (κ = 1), substantial agreement (0.81 ≤ κ < 1), moderate agreement (0.61 ≤ κ < 0.81), fair Agreement (0.41 ≤ κ < 0.61), slight agreement (0.21 ≤ κ < 0.41), and poor agreement (κ < 0.21). Data analysis was performed using IBM SPSS Statistics 22.

Result

Findings of phase 1: usability evaluation

Figure 2 illustrates the number of usability issues and their average severity ratings based on Nielsen’s severity rating scale.

The results revealed 66 usability problems. Most of these issues were associated with helping users recognize, diagnose, and recover from errors, as well as to help and documentation, with 11 being minor severity issues. The average severity scores for the identified heuristic violations ranged from 0 (indicating a good match between the system and the real world) to 3.00 (indicating issues with consistency, standards, aesthetics, and minimalist design). Scores closer to 0 suggest a more usable system (Fig. 3).Fig. 3 The number of usability problems and the average severity scores on Nielsen’s severity rating scale

Findings of phase 2: reliability assessment

In this study, out of 109 participants, 63 (57.8%) were female and 46 (42.2%) were male, with an average age of 47 ± 19.46 years. Table 2 presents the concordance results between the gold standard and Examiner 1 and Examiner 2, as well as the internal agreement between the two examiners. The findings, detailed in Table 1, demonstrate that the web-based teleconsultation system performed substantially (0.81 ≤ κ < 1) with statistical significance (P > 0.05). Table 2 Results of the concordance between face-to-face oral examination diagnosis and remote diagnosis

Agreement type	Kappa (95% CI)	N	p-value	
Gold standard- examiner1	0.91(0.85–0.96)	109	P < 0.001	
Gold standard- examiner2	0.89 (0.83–0.95)	109	P < 0.001	
Examiner1- examiner2	0.85(0.78–0.92)	109	P < 0.001	

Discussion

The usability evaluation revealed 66 issues, primarily concerning users` ability to recognize, diagnose, and recover from errors, as well as help and documentation. Nielsen's usability evaluation, employed in this study, assessed how well the user interface adhered to design principles, making it a useful tool for software and information system developers [43]. Similar heuristic evaluation methods have been applied in various studies to evaluate the usability of telehealth systems. For instance, Stein et al. (2016) utilized the heuristic method to evaluate a prototype mobile application for emergency triage in teledentistry, revealing usability issues related to stability and standards [44]. Lilleholt et al. (2015) employed Nielsen's exploratory evaluation method to evaluate a telehealth system, identifying key issues with system-world match, consistency and standards, and aesthetic minimalist design [45]. Khashe et al. (2021) used Nielsen's heuristic evaluation to assess a virtual visit system, noting usability issues related to error prevention, help and documentation, visibility of system status, system-world match, and user control [46]. While these studies differ in focus, they all consistently address usability issues.

The results highlight the significant performance of the web-based teleconsultation system. Key factors contributing to this performance include the use of the iPhone 13 Pro Max for capturing oral cavity photos. Improving photo quality is crucial for enhancing the accuracy and precision of remote diagnosis [14, 31]. This study also addressed essential requirements for oral cavity photography, such as proper patient positioning [25], adequate lighting [21, 25, 47], and using the back camera of a smartphone [25].

Examiners using this web-based teleconsultation system can access comprehensive and detailed patient information, including demographic data, clinical examinations, and laboratory test results. This access improves the accuracy of remote diagnoses and increases agreement between on-site and remote diagnoses. Having correct, accurate, timely and complete information is a key facilitator in the effectiveness of teledentistry [9].

The study`s findings aligned with those of Vinayagamoorthy et al. (2019), who also found significant agreement between diagnosing oral potentially malignant disorders via the WhatsApp application and face-to-face examination [31]. However, the level of agreement in our study was higher, which may be attributed to the use of different teledentistry platforms and variations in study methodologies. Our results are also aligned with Estai et al. (2017) who compared in-person and teledentistry methods for diagnosing dental caries using a Remote-I platform and found high agreement between the two methods [34]. Additionally, our findings are in line with those of AlShaya et al. (2020), supporting the acceptable reliability of telediagnosis of oral lesions through photographs and teledentistry platforms [35].

Despite these findings, there are some limitations to consider. The study protocol did not include measurement of sensitivity, specificity, positive predictive value, and negative predictive value. Consequently, reliability was assessed based on the calculation of Cohen's kappa statistic and the agreement between the two evaluators. Furthermore, the agreement between face-to-face and remote diagnosis was considered binary, with diagnoses categorized as either 0 or 1 based on an exact diagnostic match. The use of the iPhone 13 Pro Max likely improved the quality of oral lesion images, which may have positively impacted the reliability of the results.

Conclusion

In summary, the web-based teleconsultation system demonstrated substantial performance in remotely diagnosing oral lesions, making it a valuable alternative, especially during emergencies like the COVID-19 pandemic. When face-to-face visits are not possible for timely diagnosis or ongoing treatment, this system provides a practical solution. Moreover, it offers the convenience of remote follow-up sessions for patients residing in rural areas who might have difficulty attending in-person appointments.

The versatility of the system also facilitates consultations among oral medicine specialists, physicians, dentists, and other healthcare professionals, promoting collaborative decision-making and knowledge sharing.

However, it is crucial to recognize that the teleconsultation system faces challenges. The identification of various usability issues highlights the need to address these concerns to optimize the effectiveness and user experience of the system. Ongoing refinement and improvements in usability are essential for the system`s overall success and acceptance in the field of oral medicine.

Acknowledgements

The authors wish to thank all study participants for their valuable cooperation. Additionally, the authors would like to thank the professors, residents, and staff of Oral, Maxillofacial, and Dental Diseases Department at the School of Dentistry, Shiraz University of Medical Science, for their assistance and support.

Authors’ contributions

FN performed data collection, data analysis, writing the manuscript, contributed in the study design, and trained and supervised residents and professors to use teleconsultation system, MM was clinical advisor and contributed in clinical examination and study design, R SH and AB were supervisors and contributed in the study design, PB performed English language editing and critical revision of the manuscript, DH performed critical revision of the manuscript, AS contributed in data collection, RA was a technical advisor and confirmed the technical specifications of the teleconsultation system.

Funding

No funding.

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to the need to protect the privacy of study participants, but are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

The study protocol was approved by the Research Ethics Committee of Shiraz University of Medical Science, Iran, under code IR.SUMS.DENTAL.REC.1401.072. All methods were carried out according to the relevant guidelines and regulations. Informed consent was obtained from all the participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Radwan-Oczko M Sokół I Babuśka K Owczarek-Drabińska JE Prevalence and characteristic of oral mucosa lesions Symmetry 2022 14 2 307 10.3390/sym14020307
Radwan-Oczko M, Sokół I, Babuśka K, Owczarek-Drabińska JE. Prevalence and characteristic of oral mucosa lesions. Symmetry. 2022;14(2):307.
2. Ge S Liu L Zhou Q Lou B Zhou Z Lou J Prevalence of and related risk factors in oral mucosa diseases among residents in the Baoshan District of Shanghai China PeerJ 2020 8 e8644 10.7717/peerj.8644 32140308
Ge S, Liu L, Zhou Q, Lou B, Zhou Z, Lou J, et al. Prevalence of and related risk factors in oral mucosa diseases among residents in the Baoshan District of Shanghai. China PeerJ. 2020;8:e8644.32140308
3. Beaty CS, Short AG, Mewar P. Oral Mucosal Lesions, Immunologic Diseases. 2023 Nov 14. In: StatPearls [Internet]. Treasure Island: StatPearls Publishing; 2024.
4. Chiang CP Chang JYF Wang YP Wu YH Lu SY Sun A Oral lichen planus–differential diagnoses, serum autoantibodies, hematinic deficiencies, and management J Formos Med Assoc 2018 117 9 756 65 10.1016/j.jfma.2018.01.021 29472048
Chiang CP, Chang JYF, Wang YP, Wu YH, Lu SY, Sun A. Oral lichen planus–differential diagnoses, serum autoantibodies, hematinic deficiencies, and management. J Formos Med Assoc. 2018;117(9):756–65.29472048
5. Glick M, Greenberg MS, Lockhart PB, Challacombe SJ. Introduction to oral medicine and oral diagnosis: patient evaluation. Burket's Oral Medicine. 13th ed. Wiley-Blackwell; 2021. p. 1–18.
6. Zahid E Bhatti O Zahid MA Stubbs M Overview of common oral lesions Malays Fam Physician 2022 17 3 9 10.51866/rv.37 36606178
Zahid E, Bhatti O, Zahid MA, Stubbs M. Overview of common oral lesions. Malays Fam Physician. 2022;17(3):9.36606178
7. Flores APdC Lazaro SA Molina-Bastos CG Guattini VLdO Umpierre RN Gonçalves MR Teledentistry in the diagnosis of oral lesions: A systematic review of the literature J Am Med Inform Assoc 2020 27 7 1166 72 10.1093/jamia/ocaa069 32568392
Flores APdC, Lazaro SA, Molina-Bastos CG, Guattini VLdO, Umpierre RN, Gonçalves MR, et al. Teledentistry in the diagnosis of oral lesions: A systematic review of the literature. J Am Med Inform Assoc. 2020;27(7):1166–72.32568392
8. Torres-Pereira CC Morosini IdAC Possebon RS Giovanini AF Bortoluzzi MC Leao JC Teledentistry: distant diagnosis of oral disease using e-mails Telemed Health 2013 19 2 117 21 10.1089/tmj.2012.0087
Torres-Pereira CC, Morosini IdAC, Possebon RS, Giovanini AF, Bortoluzzi MC, Leao JC, et al. Teledentistry: distant diagnosis of oral disease using e-mails. Telemed Health. 2013;19(2):117–21.
9. Niknam F Sharifian R Bashiri A Mardani M Akbari R Tuffaha H Tele-dentistry, its trends, scope, and future framework in oral medicine; a scoping review during January 1999 to December 2021 Arch Pub Health 2023 81 1 1 14 36600260
Niknam F, Sharifian R, Bashiri A, Mardani M, Akbari R, Tuffaha H, et al. Tele-dentistry, its trends, scope, and future framework in oral medicine; a scoping review during January 1999 to December 2021. Arch Pub Health. 2023;81(1):1–14.36600260
10. Pérez González A, Gallas Torreira M, Chamorro Petronacci CM, Pérez Sayáns M. Teledentistry: a new approach in dental medicine. In Enhanced Telemedicine and e-Health: Advanced IoT Enabled Soft Computing Framework. Cham: Springer International Publishing; 2021. p. 41–64.
11. Tesfalul M Littman-Quinn R Antwi C Ndlovu S Motsepe D Phuthego M Evaluating the potential impact of a mobile telemedicine system on coordination of specialty care for patients with complicated oral lesions in Botswana J Am Med Inform Assoc 2016 23 e1 e142 e145 10.1093/jamia/ocv140 26510877
Tesfalul M, Littman-Quinn R, Antwi C, Ndlovu S, Motsepe D, Phuthego M, et al. Evaluating the potential impact of a mobile telemedicine system on coordination of specialty care for patients with complicated oral lesions in Botswana. J Am Med Inform Assoc. 2016;23(e1):e142–5.26510877
12. Goffin G Carter N Sari Widyarman A Erri Astoeti T Kabir Bulbul H Puplampu P Role of teledentistry in enabling improved oral care outcomes Br Dent J 2024 236 3 162 168 10.1038/s41415-024-7055-y 38332075
Goffin G, Carter N, Sari Widyarman A, Erri Astoeti T, Kabir Bulbul H, Puplampu P, et al. Role of teledentistry in enabling improved oral care outcomes. Br Dent J. 2024;236(3):162–8.38332075
13. Di Fede O Panzarella V Buttacavoli F La Mantia G Campisi G Doctoral: A smartphone-based decision support tool for the early detection of oral potentially malignant disorders Digital Health 2023 9 20552076231177141 37256016
Di Fede O, Panzarella V, Buttacavoli F, La Mantia G, Campisi G. Doctoral: A smartphone-based decision support tool for the early detection of oral potentially malignant disorders. Digital Health. 2023;9:20552076231177140.37256016
14. Birur NP Gurushanth K Patrick S Sunny SP Raghavan SA Gurudath S Role of community health worker in a mobile health program for early detection of oral cancer Indian J Cancer 2019 56 2 107 113 10.4103/ijc.IJC_232_18 31062727
Birur NP, Gurushanth K, Patrick S, Sunny SP, Raghavan SA, Gurudath S, et al. Role of community health worker in a mobile health program for early detection of oral cancer. Indian J Cancer. 2019;56(2):107–13.31062727
15. Kengne Talla P Allison P Bussières A Giraudeau N Komarova S Basiren Q Teledentistry for improving access to, and quality of oral health care: A protocol for an overview of systematic reviews and meta-analyses PLoS ONE 2024 19 1 e0288677 10.1371/journal.pone.0288677 38165889
Kengne Talla P, Allison P, Bussières A, Giraudeau N, Komarova S, Basiren Q, et al. Teledentistry for improving access to, and quality of oral health care: A protocol for an overview of systematic reviews and meta-analyses. PLoS ONE. 2024;19(1):e0288677.38165889
16. Alves FA Saunders D Sandhu S Xu Y de Mendonça NF Treister NS Implication of COVID-19 in oral oncology practices in Brazil, Canada, and the United States Oral Dis 2021 27 Suppl 3 793 10.1111/odi.13493 32557965
Alves FA, Saunders D, Sandhu S, Xu Y, de Mendonça NF, Treister NS. Implication of COVID-19 in oral oncology practices in Brazil, Canada, and the United States. Oral Dis. 2021;27(Suppl 3):793.32557965
17. Bastani P Niknam F Rezazadeh M Rossi-Fedele G Edirippulige S Samadbeik M Dentistry website analysis: An overview of the content of formulated questions and answers Heliyon. 2022 8 e10250 10.1016/j.heliyon.2022.e10250 36042730
Bastani P, Niknam F, Rezazadeh M, Rossi-Fedele G, Edirippulige S, Samadbeik M. Dentistry website analysis: An overview of the content of formulated questions and answers. Heliyon. 2022;8:e10250.36042730
18. Blomstrand L Sand LP Gullbrandsson L Eklund B Kildal M Hirsch J-M Telemedicine—A complement to traditional referrals in oral medicine Telemedicine and e-Health 2012 18 7 549 553 10.1089/tmj.2011.0207 22823077
Blomstrand L, Sand LP, Gullbrandsson L, Eklund B, Kildal M, Hirsch J-M. Telemedicine—A complement to traditional referrals in oral medicine. Telemedicine and e-Health. 2012;18(7):549–53.22823077
19. Macken J Fortune F Buchanan J Remote telephone clinics in oral medicine: reflections on the place of virtual clinics in a specialty that relies so heavily on visual assessment. A note of caution Br J Oral and Maxillofac Surg 2021 59 5 605 8 10.1016/j.bjoms.2020.11.010 33863593
Macken J, Fortune F, Buchanan J. Remote telephone clinics in oral medicine: reflections on the place of virtual clinics in a specialty that relies so heavily on visual assessment. A note of caution. Br J Oral and Maxillofac Surg. 2021;59(5):605–8.33863593
20. Fricton J Chen H Using teledentistry to improve access to dental care for the underserved Dental Clinics 2009 53 3 537 548 19482128
Fricton J, Chen H. Using teledentistry to improve access to dental care for the underserved. Dental Clinics. 2009;53(3):537–48.19482128
21. Haron N Zain RB Nabillah WM Saleh A Kallarakkal TG Ramanathan A Mobile phone imaging in low resource settings for early detection of oral cancer and concordance with clinical oral examination Telemedicine and e-Health 2017 23 3 192 199 10.1089/tmj.2016.0128 27541205
Haron N, Zain RB, Nabillah WM, Saleh A, Kallarakkal TG, Ramanathan A, et al. Mobile phone imaging in low resource settings for early detection of oral cancer and concordance with clinical oral examination. Telemedicine and e-Health. 2017;23(3):192–9.27541205
22. Maret D Warnakulasuriya S Herbault-Barres B Savall F Vigarios E Telemedicine contributing to an incidental finding of a premalignant lesion Oral Oncol 2021 118 105331 10.1016/j.oraloncology.2021.105331 34024725
Maret D, Warnakulasuriya S, Herbault-Barres B, Savall F, Vigarios E. Telemedicine contributing to an incidental finding of a premalignant lesion. Oral Oncol. 2021;118:105331.34024725
23. Perdoncini NN Schussel JL Amenábar JM Torres-Pereira CC Use of smartphone video calls in the diagnosis of oral lesions: Teleconsultations between a specialist and patients assisted by a general dentist J Am Dent Assoc 2021 152 2 127 135 10.1016/j.adaj.2020.10.013 33494867
Perdoncini NN, Schussel JL, Amenábar JM, Torres-Pereira CC. Use of smartphone video calls in the diagnosis of oral lesions: Teleconsultations between a specialist and patients assisted by a general dentist. J Am Dent Assoc. 2021;152(2):127–35.33494867
24. Deshpande S Patil D Dhokar A Bhanushali P Katge F Teledentistry: A boon amidst COVID-19 lockdown—a narrative review Int J Telemed Appl 2021 2021 1 6
Deshpande S, Patil D, Dhokar A, Bhanushali P, Katge F. Teledentistry: A boon amidst COVID-19 lockdown—a narrative review. Int J Telemed Appl. 2021;2021:1–6.
25. Maret D Peters OA Auria JP Savall F Vigarios E Smartphone oral self-photography in teledentistry: recommendations for the patient J Telemed Telecare 2021 30 186 193 10.1177/1357633X211028513 34160332
Maret D, Peters OA, Auria JP, Savall F, Vigarios E. Smartphone oral self-photography in teledentistry: recommendations for the patient. J Telemed Telecare. 2021;30:186–93.34160332
26. Di Fede O La Mantia G Cimino MG Campisi G Protection of patient data in digital Oral and general health care: A scoping review with respect to the current regulations Oral 2023 3 2 155 165 10.3390/oral3020014
Di Fede O, La Mantia G, Cimino MG, Campisi G. Protection of patient data in digital Oral and general health care: A scoping review with respect to the current regulations. Oral. 2023;3(2):155–65.
27. Zhou MX Johnson EF Arce K Gruwell SF Teledentistry in the management of a non-Hodgkin's lymphoma manifesting as a gingival swelling: A case report Spec Care Dentist 2022 42 1 86 90 10.1111/scd.12638 34329504
Zhou MX, Johnson EF, Arce K, Gruwell SF. Teledentistry in the management of a non-Hodgkin’s lymphoma manifesting as a gingival swelling: A case report. Spec Care Dentist. 2022;42(1):86–90.34329504
28. Lv N Sun M Polonowita A Mei L Guan G Management of oral medicine emergencies during COVID-19: A study to develop practise guidelines Journal of dental sciences 2021 16 1 493 500 10.1016/j.jds.2020.07.016 32837687
Lv N, Sun M, Polonowita A, Mei L, Guan G. Management of oral medicine emergencies during COVID-19: A study to develop practise guidelines. Journal of dental sciences. 2021;16(1):493–500.32837687
29. Fonseca BB Perdoncini NN da Silva VC Gueiros LAM Carrard VC Lemos CA Jr Telediagnosis of oral lesions using smartphone photography Oral Dis 2022 28 6 1573 1579 10.1111/odi.13972 34289201
Fonseca BB, Perdoncini NN, da Silva VC, Gueiros LAM, Carrard VC, Lemos CA Jr, et al. Telediagnosis of oral lesions using smartphone photography. Oral Dis. 2022;28(6):1573–9.34289201
30. Gambino O, Lima F, Pirrone R, Ardizzone E, Campisi G, di Fede O. Second opinion system for intraoral lesions. In 2014 IEEE 27th International Symposium on Computer-Based Medical Systems. IEEE; 2014. p. 495–6.
31. Vinayagamoorthy K Acharya S Kumar M Pentapati KC Acharya S Efficacy of a remote screening model for oral potentially malignant disorders using a free messaging application: a diagnostic test for accuracy study Aust J Rural Health 2019 27 2 170 176 10.1111/ajr.12496 30942518
Vinayagamoorthy K, Acharya S, Kumar M, Pentapati KC, Acharya S. Efficacy of a remote screening model for oral potentially malignant disorders using a free messaging application: a diagnostic test for accuracy study. Aust J Rural Health. 2019;27(2):170–6.30942518
32. Valizadeh-Haghi H, Valizadeh-Haghi S, Naslseraji N, Zandian H. Smartphone Photography as a Teledentistry Method to Evaluate Anterior Composite Restorations. Int J Dent. 2023;2023(1):3171140.
33. Jeong J-S Pang N-S Choi Y Park K-M Kim T Xu X Importance of photography education to improve image quality for accurate remote diagnoses in dental trauma patients: observational study JMIR Mhealth Uhealth 2020 8 3 e15152 10.2196/15152 32213475
Jeong J-S, Pang N-S, Choi Y, Park K-M, Kim T, Xu X, et al. Importance of photography education to improve image quality for accurate remote diagnoses in dental trauma patients: observational study. JMIR Mhealth Uhealth. 2020;8(3):e15152.32213475
34. Estai M Kanagasingam Y Huang B Shiikha J Kruger E Bunt S Comparison of a smartphone-based photographic method with face-to-face caries assessment: a mobile teledentistry model Telemed e-Health 2017 23 5 435 440 10.1089/tmj.2016.0122
Estai M, Kanagasingam Y, Huang B, Shiikha J, Kruger E, Bunt S, et al. Comparison of a smartphone-based photographic method with face-to-face caries assessment: a mobile teledentistry model. Telemed e-Health. 2017;23(5):435–40.
35. AlShaya MS Assery MK Pani SC Reliability of mobile phone teledentistry in dental diagnosis and treatment planning in mixed dentition J Telemed Telecare 2020 26 1–2 45 52 10.1177/1357633X18793767 30134778
AlShaya MS, Assery MK, Pani SC. Reliability of mobile phone teledentistry in dental diagnosis and treatment planning in mixed dentition. J Telemed Telecare. 2020;26(1–2):45–52.30134778
36. Jeddi FR Nabovati E Bigham R Farrahi R Usability evaluation of a comprehensive national health information system: A heuristic evaluation Informatics in Medicine Unlocked 2020 19 100332 10.1016/j.imu.2020.100332
Jeddi FR, Nabovati E, Bigham R, Farrahi R. Usability evaluation of a comprehensive national health information system: A heuristic evaluation. Informatics in Medicine Unlocked. 2020;19:100332.
37. Martins, AI, Queirós A, Silva AG, Rocha NP. Usability Evaluation Methods: A Systematic Review. Human-Computer Interaction: Concepts, Methodologies, Tools, and Applications. 2016. p. 613–36.
38. Koohjani Z, Aslani A, Abasi S, Kyiani S. A comprehensive tool for usability evaluation of telehealth. In pHealth 2019. IOS Press; 2019. p. 168–73.
39. Kasim HF, Salih AI, Attash FM. Usability of telehealth among healthcare providers during COVID-19 pandemic in Nineveh Governorate, Iraq. Public Health Pract. 2023;5:100368.
40. Jaspers MW A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence Int J Med Informatics 2009 78 5 340 353 10.1016/j.ijmedinf.2008.10.002
Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Informatics. 2009;78(5):340–53.
41. Donner A Eliasziw M A goodness-of-fit approach to inference procedures for the kappa statistic: confidence and sample size estimation interval construction, significance-testing Stat Med 1992 11 11 1511 1519 10.1002/sim.4780111109 1410963
Donner A, Eliasziw M. A goodness-of-fit approach to inference procedures for the kappa statistic: confidence and sample size estimation interval construction, significance-testing. Stat Med. 1992;11(11):1511–9.1410963
42. Shoukri MM Asyali M Donner A Sample size requirements for the design of reliability study: review and new results Stat Methods Med Res 2004 13 4 251 271 10.1191/0962280204sm365ra
Shoukri MM, Asyali M, Donner A. Sample size requirements for the design of reliability study: review and new results. Stat Methods Med Res. 2004;13(4):251–71.
43. Nielsen J. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. New York: Association for Computing Machinery; 1994. p. 152–158.
44. Stein CD Xiao X Levine S Schleyer TK Hochheiser H Thyvalikakath TP A prototype mobile application for triaging dental emergencies J Am Dent Assoc 2016 147 10 782 91 10.1016/j.adaj.2016.03.021 27206728
Stein CD, Xiao X, Levine S, Schleyer TK, Hochheiser H, Thyvalikakath TP. A prototype mobile application for triaging dental emergencies. J Am Dent Assoc. 2016;147(10):782–91. e1.27206728
45. Lilholt PH Jensen MH Hejlesen OK Heuristic evaluation of a telehealth system from the Danish TeleCare North Trial Int J Med Informatics 2015 84 5 319 326 10.1016/j.ijmedinf.2015.01.012
Lilholt PH, Jensen MH, Hejlesen OK. Heuristic evaluation of a telehealth system from the Danish TeleCare North Trial. Int J Med Informatics. 2015;84(5):319–26.
46. Khashe Y, Tabibzadeh M, Zhou D. Improving Quality of Care in Virtual Visits: Heuristic Evaluations of the User Interface in Telehealth. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 65. Los Angeles: SAGE Publications; 2021. p. 1347–51.
47. Carrard V Roxo Gonçalves M Rodriguez Strey J Pilz C Martins M Martins M Telediagnosis of oral lesions in primary care: The EstomatoNet Program Oral Dis 2018 24 6 1012 1019 10.1111/odi.12851 29505701
Carrard V, Roxo Gonçalves M, Rodriguez Strey J, Pilz C, Martins M, Martins M, et al. Telediagnosis of oral lesions in primary care: The EstomatoNet Program. Oral Dis. 2018;24(6):1012–9.29505701
