Development of an Artificial Intelligence-Based Diagnostic System Using Dermoscopic Images and Evaluation of the Diagnostic System's Place in Dermatology Residency

Abstract

Introduction: Artificial intelligence is defined as endowing machines with abilities such as learning, problem solving, deriving meaning, and remembering. Today, artificial intelligence has rapidly developed image processing capabilities. In dermatology, clinicians diagnose conditions using dermoscopy, a method that involves analyzing specially taken images of lesions. There are studies on the analysis of dermoscopic images with artificial intelligence. In some studies, artificial intelligence has been found to be more successful than experienced specialist doctors in diagnosing dermoscopic images. However, our literature search to date has not found a study evaluating the position of artificial intelligence's successful diagnostic capability, especially in dermatology specialist training. The aim of this study is to demonstrate the contribution of an artificial intelligence trained with dermoscopic images reflecting the characteristics of our own patient group to the existing skills of residents.

Methods and Materials: Dermoscopic photos taken for diagnosis and follow up purposes at our hospital between 2013-2023 were evaluated. A dataset consisting of 4,220 images diagnosed with basal cell carcinoma, squamous cell carcinoma, melanoma, dysplastic nevus, melanocytic nevus, benign keratoses, actinic keratosis, dermatofibroma, and vascular lesions was prepared. The ISIC 19 open dataset from the literature was added to this dataset. An image processing artificial intelligence algorithm was developed. The dataset obtained was used for the training and testing of the algorithm. A web application was designed to investigate the effect of artificial intelligence on the diagnostic accuracy of residents. The study included n=17 research assistants who had received at least one year of dermoscopy training in our unit's academic training schedule and had at least one year of clinical experience in dermoscopy. Participants were asked a total of n=54 dermoscopic image diagnosis questions through the our application, first answering themselves and then with the support of artificial intelligence. Data was recorded through the application and the interaction between dermatology assistants and artificial intelligence was analyzed.

Findings: The dataset was created using n=24,731 (85%) dermoscopic photos from ISIC19 and n=4,220 (15%) dermoscopic photos from patients at Hacettepe University (HU), totaling n=28,951 photos. The artificial intelligence was tested with n=5,910 photos. It was found that the artificial intelligence achieved a diagnostic accuracy of 0.91 (91%). The balanced accuracy rate of the algorithm was calculated as 0.78 (78%) and the F1 score as 0.80. According to a 54-question participant evaluation test prepared from the test set, the change in accuracy rates of the participants (n=17) with artificial intelligence support was calculated as 0.13 (p<0.001). With artificial intelligence support, the sensitivity increase ranged from a minimum of 0.01 to a maximum of 0.24 across all diagnoses. The highest improvement in sensitivity was observed in basal cell carcinoma with 0.24 (p=0.001). This was followed by melanocytic nevus with 0.20 (p=0.001) and squamous cell carcinoma with 0.19 (p=0.002). The lowest improvement in sensitivity value was calculated as 0.01 (p=0.773) for the diagnosis of dysplastic nevus. In the subgroup analysis according to experience, first-year research assistants had the highest change in accuracy rates with artificial intelligence support, increasing by 0.18 (p=0.068). For second-year research assistants, the increase in accuracy rate with artificial intelligence support was 0.10 (p=0.144); for the third year, it was 0.12 (p=0.068), and for the fourth year, the change was 0.11 (p=0.042). No significant relationship was found between the increase in accuracy rate provided by artificial intelligence support and the research assistants' years of experience.

Conclusion: Our study has shown that artificial intelligence using dermoscopic images increased the diagnostic accuracy of dermatology residents. Besides, artificial intelligence support has been observed to increase sensitivity in malignant diagnoses, including melanoma and especially non-melanoma skin cancers.