Continuous Sign Language Translation on the New Educational Turkish Sign Language Dataset (E-TSL) Using Neural Machine Translation Methods

Abstract

Sign language is a fundamental communication tool for people with hearing and speech disabilities. However, it is not widely known by others, making communication challenging for those who rely on it. Additionally, sign language varies by country and evolves over time, which further complicates communication among sign language users. To address these challenges, recent technological advances have led to numerous studies on sign language processing. These studies focus on sign language translation (sign to text) and production (text to sign). As with most deep learning models, large datasets are essential. Widely used datasets include Phoenix-2014, Phoenix-2014T, SWISSTXT-NEWS, VRT-NEWS, BSL Corpus, and How2Sign ASL. Turkish Sign Language (TSL) datasets are also available, but they are typically isolated. Currently, there is no Turkish Sign Language dataset suitable for continuous sign language translation.

In this thesis, we created the Educational Turkish Sign Language (E-TSL) dataset, featuring Turkish secondary school courses, to promote continuous sign language translation methods for TSL. The dataset includes 1,410 video clips with 11 different signers, totaling nearly 24 hours of content. Due to the agglutinative nature of Turkish, sign language translation faces additional challenges. After lemmatizing the words, the E-TSL dataset's dictionary shows that 64\% of the words are singletons, and 85\% are rare words appearing less than five times, posing significant challenges for translation.

To address these challenges, we developed transformer-based pose-to-text (P2T-T) and graph neural network-based transformer (GNN-T) models. Despite the dataset's complexity, our GNN-T model achieved ROUGE-L, BLEU-1, and BLEU-4 scores of 22.93, 21.01, and 3.49, respectively. These results highlight the difficulty of the E-TSL dataset compared to others. To validate our models, we used the Phoenix-Weather 2014T dataset as a benchmark, providing comparative results. Finally, we evaluated the performance of our E-TSL dataset against other commonly used datasets.