University of Thessaly (UTH)
Automatic sign language recognition algorithms and their incorporation into a greek sign language educational platform
Abstract
dc:descriptionThis Ph.D. Thesis focuses on the problem of automatic Sign Language Recognition (SLR)from video data, with a use-case of learning Greek Sign Language (GSL) through a properly designed platform with integrated functionality of automatic GSL recognition. The SLR problem is complex and challenging, due to the multitude of articulators, the complexity of its formation, the variation of the articulation process among signers, the difficulty of robust visual detection and tracking in a variety of non-ideal environments with often low-quality camera sensors, as well as the lack of large-volume SLR databases. At the same time, the process of learning sign language as a “2nd language” requires a very significant effort and student-teacher interaction both during learning as well as objective assessment of students by their teachers, both lacking appropriate supporting tools. The main innovation and contribution of the Thesis lies in the combination of these two problems, i.e., both in the development of innovative and effective algorithms for SLR and their successful integration into a GSL learning platform, demonstrating success both through the evaluation of recognition on various SLR databases, as well as the evaluation of the learning platform objectively and subjectively by a large number of learner users. Specifically, in this Thesis: • The largest GSL database to date has been collected (at least 3 times longer in duration and number of interpreters than the largest database previously), allowing the training and evaluation of GSL models and their integration into the educational platform, while also made available to the scientific community to further advance the field. • Numerous algorithms have been investigated for visual detection and tracking of signing articulators and then for extracting visual features from them. Further, it was determined which ones are most suitable for SLR, through experiments on isolated sign recognition. Many of these algorithms have not been used before for this particular problem. • Innovative algorithms were developed for SLR, with a primary emphasis on sequential learning architectures. These were used to recognize isolated signs, continuous signing, continuous fingerspelling, and finally “cued speech”. These techniques were evaluated on multiple databases, including the aforementioned GSL one. In many cases the algorithms advanced the state-of-the-art. • Finally, computationally efficient models for GSL recognition (specifically for isolated signing, continuous signing, and continuous fingerspelling) were developed, which were integrated into the GSL educational platform, further aided by appropriate GSL language models. The system was evaluated by a large number of users (GSL students at University of Thessaly and GSL experts), achieving excellent subjective and objective results, the latter being in perfect agreement with the off-line SLR results.
Degree
thesis:*- Grantor dc:publisher
- University of Thessaly (UTH)
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Papadimitriou, Aikaterini
- Παπαδημητρίου, Αικατερίνη
Subjects
dc:subject × 22- Αναγνώριση μεμονωμένων νοημάτων
- Αναγνώριση συνεχούς νοηματισμού
- Αναγνώριση συνεχούς δακτυλοσυλλαβισμού
- Αναγνώριση συνεπαγομένης ομιλίας
- Οπτική ανίχνευση και παρακολούθηση
- Εξαγωγή χαρακτηριστικών
- Ακολουθιακή μάθηση
- Αναγνώριση Ελληνικής νοηματικής γλώσσας
- Ιsolated sign language recognition
- Continuous sign language recognition
- Continuous fingerspelling recognition
- Cued speech recognition
- Visual detection and tracking
- Feature extraction
- Sequence learning
- Greek sign language recognition
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Επιστήμες Μηχανικού και Τεχνολογία
- Ρομποτική
- Electrical Engineering, Electronic Engineering, Information Engineering
- Engineering and Technology
- Robotics
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/56096