National Technical University of Athens (NTUA)
Οπτικοακουστική μοντελοποίηση προσοχής και πολυ-αισθητηριακή κατανόηση ακουστικής σκηνής
Abstract
dc:descriptionThe scope of this work is the investigation and development of a 2D computational audiovisual saliency model based on behavioral findings that will be able to produce accurate human fixation predictions in a 2D audiovisual scene, i.e. in videos. The investigation is carried out with two different ways: with signal processing techniques and with deep learning techniques. Regarding the former, several fusion schemes between visual and auditory saliency models are being investigated, and the resulting models are behaviorally validated through comparisons with results from behavioral experiments and evaluated with audiovisual human eye-tracking data and fMRI data. Results from both behavioral and eye-tracking experiments indicate that audiovisual saliency modeling indeed improves saliency estimation performance. Regarding deep learning techniques, a new spatio-temporal audiovisual saliency network is developed, that includes a visual saliency network, an audio representation network, a sound localization module, and an audiovisual saliency fusion module. All modules are integrated under a single network that is trained end-to-end. The network performance is evaluated in several eye-tracking databases and results from comparisons with other methods highlight the effectiveness of the presented network, that opens the way for estimating saliency ``in-the-wild". In parallel, research has been carried out in the direction of auditory scene understanding. Specifically, a speaker localization system has been developed, as well as a baseline distant speech recognition system in Greek and English and a speech understanding/dialog system. These systems have been adapted and applied either to a smart home environment and/or to a multi-sensory human/child - robot interaction application. They are also evaluated through experiments in appropriate databases. Finally, except for the development and evaluation of new algorithmic methods to successfully address the above problems, an important contribution of this thesis lies in the collection of two new databases: An audiovisual human eye-tracking database employing 20 subjects and 37 videos has been collected, as well as a multi-channel speech database in Greek with data from 20 speakers.
Degree
thesis:*- Grantor dc:publisher
- National Technical University of Athens (NTUA)
- Year dc:date
- 2019
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Tsiami, Antigoni
- Τσιάμη, Αντιγόνη
Subjects
dc:subject × 10- Οπτικοακουστική προσοχή
- Αναγνώριση φωνής
- Μηχανική μάθηση
- Audiovisual saliency
- Speech recognition
- Machine learning
- Επιστήμες Μηχανικού και Τεχνολογία
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Engineering and Technology
- Electrical Engineering, Electronic Engineering, Information Engineering
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/47133