Back to search

National Technical University of Athens (NTUA)

Οπτικοακουστική μοντελοποίηση προσοχής και πολυ-αισθητηριακή κατανόηση ακουστικής σκηνής

Abstract

dc:description

The scope of this work is the investigation and development of a 2D computational audiovisual saliency model based on behavioral findings that will be able to produce accurate human fixation predictions in a 2D audiovisual scene, i.e. in videos. The investigation is carried out with two different ways: with signal processing techniques and with deep learning techniques. Regarding the former, several fusion schemes between visual and auditory saliency models are being investigated, and the resulting models are behaviorally validated through comparisons with results from behavioral experiments and evaluated with audiovisual human eye-tracking data and fMRI data. Results from both behavioral and eye-tracking experiments indicate that audiovisual saliency modeling indeed improves saliency estimation performance. Regarding deep learning techniques, a new spatio-temporal audiovisual saliency network is developed, that includes a visual saliency network, an audio representation network, a sound localization module, and an audiovisual saliency fusion module. All modules are integrated under a single network that is trained end-to-end. The network performance is evaluated in several eye-tracking databases and results from comparisons with other methods highlight the effectiveness of the presented network, that opens the way for estimating saliency ``in-the-wild". In parallel, research has been carried out in the direction of auditory scene understanding. Specifically, a speaker localization system has been developed, as well as a baseline distant speech recognition system in Greek and English and a speech understanding/dialog system. These systems have been adapted and applied either to a smart home environment and/or to a multi-sensory human/child - robot interaction application. They are also evaluated through experiments in appropriate databases. Finally, except for the development and evaluation of new algorithmic methods to successfully address the above problems, an important contribution of this thesis lies in the collection of two new databases: An audiovisual human eye-tracking database employing 20 subjects and 37 videos has been collected, as well as a multi-channel speech database in Greek with data from 20 speakers.

Degree

thesis:*
Grantor dc:publisher
National Technical University of Athens (NTUA)
Year dc:date
2019

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Tsiami, Antigoni
  • Τσιάμη, Αντιγόνη

Subjects

dc:subject × 10

Rights

Language dc:language
gre

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/47133

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Tsiami, Antigoni; Τσιάμη, Αντιγόνη. Οπτικοακουστική μοντελοποίηση προσοχής και πολυ-αισθητηριακή κατανόηση ακουστικής σκηνής. National Technical University of Athens (NTUA), 2019. http://hdl.handle.net/10442/hedi/47133