Aristotle University Of Thessaloniki (AUTH)
Συστήματα επεξεργασίας οπτικοακουστικών πόρων
Abstract
dc:descriptionThis work introduces and evaluates new methodologies for audiovisual content classification and retrieval, in media asset management systems. The proposed methodologies focus on the unimodal and multimodal semantic analysis of audiovisual content with emphasis on the employment of machine learning techniques. A new media asset management model, supporting the incorporation of semantic analysis algorithms on the basis of a hierarchical architecture, is introduced. Additionally, a collaborative annotation method based on the user social interaction is introduced, aiming at the development of large-scale, annotated ground truth datasets. In the context of the proposed hierarchical architecture, a number of novel audio-driven content analysis methodologies are presented and evaluated. A new method for audiovisual content segmentation based on speech / music discrimination is developed. The proposed algorithm relies on the hypothesis that short-time frame-level discrimination performance can be enhanced by identifying transition points between longer, semantically homogeneous segments of audio. As part of it, a two-step segmentation approach is employed to identify transition points between the homogeneous regions and subsequently classify the derived segments using a supervised binary classifier. The algorithm is evaluated against three publicly available datasets and experimental results are presented in comparison with existing state-of-the-art implementations. Furthermore, results from the MIREX 2015 competition, where the proposed system won the third place, are presented and discussed. Afterwards, a music structure segmentation algorithm, relying on the detection of transition points between repeated sections of content, is introduced. Vector-quantization is employed as a novel adaptive filtering mechanism for the extraction of the repeated time-series data. The method is evaluated against 124 pop songs from the INRIA Eurovision dataset and in comparison with an existing state-of-the-art music structure analysis algorithm. Finally, a multimodal speaker diarization approach, based on deep neural network embeddings, is proposed. A novel Long Short-term Memory, Siamese neural network is employed for the analysis of the audio mode, whereas a pre-trained convolutional neural network is deployed for the analysis of the visual modality. A multimodal fusion stage is introduced to enhance audio-driven speaker diarization performance by exploiting information derived from the visual input. The proposed method is assessed against two publicly available datasets and results are presented in comparison with an existing state-of-the-art algorithm.
Degree
thesis:*- Grantor dc:publisher
- Aristotle University Of Thessaloniki (AUTH)
- Year dc:date
- 2018
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Tsipas, Nikolaos
- Τσίπας, Νικόλαος
Subjects
dc:subject × 20- Σημασιολογική ανάλυση οπτικοακουστικού περιεχομένου
- Βαθιά μάθηση
- Συνεργατική επισημείωση περιεχομένου
- Διάκριση ομιλίας / μουσικής
- Κατάτμηση μουσικού περιεχομένου
- Πολυτροπική δεικτοδότηση ομιλητών
- Semantic analysis of audiovisual content
- Deep learning
- Collaborative annotation
- Speech / music discrimination
- Music structure segmentation
- Multimodal speaker diarization
- Φυσικές Επιστήμες
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Επιστήμες Μηχανικού και Τεχνολογία
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Natural Sciences
- Computer and Information Sciences
- Engineering and Technology
- Electrical Engineering, Electronic Engineering, Information Engineering
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/44825