National and Kapodistrian University of Athens
Πολυτροπική ολοκλήρωση υψηλού επιπέδου και εξαγωγή σημασιολογίας
Abstract
dc:descriptionThis thesis on multimodal fusion and semantics extraction, focuses on automated detection and annotation of harmful content in video data. The aim is not only to reason out the existence of violence or not (i.e., the binary problem), but also to determine the type of violence (e.g., fight, explosion, murder). An introductory comparative study of several harmful content detection approaches is first presented and the lack of knowledge representation and reasoning approaches for the problem at hand is acknowledged. Towards filling this gap, we propose a semantic fusion approach that combines low to mid level modality specific semantics through ontological and rule reasoning. A major part of the proposed framework is the movie segmentation into meaningful and easy to handle units. Towards this direction, we investigate and evaluate a set of shot boundary detection approaches and their combination through a majority voting scheme. In the sequel, state of the art classification methods are employed to extract mid level semantics. The segmentation module and the modality specific analysis algorithms instantiate the corresponding video structure and modality specific ontologies developed in the context of the knowledge engineering framework. A set of consecutive and interleaved ontological and SWRL rule reasoning steps map sets and sequences of extracted semantics into higher level concepts represented in the harmful content ontology. We present the involved ontologies, the corresponding SWRL rule sets and the reasoning mechanism in detail. A comparative study of the most common ontological reasoners for the realization procedure of the semantic framework and the classification of the harmful content ontology is presented in detail. In the final part of the thesis we evaluate the proposed approach in a preanotated movie dataset and compare its results with the single modality approaches and a kNN late fusion classifier. We comment on the higher level semantics extraction ability and evaluate a set of extensions employed in the basic structure of the framework. The extensions concern the development of a scene detection module that combines Markov clustering with SQWRL queries and the incorporation of existing rating and movie genre metadata in the reasoning process.
Degree
thesis:*- Grantor dc:publisher
- National and Kapodistrian University of Athens
- Year dc:date
- 2011
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Perperis, Thanasis
- Περπερής, Αθανάσιος
Subjects
dc:subject × 14- Σημασιολογική ανάλυση βίντεο
- Ανάκτηση με βάση το περιεχόμενο
- Αναγνώριση επιβλαβούς περιεχομένου
- Αναπαράσταση γνώσης
- Συλλογιστική
- Semantic video analysis
- Harmful content detection
- Rules
- Knowledge engineering
- Reasoning
- Natural Sciences
- Computer and Information Sciences
- Φυσικές Επιστήμες
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/26721