Tese de mestrado. Gestão de Informação. 2004. Faculdade de Engenharia. Universidade do Porto

Cette thèse relève de l'indexation et de la représentation par le contenu des données multimédias, dans le contexte spécifique de la normalisationinternationale de l'image et notamment dans le cadre du processus de standardisation MPEG-7.<br /><br />Nous proposons tout d'abord de nouvelles mesures de similarité pour les descriptions de mouvement par modèles paramétriques 2D, fondées sur une famille de fonctions distance entre les champs de vitesses. Les problèmes d'optimisation en temps de calcul, d'alignement spatio-temporel et de pondération des composantes translationnelle et homogène de mouvement sont analysés et une solution mathématique proposée, mise en oeuvre et évaluée objectivement sur les bases de test naturelle et synthétique avec vérité terrain, que nous avons constituées et qui ont été retenues pour l'évaluation des descripteurs de mouvement MPEG-7.<br /><br />Deux descripteurs de forme pour indexer des modèles 3D maillés sont ensuite proposés. Partant des propriétés d'invariance géométrique et topologique que doit satisfaire naturellement un descripteur de forme d'objet 3D maillé, nous avons tout d'abord défini le spectre de forme 3D, que nous avons proposé et promu dans le standard MPEG-7. Celui-ci exploite uniquement la structure géométrique locale d'une surface 3D, fournit une représentation très compacte, mais présente une grande sensibilité aux descriptions topologiques des maillages. En considérant la transformée de Hough 3D d'un maillage, nous avons ensuite construit le descripteur de Hough 3D optimisé, intrinsèquement invariant aux changements de connexité, rendu indépendant aux transformations géométriques et optimisé en terme de compacité de représentation, via une partition invariante aux changements de repère canonique de la sphère unité.<br /><br />Enfin, nous proposons une plate-forme d'indexation compatible MPEG-7, intégrant des outils d'annotation, de navigation, de visualisation et de requêtes par similarité, et supportant des applications comme l'archivage vidéo, la vidéo cliquable ou l'indexation MPEG-7 de la langue des signes française. L'ensemble logiciel ainsi réalisé démontre pour la première fois en grandeur réelle, pour les applications d'indexation multimédia considérées, le caractère effectivement opérationnel des schémas de description génériques, normalisés MPEG7.

Digital multimedia content production and the amount of content present all over the world have exploded in the recent years. The consequences of this fact can be observed everywhere in many different forms, to exemplify, huge digital video archives of broadcasting companies, commercial image archives, virtual museums, etc. In order for these sources to be useful and accessible, this technological advance must be accompanied by the effective techniques of indexing and retrieval. The most effective way of indexing is the one providing a basis for retrieval in terms of semantic concepts, upon which ordinary users of multimedia databases base their queries. On the other hand, semantic classification of images using low-level features is a challenging problem. Combining experts with different classifier structures, trained by MPEG-7low-level color and texture descriptors, is examined as a solution alternative. For combining different classifiers and features, advanced decision mechanisms are proposed, which utilize basic expert combination strategies in different settings. Each of these decision mechanisms, namely Single Feature Combination (SFC), Multiple Feature Direct Combination (MFDC), and Multiple Feature Cascaded Combination (MFCC) enjoy significant classification performance improvements over single experts. Simulations are conducted on eight different visual semantic classes, resulting in accuracy improvements between 3.5-6.5%, when they are compared with the best performance of single expert systems.

In today’s fast paced world, as the number of stations of television programming offered is increasing rapidly, time accessible to watch them remains same or decreasing. Sports videos are typically lengthy and they appeal to a massive crowd. Though sports video is lengthy, most of the viewer’s desire to watch specific segments of the video which are fascinating, like a home-run in a baseball or goal in soccer i.e., users prefer to watch highlights to save time. When associated to the entire span of the video, these segments form only a minor share. Hence these videos need to be summarized for effective presentation and data management. This thesis explores the ability to extract highlights automatically using MPEG-7 features and hidden Markov model (HMM), so that viewing time can be reduced. Video is first segmented into scene shots, in which the detection of the shot is the fundamental task. After the video is segmented into shots, extraction of key frames allows a suitable representation of the whole shot. Feature extraction is crucial processing step in the classification, video indexing and retrieval system. Frame features such as color, motion, texture, edges are extracted from the key frames. A baseball highlight contains certain types of scene shots and these shots follow a particular transition pattern. The shots are classified as close-up, out-field, base and audience. I first try to identify the type of the shot using low level features extracted from the key frames of each shot. For the identification of the highlight I use the hidden Markov model using the transition pattern of the shots in time domain. Experimental results suggest that with reasonable accuracy highlights can be extracted from the video.

In order to manage the content of multimedia data, the content must be annotated. Although any user-defined annotation is acceptable, it is preferable if systems agree on the same annotation format. MPEG-7 is a widely accepted standard for multimedia content annotation. However, in MPEG-7, semantically identical metadata can be represented in multiple ways due to lack of precise semantics in its XML-based syntax. Unfortunately this prevents metadata interoperability. To overcome this problem, MPEG-7 standard is translated into an ontology. In this thesis, MPEG-7 ontology is used on top and the given user-defined ontologies are attached to the MPEG-7 ontology via a user friendly interface, thus building MPEG-7 based ontologies automatically. Our proposed system is an ontology-based multimedia information management framework due to its modular architecture, ease of integrating with domain specific ontologies naturally and automatic harmonization of MPEG-7 ontology and domain-specific ontologies. Integration with domain specific ontologies is carried out by enabling import of domain ontologies via a user-friendly interface which makes the system independent of application domains.

For some new multimedia applications using Mpeg-4 or Mpeg-7 video coding standards, it is important to find the main objects in a video frame. In this thesis, we propose a neuro-fuzzy modeling approach to the detection of human face and body. Firstly, a fuzzy clustering technique is performed to segment a video frame into clusters to generating several fuzzy rules. Secondly, chrominance and motion features are used to roughly classify the clusters into foreground and background, respectively. Finally, the fuzzy rules are refined by a fuzzy neural network, and the ambiguous regions between foreground and background are further distinguished by the fuzzy neural network. Our method improves the correctness of human face and body detection by getting training data more precisely. Besides, we can extract the VOs correctly even the VOs have no obvious motion in the video sequence.

The value of information often depends on how easy it can be found, retrieved, accessed, filtered and managed. This challenging situation demands a timely solution to the problem. MPEG-7 is the answer to this need. MPEG-7, formally named “Multimedia Content Description Interface”, is a standard for describing the multimedia content data that supports some degree of interpretation of the information’s meaning, which can be passed onto, or accessed by, a device or a computer code. The main objective of this project is to create low complicated, small size, more concrete system, which main features are finding, retrieval, access, filtration and management of broad range of pictures.

The diploma thesis is focused on problematic of management of digital libraries, more precisely it’s focused on standard MPEG-7 ISO/IEC 15938. The thesis defines description of multimedia content, its basic rules and more deeply it describes work with static visual data. The practical part of the thesis is focused on creating standalone application in programming language Java. Its purpose is using XML files to manage, save, search and for general work with visual data and its parameters. An application is focused on Scalable Color and Dominant Color descriptors and propose alternative calculation procedure by variable parameters.

O trabalho apresentado nesta tese trata do desenvolvimento de técnicas com suporte à ciência de contexto, baseadas nos padrões MPEG-4 e MPEG-7, para personalizar e adaptar conteúdo em TV Interativa. Um dos desafios dessa área é desenvolvimento de programas personalizados com rico conteúdo multimídia, com alta interatividade e que, além disso, sejam acessíveis a partir de uma variedade de dispositivos (fixos ou móveis), atendendo às expectativas de interação e de acesso dos usuários. Grande parte do problema está no fato de que os modos encontrados na literatura para representar, descrever e compor programas de TV Interativa não oferecem suporte a contexto, não permitem a separação entre descrições de programas e descrições de objetos e possuem baixa granulosidade de segmentação. Essas características dificultam e, em alguns casos, impedem o desenvolvimento de aplicações avançadas em TV Interativa. As técnicas desenvolvidas neste trabalho são baseadas em esquemas de descrição, compatíveis com o padrão MPEG-7, e na segmentação de programas em objetos MPEG-4. Os esquemas são utilizados para descrever a estrutura, a composição e a semântica de programas e de seus objetos componentes. Também foi definida e implantada uma infra-estrutura para produção, distribuição e consumo de programas. A utilização conjunta da infra-estrutura e das técnicas permite o desenvolvimento de aplicações avançadas em TV Interativa. Como um exemplo dessas aplicações, foi desenvolvido um serviço automático para personalizar e adaptar programas de TV Interativa, permitindo que um usuário possa acessar, sob demanda, programas especialmente produzidos para ele, contendo apenas assuntos de seu interesse e permitindo que o acesso possa ser realizado por dispositivos fixos ou móveis. / The work presented in this thesis developed techniques with context-awareness support, based on the MPEG-4 and MPEG-7 standards, in order to personalize and to adapt Interactive TV content. One of the challenges in this area is the development of personalized programs with rich multimedia content, high interactivity and accessibility through a variety of devices (mobile and non-mobile). Most part of the problem is that the approaches found in literature do not provide context support, do not allow separation between programs and objects descriptions and have low level of segmentation granularity. These features make difficult or impossible, in some cases, the development of Interactive TV applications. The techniques developed in this work are based on MPEG-7 compliant schemes and on programs segmentation into MPEG-4 objects. The schemes are used to describe structure, composition and semantics of programs and component objects. An infra-structure to creation, delivery and consumption of Interactive TV programs was also defined. The joint utilization of infra-structure and techniques allows for the development of Interactive TV advanced applications. As an example of these applications, this work developed an automatic Interactive TV personalization and adaptation service. This service allows a user to access, on-demand, a program specially designed to match his interests and allowing content access through devices with mobile and non-mobile features.

O trabalho apresentado nesta tese trata do desenvolvimento de técnicas com suporte à ciência de contexto, baseadas nos padrões MPEG-4 e MPEG-7, para personalizar e adaptar conteúdo em TV Interativa. Um dos desafios dessa área é desenvolvimento de programas personalizados com rico conteúdo multimídia, com alta interatividade e que, além disso, sejam acessíveis a partir de uma variedade de dispositivos (fixos ou móveis), atendendo às expectativas de interação e de acesso dos usuários. Grande parte do problema está no fato de que os modos encontrados na literatura para representar, descrever e compor programas de TV Interativa não oferecem suporte a contexto, não permitem a separação entre descrições de programas e descrições de objetos e possuem baixa granulosidade de segmentação. Essas características dificultam e, em alguns casos, impedem o desenvolvimento de aplicações avançadas em TV Interativa. As técnicas desenvolvidas neste trabalho são baseadas em esquemas de descrição, compatíveis com o padrão MPEG-7, e na segmentação de programas em objetos MPEG-4. Os esquemas são utilizados para descrever a estrutura, a composição e a semântica de programas e de seus objetos componentes. Também foi definida e implantada uma infra-estrutura para produção, distribuição e consumo de programas. A utilização conjunta da infra-estrutura e das técnicas permite o desenvolvimento de aplicações avançadas em TV Interativa. Como um exemplo dessas aplicações, foi desenvolvido um serviço automático para personalizar e adaptar programas de TV Interativa, permitindo que um usuário possa acessar, sob demanda, programas especialmente produzidos para ele, contendo apenas assuntos de seu interesse e permitindo que o acesso possa ser realizado por dispositivos fixos ou móveis. / The work presented in this thesis developed techniques with context-awareness support, based on the MPEG-4 and MPEG-7 standards, in order to personalize and to adapt Interactive TV content. One of the challenges in this area is the development of personalized programs with rich multimedia content, high interactivity and accessibility through a variety of devices (mobile and non-mobile). Most part of the problem is that the approaches found in literature do not provide context support, do not allow separation between programs and objects descriptions and have low level of segmentation granularity. These features make difficult or impossible, in some cases, the development of Interactive TV applications. The techniques developed in this work are based on MPEG-7 compliant schemes and on programs segmentation into MPEG-4 objects. The schemes are used to describe structure, composition and semantics of programs and component objects. An infra-structure to creation, delivery and consumption of Interactive TV programs was also defined. The joint utilization of infra-structure and techniques allows for the development of Interactive TV advanced applications. As an example of these applications, this work developed an automatic Interactive TV personalization and adaptation service. This service allows a user to access, on-demand, a program specially designed to match his interests and allowing content access through devices with mobile and non-mobile features.