A experiência com autoria de material didático multimídia para propósitos educacionais mostra um grande problema: como prover uma maneira de tratar objetos multimídia de modo que usuários inexperientes (como professores) possam estar aptos a projetar e construir suas próprias apresentações? A criação de tais apresentações envolve fatores como armazenamento, entrega, busca e apresentação de material multimídia (vídeo em especial). Uma infra-estrutura básica que armazene e entregue eficientemente os dados de vídeo é necessária, porém, outro ponto importante é organizar esses dados armazenados no servidor de forma a facilitar seu acesso por parte dos usuários. Neste trabalho, isto é alcançado através do uso de um sistema interativo de recuperação e gerenciamento de informações projetado para facilitar o acesso a itens (ou parte deles) armazenados no servidor. A principal característica de tal sistema é o uso de uma base de metadados contendo os atributos dos vídeos armazenados no servidor. Buscas podem ser feitas por título, assunto, tamanho, autor, conteúdo ou, mais importante no caso de material didático, por cenas ou frames específicos. O sistema foi implementado segundo uma abordagem cliente/servidor utilizando a linguagem de programação JAVA. A comunicação entre clientes e servidores é realizada através do uso do Visibroker 3.0, que é uma ferramenta de programação para Objetos Distribuídos segundo o padrão CORBA. O acesso aos dados a partir da base de metadados é realizado através do uso de um driver PostgreSQL que segue a API JDBC. Para propósitos de avaliação do sistema um player foi construído utilizando a ferramenta Java Media Framework (JMF). Foi realizada uma análise para a verificação do impacto da utilização das tecnologias CORBA e JDBC no sistema. Foi detectado que a utilização da tecnologia JDBC impõe um atraso muito mais significante que a utilização da tecnologia CORBA. Outra conclusão é que a utilização de metadados provê uma melhor interatividade em buscas, permite economia de tempo durante o processo de edição e provê economia de espaço de armazenamento através do compartilhamento de objetos como vídeos, cenas e frames. / The experience with authoring multimedia material for educational purposes shows a major problem: how to provide an easy and efficient way to handle multimedia objects in a manner that non-expert users (namely school teachers) can be able to design and build their own presentations? The creation of this presentations involves factors like storage, delivery, search and presentation of multimedia material (video in special). A basic infra-structure that stores and efficiently deliver the video data is needed. However, another important point is the organization of these data stored into the server in a way to facilitate the access to them from the users. In the system wich is the subject of this work, this is achived through the use of an interactive information management and retrieval system designed to facilitate the access to items (or parts of the items) stored in the server. The main characteristic of the system is the use of a metadata base which contains attributes of the videos stored in the server. Searches can be made by title, subject, length, author, content or, most important in the didatic multimedia material case, by a specific scene or frame. The system was built with JAVA programming language in a client/server way. The communication between clients and servers is realized through the use of the Visibroker 3.0, which is a Distributed Objects programming tool according to the CORBA standard. The data access from the metadata base use a PostgreSQL driver which follows the JDBC API. For evaluation purposes a playback tool was built using Java Media Framework (JMF). An analisys was carried out to verify the impact of the utilization of CORBA and JDBC technologies in the system. It was detected that JDBC technology utilization imposes a much more significate delay than the CORBA technology utilization. Another conclusion is that metadata utilization provide better interactivity searches, making the editing process faster and save storage space through the sharing of objects like videos, scenes and frames.

A experiência com autoria de material didático multimídia para propósitos educacionais mostra um grande problema: como prover uma maneira de tratar objetos multimídia de modo que usuários inexperientes (como professores) possam estar aptos a projetar e construir suas próprias apresentações? A criação de tais apresentações envolve fatores como armazenamento, entrega, busca e apresentação de material multimídia (vídeo em especial). Uma infra-estrutura básica que armazene e entregue eficientemente os dados de vídeo é necessária, porém, outro ponto importante é organizar esses dados armazenados no servidor de forma a facilitar seu acesso por parte dos usuários. Neste trabalho, isto é alcançado através do uso de um sistema interativo de recuperação e gerenciamento de informações projetado para facilitar o acesso a itens (ou parte deles) armazenados no servidor. A principal característica de tal sistema é o uso de uma base de metadados contendo os atributos dos vídeos armazenados no servidor. Buscas podem ser feitas por título, assunto, tamanho, autor, conteúdo ou, mais importante no caso de material didático, por cenas ou frames específicos. O sistema foi implementado segundo uma abordagem cliente/servidor utilizando a linguagem de programação JAVA. A comunicação entre clientes e servidores é realizada através do uso do Visibroker 3.0, que é uma ferramenta de programação para Objetos Distribuídos segundo o padrão CORBA. O acesso aos dados a partir da base de metadados é realizado através do uso de um driver PostgreSQL que segue a API JDBC. Para propósitos de avaliação do sistema um player foi construído utilizando a ferramenta Java Media Framework (JMF). Foi realizada uma análise para a verificação do impacto da utilização das tecnologias CORBA e JDBC no sistema. Foi detectado que a utilização da tecnologia JDBC impõe um atraso muito mais significante que a utilização da tecnologia CORBA. Outra conclusão é que a utilização de metadados provê uma melhor interatividade em buscas, permite economia de tempo durante o processo de edição e provê economia de espaço de armazenamento através do compartilhamento de objetos como vídeos, cenas e frames. / The experience with authoring multimedia material for educational purposes shows a major problem: how to provide an easy and efficient way to handle multimedia objects in a manner that non-expert users (namely school teachers) can be able to design and build their own presentations? The creation of this presentations involves factors like storage, delivery, search and presentation of multimedia material (video in special). A basic infra-structure that stores and efficiently deliver the video data is needed. However, another important point is the organization of these data stored into the server in a way to facilitate the access to them from the users. In the system wich is the subject of this work, this is achived through the use of an interactive information management and retrieval system designed to facilitate the access to items (or parts of the items) stored in the server. The main characteristic of the system is the use of a metadata base which contains attributes of the videos stored in the server. Searches can be made by title, subject, length, author, content or, most important in the didatic multimedia material case, by a specific scene or frame. The system was built with JAVA programming language in a client/server way. The communication between clients and servers is realized through the use of the Visibroker 3.0, which is a Distributed Objects programming tool according to the CORBA standard. The data access from the metadata base use a PostgreSQL driver which follows the JDBC API. For evaluation purposes a playback tool was built using Java Media Framework (JMF). An analisys was carried out to verify the impact of the utilization of CORBA and JDBC technologies in the system. It was detected that JDBC technology utilization imposes a much more significate delay than the CORBA technology utilization. Another conclusion is that metadata utilization provide better interactivity searches, making the editing process faster and save storage space through the sharing of objects like videos, scenes and frames.

[pt] A quantidade de vídeos gravados no mundo cresce muito, não somente devido aos interesses e hábitos humanos em relação a esse tipo de mídia, mas também pela diversidade de dispositivos utilizados para criação de vídeos. No entanto, faltam informações sobre conteúdos em vídeo porque a geração de metadados é complexa e requer muito tempo para ser executado por humanos. Do ponto de vista da tecnologia, não é fácil superar os obstáculos relacionados à grande quantidade e diversidade de frames de vídeo. O trabalho propõe um sistema automatizado de reconhecimento facial para detectar personagens em vídeos. Ele foi desenvolvido para reconhecer personagens, a fim de aumentar os metadados de vídeo. Ele combina técnicas padrão de visão computacional para melhorar a precisão, processando os dados de saída dos modelos existentes de maneira complementar. O modelo teve um desempenho satisfatório usando um conjunto de dados da vida real de uma grande empresa de mídia. / [en] The amount of recorded video in the world is increasing a lot due not only to the humans interests and habits regarding this kind of media, but also the diversity of devices used to create them. However, there is a lack of information about video content because generating video meta-data is complex. It demands too much time to be performed by humans, and from the technology perspective, it is not easy to overcome obstacles regarding the huge amount and diversity of video frames. In this work we propose an automated face recognition system to detect and recognize humans within videos. It was developed to recognize characters,in order to increase video meta-data. It combines standard computer vision techniques to improved accuracy by processing existing models output data in a complementary manner. We evaluated the performance of the system in a real data set from a large media company.

Im Vergleich zu einer stichwortbasierten Suche ermöglicht die semantische Suche ein präziseres und anspruchsvolleres Durchsuchen von (Web)-Dokumenten, weil durch die explizite Semantik Mehrdeutigkeiten von natürlicher Sprache vermieden und semantische Beziehungen in das Suchergebnis einbezogen werden können. Eine semantische, Entitäten-basierte Suche geht von einer Anfrage mit festgelegter Bedeutung aus und liefert nur Dokumente, die mit dieser Entität annotiert sind als Suchergebnis. Die wichtigste Voraussetzung für eine Entitäten-zentrierte Suche stellt die Annotation der Dokumente im Archiv mit Entitäten und Kategorien dar. Textuelle Informationen werden analysiert und mit den entsprechenden Entitäten und Kategorien versehen, um den Inhalt semantisch erschließen zu können. Eine manuelle Annotation erfordert Domänenwissen und ist sehr zeitaufwendig. Die semantische Annotation von Videodokumenten erfordert besondere Aufmerksamkeit, da inhaltsbasierte Metadaten von Videos aus verschiedenen Quellen stammen, verschiedene Eigenschaften und Zuverlässigkeiten besitzen und daher nicht wie Fließtext behandelt werden können. Die vorliegende Arbeit stellt einen semantischen Analyseprozess für Video-Metadaten vor. Die Eigenschaften der verschiedenen Metadatentypen werden analysiert und ein Konfidenzwert ermittelt. Dieser Wert spiegelt die Korrektheit und die wahrscheinliche Mehrdeutigkeit eines Metadatums wieder. Beginnend mit dem Metadatum mit dem höchsten Konfidenzwert wird der Analyseprozess innerhalb eines Kontexts in absteigender Reihenfolge des Konfidenzwerts durchgeführt. Die bereits analysierten Metadaten dienen als Referenzpunkt für die weiteren Analysen. So kann eine möglichst korrekte Analyse der heterogen strukturierten Daten eines Kontexts sichergestellt werden. Am Ende der Analyse eines Metadatums wird die für den Kontext relevanteste Entität aus einer Liste von Kandidaten identifiziert - das Metadatum wird disambiguiert. Hierfür wurden verschiedene Disambiguierungsalgorithmen entwickelt, die Beschreibungstexte und semantische Beziehungen der Entitätenkandidaten zum gegebenen Kontext in Betracht ziehen. Der Kontext für die Disambiguierung wird für jedes Metadatum anhand der Eigenschaften und Konfidenzwerte zusammengestellt. Der vorgestellte Analyseprozess ist an zwei Hypothesen angelehnt: Um die Analyseergebnisse verbessern zu können, sollten die Metadaten eines Kontexts in absteigender Reihenfolge ihres Konfidenzwertes verarbeitet werden und die Kontextgrenzen von Videometadaten sollten durch Segmentgrenzen definiert werden, um möglichst Kontexte mit kohärentem Inhalt zu erhalten. Durch ausführliche Evaluationen konnten die gestellten Hypothesen bestätigt werden. Der Analyseprozess wurden gegen mehrere State-of-the-Art Methoden verglichen und erzielt verbesserte Ergebnisse in Bezug auf Recall und Precision, besonders für Metadaten, die aus weniger zuverlässigen Quellen stammen. Der Analyseprozess ist Teil eines Videoanalyse-Frameworks und wurde bereits erfolgreich in verschiedenen Projekten eingesetzt. / The Semantic Web provides information contained in the World Wide Web as machine-readable facts. In comparison to a keyword-based inquiry, semantic search enables a more sophisticated exploration of web documents. By clarifying the meaning behind entities, search results are more precise and the semantics simultaneously enable an exploration of semantic relationships. However, unlike keyword searches, a semantic entity-focused search requires that web documents are annotated with semantic representations of common words and named entities. Manual semantic annotation of (web) documents is time-consuming; in response, automatic annotation services have emerged in recent years. These annotation services take continuous text as input, detect important key terms and named entities and annotate them with semantic entities contained in widely used semantic knowledge bases, such as Freebase or DBpedia. Metadata of video documents require special attention. Semantic analysis approaches for continuous text cannot be applied, because information of a context in video documents originates from multiple sources possessing different reliabilities and characteristics. This thesis presents a semantic analysis approach consisting of a context model and a disambiguation algorithm for video metadata. The context model takes into account the characteristics of video metadata and derives a confidence value for each metadata item. The confidence value represents the level of correctness and ambiguity of the textual information of the metadata item. The lower the ambiguity and the higher the prospective correctness, the higher the confidence value. The metadata items derived from the video metadata are analyzed in a specific order from high to low confidence level. Previously analyzed metadata are used as reference points in the context for subsequent disambiguation. The contextually most relevant entity is identified by means of descriptive texts and semantic relationships to the context. The context is created dynamically for each metadata item, taking into account the confidence value and other characteristics. The proposed semantic analysis follows two hypotheses: metadata items of a context should be processed in descendent order of their confidence value, and the metadata that pertains to a context should be limited by content-based segmentation boundaries. The evaluation results support the proposed hypotheses and show increased recall and precision for annotated entities, especially for metadata that originates from sources with low reliability. The algorithms have been evaluated against several state-of-the-art annotation approaches. The presented semantic analysis process is integrated into a video analysis framework and has been successfully applied in several projects for the purpose of semantic video exploration of videos.

<p dir="ltr">The rapid development of machine learning techniques makes it possible to manipulate or synthesize video and audio information while introducing nearly indetectable artifacts. Most media forensics methods analyze the high-level data (e.g., pixels from videos, temporal signals from audios) decoded from compressed media data. Since media manipulation or synthesis methods usually aim to improve the quality of such high-level data directly, acquiring forensic evidence from these data has become increasingly challenging. In this work, we focus on media forensics techniques using the metadata in media formats, which includes container metadata and coding parameters in the encoded bitstream. Since many media manipulation and synthesis methods do not attempt to hide metadata traces, it is possible to use them for forensics tasks. First, we present a video forensics technique using metadata embedded in MP4/MOV video containers. Our proposed method achieved high performance in video manipulation detection, source device attribution, social media attribution, and manipulation tool identification on publicly available datasets. Second, we present a transformer neural network based MP3 audio forensics technique using low-level codec information. Our proposed method can localize multiple compressed segments in MP3 files. The localization accuracy of our proposed method is higher compared to other methods. Third, we present an H.264-based video device matching method. This method can determine if the two video sequences are captured by the same device even if the method has never encountered the device. Our proposed method achieved good performance in a three-fold cross validation scheme on a publicly available video forensics dataset containing 35 devices. Fourth, we present a Graph Neural Network (GNN) based approach for the analysis of MP4/MOV metadata trees. The proposed method is trained using Self-Supervised Learning (SSL), which increased the robustness of the proposed method and makes it capable of handling missing/unseen data. Fifth, we present an efficient approach to compute the spectrogram feature with MP3 compressed audio signals. The proposed approach decreases the complexity of speech feature computation by ~77.6% and saves ~37.87% of MP3 decoding time. The resulting spectrogram features lead to higher synthetic speech detection performance.</p>