O avanço tecnológico atual está permitindo que as pessoas recebam cada vez mais informações visuais dos mais diferentes tipos, nas mais variadas mídias. Esse aumento fantástico está obrigando os pesquisadores e as indústrias a imaginar soluções para o armazenamento e recuperação deste tipo de informação, pois nossos computadores ainda utilizam, apesar dos grandes avanços nessa área, um sistema de arquivos imaginado há décadas, quando era natural trabalhar com informações meramente textuais. Agora, nos deparamos com novos problemas: Como encontrar uma paisagem específica em um banco de imagens, em que trecho de um filme aparece um cavalo sobre uma colina, em que parte da fotografia existe um gato, como fazer um robô localizar um objeto em uma cena, entre outras necessidades. O objetivo desse trabalho é propor uma arquitetura de rede neural artificial que permita o reconhecimento de objetos genéricos e de categorias em banco de imagens digitais, de forma que se possa recuperar imagens específicas a partir da descrição da cena fornecida pelo usuário. Para que esse objetivo fosse alcançado, foram utilizadas técnicas de Visão Computacional e Processamento de Imagens na etapa de extração de feições de baixo nível e de Redes Neurais(Mapas Auto-Organizáveis de Kohonen) na etapa de agrupamento de classes de objetos. O resultado final desse trabalho pretende ser um embrião para um sistema de reconhecimento de objetos mais genérico, que possa ser estendido para a criação de indices de forma automática ou semi-automática em grandes bancos de imagens. / The current technological progress allows people to receive more and more visual information of the most different types, in different medias. This huge augmentation of image availability forces researchers and industries to propose efficient solutions for image storage and recovery. Despite the extraordinary advances in computational power, the data files system remain the same for decades, when it was natural to deal only with textual information. Nowadays, new problems are in front of us in this field. For instance, how can we find an specific landscape in a image database, in which place of a movie there is a horse on a hill, in which part of a photographic picture there is a cat, how can a robot find an object in a scene, among other queries. The objective of this work is to propose an Artificial Neural Network (ANN) architecture that performs the recognition of generic objects and object’s categories in a digital image database. With this implementation, it becomes possible to do image retrieval through the user´s scene description. To achieve our goal, we have used Computer Vision and Image Processing techniques in low level features extraction and Neural Networks (namely Kohonen’s Self-Organizing Maps) in the phase of object classes clustering. The main result of this work aims to be a seed for a more generic object recognition system, which can be extended to the automatic or semi-automatic index creation in huge image databases.

Orientador: Ricardo da Silva Torres / Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Computação / Made available in DSpace on 2018-08-17T03:09:27Z (GMT). No. of bitstreams: 1 Caceres_SheilaMaricelaPinto_M.pdf: 5019224 bytes, checksum: cf87ae15c741c8322d4d398124abee74 (MD5) Previous issue date: 2010 / Resumo: Um sistema de Recuperação de Imagens por Conteúdo (CBIR) oferece mecanismos necessários para busca e recuperação de imagens baseando-se em propriedades visuais como cor, textura, forma, etc. Em um processo de busca de imagens, a apresentação de resultados é um componente essencial, na medida em que a obtenção desses resultados é o motivo da existência do sistema. Consequentemente, o uso de técnicas de visualização apropriadas pode determinar o sucesso ou o fracasso de um sistema CBIR. Técnicas de visualização são valiosas ferramentas na exploração de grandes quantidades de dados, como coleções de imagens. Contudo, técnicas para visualizar imagens retornadas por sistemas CBIR têm sido pobremente exploradas. Este trabalho apresenta um estudo comparativo e avaliação de várias técnicas de visualização para sistemas CBIR. Como resultado desse estudo, propõe-se um conjunto de técnicas originais que tentam suprir algumas das limitações identificadas em métodos da literatura. Dentre as características das técnicas propostas, destacam-se o enfoque baseado no centro e o uso de técnicas de agrupamento de dados para representar a similaridade intrínseca entre as imagens retornadas. Resultados experimentais mostram que os métodos propostos superam outras estratégias de visualização, considerando-se diversos critérios, como adequação para mostrar resultados em sistemas CBIR, quantidade de informação oferecida, satisfação de usuário, etc. As principais contribuições deste trabalho são: (i) estudo comparativo e análise de sete técnicas de visualização, quatro delas existentes na literatura e três técnicas novas propostas; (ii) avaliação de duas técnicas da literatura nunca antes avaliadas: anéis concêntricos e espiral; (iii) especificação e implementação de três novas técnicas de visualização baseadas em agrupamento; (iv) especificação e implementação de um framework para desenvolvimento de novas estruturas visuais para sistemas CBIR no qual foram implementadas as técnicas de visualização estudadas / Abstract: A Content-Based Image Retrieval (CBIR) system offers mechanisms needed to search and retrieve images based on visual properties such as color, texture, shape, etc. In an image search process, the presentation of results is an essential component as the retrieval of relevant images is the reason of the system existence. Consequently, the use of appropriate visualization techniques may determine the success of a CBIR system. Visualization techniques are valuable tools for the exploration of a great quantity of data, such as images collections. However, techniques for visualizing images in CBIR systems have been poorly explored. This work presents a comparative study of several visualization techniques for CBIR systems. As a result of this study, several original techniques were proposed trying to fulfill some of the absent characteristics in existing methods, such as the central-based focus and the use of clustering approaches to represent the intrinsic similarity between retrieved images. Experimental results show that the proposed methods overcome other visualization strategies by considering several criteria such as adaptation to show CBIR results, information load, user satisfaction, etc. The main contributions of this work are: (i) comparative study and analysis of seven visualization techniques, four of them from the literature and three new ones; (ii) validation of two techniques never evaluated before: concentric rings and spiral; (iii) specification and implementation of three new techniques of visualization based on clustering; (iv) specification and implementation of a framework for developing new visual structures for content-based image retrieval systems. The studied techniques were implemented by using this framework / Mestrado / Ciência da Computação / Mestre em Ciência da Computação

Information retrieval systems are playing an important role in our day to day life for getting the required information. Many text retrieval systems are available and are working successfully. Even though internet is full of other media like images, audio and video, retrieval systems for these media are rare and have not achieved success as that of text retrieval systems. Image retrieval systems are useful in many applications; there is a high demand for effective and efficient tool for image organization and retrieval as per users need. Images are classified into text based image retrieval and content based image retrieval, we proposed a text based image retrieval system, which makes use of ontology to make the retrieval process intelligent. We worked on Cricket World Cup 2007. We combined text based image retrieval approach with content based image retrieval, which uses color and texture as basic low level features. / kvishu223@gmail.com, pravs72@yahoo.co.in.

Die vorliegende Arbeit untersucht das Potenzial von Augmented Reality zur Verbesserung von Image Retrieval Prozessen. Herausforderungen in Design und Gebrauchstauglichkeit wurden für beide Forschungsbereiche dargelegt und genutzt, um Designziele für Konzepte zu entwerfen. Eine Taxonomie für Image Retrieval in Augmented Reality wurde basierend auf der Forschungsarbeit entworfen und eingesetzt, um verwandte Arbeiten und generelle Ideen für Interaktionsmöglichkeiten zu strukturieren. Basierend auf der Taxonomie wurden Anwendungsszenarien als weitere Anforderungen für Konzepte formuliert. Mit Hilfe der generellen Ideen und Anforderungen wurden zwei umfassende Konzepte für Image Retrieval in Augmented Reality ausgearbeitet. Eins der Konzepte wurde auf einer Microsoft HoloLens umgesetzt und in einer Nutzerstudie evaluiert. Die Studie zeigt, dass das Konzept grundsätzlich positiv aufgenommen wurde und bietet Erkenntnisse über unterschiedliches Verhalten im Raum und verschiedene Suchstrategien bei der Durchführung von Image Retrieval in der erweiterten Realität. / The present work investigates the potential of augmented reality for improving the image retrieval process. Design and usability challenges were identified for both fields of research in order to formulate design goals for the development of concepts. A taxonomy for image retrieval within augmented reality was elaborated based on research work and used to structure related work and basic ideas for interaction. Based on the taxonomy, application scenarios were formulated as further requirements for concepts. Using the basic interaction ideas and the requirements, two comprehensive concepts for image retrieval within augmented reality were elaborated. One of the concepts was implemented using a Microsoft HoloLens and evaluated in a user study. The study showed that the concept was rated generally positive by the users and provided insight in different spatial behavior and search strategies when practicing image retrieval in augmented reality.

Der Einsatz von Bildern in den unterschiedlichsten Bereichen ist enorm gestiegen, da Bilder die Fähigkeit haben Erlebnisse, sowie Emotionen zu erzeugen und die Phantasie anzuregen. Zudem lässt die rasante Entwicklung im Multimediabereich die Anzahl der fotografierten und gespeicherten Bilder steigen. Die Suche nach dem „besten Bild“ für z.B. eine Kampagne gestaltet sich schwierig, da die Inhalte mehrerer Bilder zu einem Thema nicht selten eine hohe Ähnlichkeit aufweisen. Die Bilder können sich allerdings sehr deutlich in ihren Low-Level Features, wie Farbton, Sättigung und Helligkeit, unterscheiden. Jedoch ist der Emotional Gap zwischen diesen Low-Level Features und der dahinter steckenden High-Level-Semantik im inhaltsbasierten Image Retrieval nur marginal untersucht. Im Fokus dieser Arbeit steht die Analyse des Einflusses der emotionalen Wirkung eines Bildes auf die Qualität der Image Retrieval Ergebnisse. Dies umfasst zum einen die Untersuchung der von Farbeigenschaften eines Bildes ausgelösten Emotionen, sowie die Evaluation der Ergebnisse einer emotionalen Bildsuche. Durch verschiedene Experimente kann gezeigt werden, dass die Helligkeit und der Farbton die emotionale Wahrnehmung eines Bildes maßgeblich beeinflussen. Anhand der Ergebnisse konnte eine emotionale Annotation von Bildern und somit die Einbindung von Emotionen in den Suchprozess ermöglicht werden. Die anschließende Evaluierung der Suchergebnisse zeigt, dass die Qualität der Ergebnisse des Image Retrievals verbessert werden konnte. / The use of pictures in a variety of areas has increased tremendously in recent years, as they stimulate a person’s imagination and help to create first experiences and emotions. Furthermore, the rapid developments in multimedia have led to an escalation of the number of digitally stored pictures and photographs. Consequently, finding the ‘best picture’ for a convincing advertising campaign has been becoming increasingly difficult due to the abundance of available pictures. To further complicate this search process, a lot of pictures related to a specific topic are very similar with regard to their content. However, their low-level features, such as hue, saturation, and luminance, might differ considerably. Therefore, this work focusses on the influence of emotional characteristics on the image retrieval process. This includes the study of emotions caused by the color properties of a picture, as well as the evaluation of the results of an emotional image retrieval processes. Results of different experiments show that a picture’s luminance and color have the power to influence emotion. The subsequent evaluation of the results shows an improvement of emotional image retrieval processes. Consequently, one can conclude that the consideration of emotions for ranking affects the quality of the results of the Image Retrieval positively.

The purpose of dissertation is to utilize connotative messages for enhancing image retrieval and browsing. By adopting semiotics as a theoretical tool, this study explores problems of image retrieval and proposes an image retrieval model. The semiotics approach conceptually demonstrates that: 1) a fundamental reason for the dissonance between retrieved images and user needs is representation of connotative messages, and 2) the image retrieval model which makes use of denotative index terms is able to facilitate users to browse connotatively related images effectively even when the users' needs are potentially expressed in the form of denotative query. Two experiments are performed for verifying the semiotic-based image retrieval model and evaluating the effectiveness of the model. As data sources, 5,199 records are collected from Artefacts Canada: Humanities by Canadian Heritage Information Network, and the candidate terms of connotation and denotation are extracted from Art & Architecture Thesaurus. The first experiment, by applying term association measures, verifies that the connotative messages of an image can be derived from denotative messages of the image. The second experiment reveals that the association thesaurus which is constructed based on the associations between connotation and denotation facilitates assigning connotative terms to image documents. In addition, the result of relevant judgments presents that the association thesaurus improves the relative recall of retrieved image documents as well as the relative recall of browsing sets. This study concludes that the association thesaurus indicating associations between connotation and denotation is able to improve the accessibility of the connotative messages. The results of the study are hoped to contribute to the conceptual knowledge of image retrieval by providing understandings of connotative messages within an image and to the practical design of image retrieval system by proposing an association thesaurus which can supplement the limitations of the current content-based image retrieval systems (CBIR).

A recuperação de imagens baseada em conteúdo, amplamente conhecida como CBIR (do inglês Content-Based Image Retrieval), é um ramo da área da computação que vem crescendo muito nos últimos anos e vem contribuindo com novos desafios. Sistemas que utilizam tais técnicas propiciam o armazenamento e manipulação de grandes volumes de dados e imagens e processam operações de consultas de imagens a partir de características visuais extraídas automaticamente por meio de métodos computacionais. Esses sistemas devem prover uma interface de usuário visando uma interação fácil, natural e atraente entre o usuário e o sistema, permitindo que o usuário possa realizar suas tarefas com segurança, de modo eficiente, eficaz e com satisfação. Desse modo, o design da interface firma-se como um elemento fundamental para o sucesso de sistemas CBIR. Contudo, dentro desse contexto, a interface do usuário ainda é um elemento constituído de pouca pesquisa e desenvolvimento. Um dos obstáculos para eficácia de design desses sistemas consiste da necessidade em prover aos usuários uma interface de alta qualidade para permitir que o usuário possa consultar imagens similares a uma dada imagem de referência e visualizar os resultados. Para atingir esse objetivo, este trabalho visa analisar a interação do usuário em sistemas de recuperação de imagens por conteúdo e avaliar sua funcionalidade e usabilidade, aplicando técnicas de interação humano-computador que apresentam bons resultados em relação à performance de sistemas com grande complexidade, baseado em um estudo de caso aplicado à medicina / The content-based image retrieval (CBIR) is a challenging area of the computer science that has been growing in a very fast pace in the last years. CBIR systems employ techniques for extracting features from the images, composing the features vectores, and store them together with the images in data bases management system, allowing indexing and querying. CBIR systems deal with large volumes of images. Therefore, the feature vectors are extracted by automatic methods. These systems allow to query the images by content, processing similarity queries, which inherently demands user interaction. Consequently, CBIR systems must pay attention to the user interface, aiming at providing friendly, intuitive and attractive interaction, leading the user to do the tasks efficiently, getting the desired results, feeling safe and fulfilled. From the points highlighted beforehand, we can state that the human-computer interaction (HCI) is a key element of a CBIR system. However, there is still little research on HCI for CBIR systems. One of the requirements of HCI for CBIR is to provide a high quality interface to allow the user to search for similar images to a given query image, and to display the results properly, allowing further interaction. The present dissertation aims at analyzing the user interaction in CBIR systems specially suited to medical applications, evaluating their usability by applying HCI techniques. To do so, a case study was employed, and the results presented

Die vorliegende Arbeit untersucht das Potenzial von Augmented Reality zur Verbesserung von Image Retrieval Prozessen. Herausforderungen in Design und Gebrauchstauglichkeit wurden für beide Forschungsbereiche dargelegt und genutzt, um Designziele für Konzepte zu entwerfen. Eine Taxonomie für Image Retrieval in Augmented Reality wurde basierend auf der Forschungsarbeit entworfen und eingesetzt, um verwandte Arbeiten und generelle Ideen für Interaktionsmöglichkeiten zu strukturieren. Basierend auf der Taxonomie wurden Anwendungsszenarien als weitere Anforderungen für Konzepte formuliert. Mit Hilfe der generellen Ideen und Anforderungen wurden zwei umfassende Konzepte für Image Retrieval in Augmented Reality ausgearbeitet. Eins der Konzepte wurde auf einer Microsoft HoloLens umgesetzt und in einer Nutzerstudie evaluiert. Die Studie zeigt, dass das Konzept grundsätzlich positiv aufgenommen wurde und bietet Erkenntnisse über unterschiedliches Verhalten im Raum und verschiedene Suchstrategien bei der Durchführung von Image Retrieval in der erweiterten Realität.:1 Introduction 1.1 Motivation and Problem Statement 1.1.1 Augmented Reality and Head-Mounted Displays 1.1.2 Image Retrieval 1.1.3 Image Retrieval within Augmented Reality 1.2 Thesis Structure 2 Foundations of Image Retrieval and Augmented Reality 2.1 Foundations of Image Retrieval 2.1.1 Definition of Image Retrieval 2.1.2 Classification of Image Retrieval Systems 2.1.3 Design and Usability in Image Retrieval 2.2 Foundations of Augmented Reality 2.2.1 Definition of Augmented Reality 2.2.2 Augmented Reality Design and Usability 2.3 Taxonomy for Image Retrieval within Augmented Reality 2.3.1 Session Parameters 2.3.2 Interaction Process 2.3.3 Summary of the Taxonomy 3 Concepts for Image Retrieval within Augmented Reality 3.1 Related Work 3.1.1 Natural Query Specification 3.1.2 Situated Result Visualization 3.1.3 3D Result Interaction 3.1.4 Summary of Related Work 3.2 Basic Interaction Concepts for Image Retrieval in Augmented Reality 3.2.1 Natural Query Specification 3.2.2 Situated Result Visualization 3.2.3 3D Result Interaction 3.3 Requirements for Comprehensive Concepts 3.3.1 Design Goals 3.3.2 Application Scenarios 3.4 Comprehensive Concepts 3.4.1 Tangible Query Workbench 3.4.2 Situated Photograph Queries 3.4.3 Conformance of Concept Requirements 4 Prototypic Implementation of Situated Photograph Queries 4.1 Implementation Design 4.1.1 Implementation Process 4.1.2 Structure of the Implementation 4.2 Developer and User Manual 4.2.1 Setup of the Prototype 4.2.2 Usage of the Prototype 4.3 Discussion of the Prototype 5 Evaluation of Prototype and Concept by User Study 5.1 Design of the User Study 5.1.1 Usability Testing 5.1.2 Questionnaire 5.2 Results 5.2.1 Logging of User Behavior 5.2.2 Rating through Likert Scales 5.2.3 Free Text Answers and Remarks during the Study 5.2.4 Observations during the Study 5.2.5 Discussion of Results 6 Conclusion 6.1 Summary of the Present Work 6.2 Outlook on Further Work / The present work investigates the potential of augmented reality for improving the image retrieval process. Design and usability challenges were identified for both fields of research in order to formulate design goals for the development of concepts. A taxonomy for image retrieval within augmented reality was elaborated based on research work and used to structure related work and basic ideas for interaction. Based on the taxonomy, application scenarios were formulated as further requirements for concepts. Using the basic interaction ideas and the requirements, two comprehensive concepts for image retrieval within augmented reality were elaborated. One of the concepts was implemented using a Microsoft HoloLens and evaluated in a user study. The study showed that the concept was rated generally positive by the users and provided insight in different spatial behavior and search strategies when practicing image retrieval in augmented reality.:1 Introduction 1.1 Motivation and Problem Statement 1.1.1 Augmented Reality and Head-Mounted Displays 1.1.2 Image Retrieval 1.1.3 Image Retrieval within Augmented Reality 1.2 Thesis Structure 2 Foundations of Image Retrieval and Augmented Reality 2.1 Foundations of Image Retrieval 2.1.1 Definition of Image Retrieval 2.1.2 Classification of Image Retrieval Systems 2.1.3 Design and Usability in Image Retrieval 2.2 Foundations of Augmented Reality 2.2.1 Definition of Augmented Reality 2.2.2 Augmented Reality Design and Usability 2.3 Taxonomy for Image Retrieval within Augmented Reality 2.3.1 Session Parameters 2.3.2 Interaction Process 2.3.3 Summary of the Taxonomy 3 Concepts for Image Retrieval within Augmented Reality 3.1 Related Work 3.1.1 Natural Query Specification 3.1.2 Situated Result Visualization 3.1.3 3D Result Interaction 3.1.4 Summary of Related Work 3.2 Basic Interaction Concepts for Image Retrieval in Augmented Reality 3.2.1 Natural Query Specification 3.2.2 Situated Result Visualization 3.2.3 3D Result Interaction 3.3 Requirements for Comprehensive Concepts 3.3.1 Design Goals 3.3.2 Application Scenarios 3.4 Comprehensive Concepts 3.4.1 Tangible Query Workbench 3.4.2 Situated Photograph Queries 3.4.3 Conformance of Concept Requirements 4 Prototypic Implementation of Situated Photograph Queries 4.1 Implementation Design 4.1.1 Implementation Process 4.1.2 Structure of the Implementation 4.2 Developer and User Manual 4.2.1 Setup of the Prototype 4.2.2 Usage of the Prototype 4.3 Discussion of the Prototype 5 Evaluation of Prototype and Concept by User Study 5.1 Design of the User Study 5.1.1 Usability Testing 5.1.2 Questionnaire 5.2 Results 5.2.1 Logging of User Behavior 5.2.2 Rating through Likert Scales 5.2.3 Free Text Answers and Remarks during the Study 5.2.4 Observations during the Study 5.2.5 Discussion of Results 6 Conclusion 6.1 Summary of the Present Work 6.2 Outlook on Further Work

L’objectif principal de cette thèse est la découverte d’objets visuels fréquents dans de grandes collections multimédias (images ou vidéos). Comme dans de nombreux domaines (finance, génétique, . . .), il s’agit d’extraire une connaissance de manière automatique ou semi-automatique en utilisant la fréquence d’apparition d’un objet au sein d’un corpus comme critère de pertinence. Une première contribution de la thèse est de fournir un formalisme aux problèmes de découverte et de fouille d’instances d’objets visuels fréquents. La deuxième contribution de la thèse est une méthode générique de résolution de ces deux types de problème reposant d’une part sur un processus itératif d’échantillonnage d’objets candidats et d’autre part sur une méthode efficace d’appariement d’objets rigides à large échelle. La troisième contribution de la thèse s’attache à construire une fonction de vraisemblance s’approchant au mieux de la distribution parfaite, tout en restant scalable et efficace. Les expérimentations montrent que contrairement aux méthodes de l’état de l’artnotre approche permet de découvrir efficacement des objets de très petite taille dans des millions d’images. Pour finir, plusieurs scénarios d’exploitation des graphes visuels produits par notre méthode sont proposées et expérimentés. Ceci inclut la détection d’évènements médiatiques transmédias et la suggestion de requêtes visuelles. / The main goal of this thesis is to discover frequent visual objects in large multimedia collections. As in many areas (finance, genetics, . . .), it consists in extracting a knowledge, using the occurence frequency of an object in a collection as a relevance criterion. A first contribution is to provide a formalism to the problems of mining and discovery of frequent visual objects. The second contribution is a generic method to solve these two problems, based on an iterative sampling process, and on an efficient and scalable rigid objects matching. The third contribution of this work focuses on building a likelihood function close to the perfect distribution. Experiments show that contrary to state-of-the-art methods, our approach allows to discover efficiently very small objects in several millions images. Finally, several applications are presented, including trademark logos discovery, transmedia events detection or visual-based query suggestion.

京都大学 / 0048 / 新制・課程博士 / 博士(情報学) / 甲第17927号 / 情博第509号 / 新制||情||90(附属図書館) / 30747 / 京都大学大学院情報学研究科社会情報学専攻 / (主査)教授 吉川 正俊, 教授 石田 亨, 教授 田中 克己 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DGAM