In a known-item search task (KIS), the goal is to find a previously seen image in a multimedia collection. In this thesis, we discuss two different approaches based on the visual description of the image. In the first one, the user creates a collage of images (using images from an external search engine), based on which we provide the most similar results from the dataset. Our results show that preprocessing the images in the dataset by splitting them into several parts is a better way to work with the spatial information contained in the user input. We compared the approach to a baseline, which does not utilize this spatial information and an approach that alters a layer in a deep neural network. We also present an alternative approach to the KIS task, search by faces. In this approach, we work with the faces extracted from the images. We investigate face representation for the ability to sort the faces based on their similarity. Then we present a structure that allows easy exploration of the set of faces. We provide a demo, implementing all presented techniques.

Ongoing expansion of digital images requires new methods for sorting, browsing, and sear- ching through huge image databases. This is a domain of Content-Based Image Retrieval (CBIR) systems, which are database search engines for images. A user typically submit a query image or series of images and the CBIR system tries to find and to retrieve the most similar images from the database. Optimally, the retrieved images should not be sensitive to circumstances during their acquisition. Unfortunately, the appearance of natural objects and materials is highly illumination and viewpoint dependent. This work focuses on representation and retrieval of homogeneous images, called textu- res, under the circumstances with variable illumination and texture rotation. We propose a novel illumination invariant textural features based on Markovian modelling of spatial tex- ture relations. The texture is modelled by Causal Autoregressive Random field (CAR) or Gaussian Markov Random Field (GMRF) models, which allow a very efficient estimation of its parameters, without the demanding Monte Carlo minimisation. Subsequently, the estimated model parameters are transformed into the new illumination invariants, which represent the texture. We derived that our textural representation is invariant to changes of illumination intensity and...

Les documents visuels comprenant des images et des vidéos sont en croissance rapide sur Internet et dans nos collections personnelles. Cela nécessite une analyse automatique du contenu visuel qui fait appel à la conception de méthodes intelligentes pour correctement indexer, rechercher et récupérer des images et des vidéos. Cette thèse vise à améliorer la détection automatique des concepts dans les vidéos sur Internet. Nos contributions portent sur des différents niveaux dans le cadre de détection de concept et peuvent être divisés en trois parties principales. La première partie se focalise sur l’amélioration du modèle de représentation des vidéos « Bag-of-Words (BOW) » en proposant un nouveau mécanisme de construction qui utilise des étiquettes de concepts et une autre technique qui ajoute un raffinement à la signature BOW basée sur la distribution de ses éléments. Nous élaborons ensuite des méthodes pour intégrer des entités semblables et dissemblables pour construire des modèles de reconnaissance améliorés dans la deuxième partie. A ce stade-là, nous observons l’information potentielle que les concepts partagent et construisons des modèles pour les méta-concepts dont sont dérivés les résultats spécifiques de concepts. Cela améliore la reconnaissance des concepts qui ont peu d’exemples annotés. Enfin, nous concevons certaines méthodes d'apprentissage semi-supervisé pour bénéficier de la quantité importante de données non étiquetées. Nous proposons des techniques pour améliorer l'algorithme de cotraining avec une sélection optimale des classifieurs utilisés. / Visual material comprising images and videos is growing ever so rapidly over the internet and in our personal collections. This necessitates automatic understanding of the visual content which calls for the conception of intelligent methods to correctly index, search and retrieve images and videos. This thesis aims at improving the automatic detection of concepts in the internet videos by exploring all the available information and putting the most beneficial out of it to good use. Our contributions address various levels of the concept detection framework and can be divided into three main parts. The first part improves the Bag of Words (BOW) video representation model by proposing a novel BOW construction mechanism using concept labels and by including a refinement to the BOW signature based on the distribution of its elements. We then devise methods to incorporate knowledge from similar and dissimilar entities to build improved recognition models in the second part. Here we look at the potential information that the concepts share and build models for meta-concepts from which concept specific results are derived. This improves recognition for concepts lacking labeled examples. Lastly we contrive certain semi-supervised learning methods to get the best of the substantial amount of unlabeled data. We propose techniques to improve the semi-supervised cotraining algorithm with optimal view selection.

This dissertation proposes two novel manifold graph-based ranking systems for Content-Based Image Retrieval (CBIR). The two proposed systems exploit the synergism between relevance feedback-based transductive short-term learning and semantic feature-based long-term learning to improve retrieval performance. Proposed systems first apply the active learning mechanism to construct users' relevance feedback log and extract high-level semantic features for each image. These systems then create manifold graphs by incorporating both the low-level visual similarity and the high-level semantic similarity to achieve more meaningful structures for the image space. Finally, asymmetric relevance vectors are created to propagate relevance scores of labeled images to unlabeled images via manifold graphs. The extensive experimental results demonstrate two proposed systems outperform the other state-of-the-art CBIR systems in the context of both correct and erroneous users' feedback.

This work addresses the need for an alternative to keyword-based search for sifting through large PDF medical journal article document collections for literature review purposes. Despite users' best efforts to form precise and accurate queries, it is often difficult to guess the right keywords to find all the related articles while finding a minimum number of unrelated ones. Failure during literature review to find relevant, related research results in wasted research time and effort in addition to missing significant work in the related area which could affect the quality of the research work being conducted. The purpose of this work is to explore the benefits of a retrieval system for professional journal articles in PDF format that supports hybrid queries composed of both text and images. PDF medical journal articles contain formatting and layout information that imply the structure and organization of the document. They also contain figures and tables rich with content and meaning. Stripping a PDF into “full-text” for indexing purposes disregards these important features. Specifically, this work investigated the following: (1) what effect the incorporation of a document's embedded figures into the query (in addition to its text) has on retrieval performance (precision) compared to plain keyword-based search; (2) how current text-based document-query similarity methods can be enhanced by using formatting and font-size information as a structure and organization model for a PDF document; (3) whether to use the standard Euclidean distance function or the matrix distance function for content-based image retrieval; (4) how to convert a PDF into a structured, formatted, reflowable XML representation given a pure-layout PDF document; (5) what document views (such as a term frequency cloud, a document outline, or a document's figures) would help users wade through search results to quickly select those that are worth a closer look. While the results of the experiments were unexpectedly worse than their baselines of comparison (see the conclusion for a summary), the experimental methods are very valuable in showing others what directions have already been pursued and why they did not work and what remaining problems need to be solved in order to achieve the goal of improving literature review through use of a hybrid text and image retrieval system.

As Machine Learning (ML) methods have gained traction in recent years, someproblems regarding the construction of such methods have arisen. One such problem isthe collection and labeling of data sets. Specifically when it comes to many applicationsof Computer Vision (CV), one needs a set of images, labeled as either being of someclass or not. Creating such data sets can be very time consuming. This project setsout to tackle this problem by constructing an end-to-end system for searching forobjects in images (i.e. an Object Based Image Retrieval (OBIR) method) using an objectdetection framework (You Only Look Once (YOLO) [16]). The goal of the project wasto create a method that; given an image of an object of interest q, search for that sameor similar objects in a set of other images S. The core concept of the idea is to passthe image q through an object detection model (in this case YOLOv5 [16]), create a”fingerprint” (can be seen as a sort of identity for an object) from a set of feature mapsextracted from the YOLOv5 [16] model and look for corresponding similar parts of aset of feature maps extracted from other images. An investigation regarding whichvalues to select for a few different parameters was conducted, including a comparisonof performance for a couple of different similarity metrics. In the table below,the parameter combination which resulted in the highest F_Top_300-score (a measureindicating the amount of relevant images retrieved among the top 300 recommendedimages) in the parameter selection phase is presented. Layer: 23Pool Methd: maxSim. Mtrc: eucFP Kern. Sz: 4 Evaluation of the method resulted in F_Top_300-scores as can be seen in the table below. Mouse: 0.820Duck: 0.640Coin: 0.770Jet ski: 0.443Handgun: 0.807Average: 0.696 / Medan ML-metoder har blivit mer populära under senare år har det uppstått endel problem gällande konstruktionen av sådana metoder. Ett sådant problem ärinsamling och annotering av data. Mer specifikt när det kommer till många metoderför datorseende behövs ett set av bilder, annoterande att antingen vara eller inte varaav en särskild klass. Att skapa sådana dataset kan vara väldigt tidskonsumerande.Metoden som konstruerades för detta projekt avser att bekämpa detta problem genomatt konstruera ett end-to-end-system för att söka efter objekt i bilder (alltså en OBIR-metod) med hjälp av en objektdetekteringsalgoritm (YOLO). Målet med projektet varatt skapa en metod som; givet en bild q av ett objekt, söka efter samma eller liknandeobjekt i ett bibliotek av bilder S. Huvudkonceptet bakom idén är att köra bilden qgenom objektdetekteringsmodellen (i detta fall YOLOv5 [16]), skapa ett ”fingerprint”(kan ses som en sorts identitet för ett objekt) från en samling feature maps extraheradefrån YOLOv5-modellen [16] och leta efter liknande delar av samlingar feature maps iandra bilder. En utredning angående vilka värden som skulle användas för ett antalolika parametrar utfördes, inklusive en jämförelse av prestandan som resultat av olikalikhetsmått. I tabellen nedan visas den parameterkombination som gav högst F_Top_300(ett mått som indikerar andelen relevanta bilder bland de 300 högst rekommenderadebilderna). Layer: 23Pool Methd: maxSim. Mtrc: eucFP Kern. Sz: 4 Evaluering av metoden med parameterval enligt tabellen ovan resulterade i F_Top_300enligt tabellen nedan. Mouse: 0.820Duck: 0.640Coin: 0.770Jet ski: 0.443Handgun: 0.807Average: 0.696

Taxonomic classification has always been important to the study of any biological system. Many biological species will go unclassified and become lost forever at the current rate of classification. The current state of computer technology makes image storage and retrieval possible on a global level. As a result, computer-aided taxonomy is now possible. Content based image retrieval techniques utilize visual features of the image for classification. By utilizing image content and computer technology, the gap between taxonomic classification and species destruction is shrinking. This content based study utilizes the Fourier Descriptors of fifteen known landmark features on three Carpiodes species: C.carpio, C.velifer, and C.cyprinus. Classification analysis involves both unsupervised and supervised machine learning algorithms. Fourier Descriptors of the fifteen known landmarks provide for strong classification power on image data. Feature reduction analysis indicates feature reduction is possible. This proves useful for increasing generalization power of classification.

Neste trabalho são apresentados dois novos descritores de forma para tarefas de recuperação de imagens por conteúdo (CBIR) e análise de formas, que são construídos sobre uma extensão do conceito de tensor scale baseada na Transformada de Distância Euclidiana (EDT). Primeiro, o algoritmo de tensor scale é utilizado para extrair informações da forma sobre suas estruturas locais (espessura, orientação e anisotropia) representadas pela maior elipse contida em uma região homogênea centrada em cada pixel da imagem. Nos novos descritores, o limite do intervalo das orientações das elipses do modelo de tensor scale é estendido de 180º para 360º, de forma a melhor discriminar a descrição das estruturas locais. Então, com base em diferentes abordagens de amostragem, visando resumir informações mais relevantes, os novos descritores são construídos. No primeiro descritor proposto, Tensor Scale Sector (TSS), a distribuição das orientações relativas das estruturas locais em setores circulares é utilizada para compor um vetor de características de tamanho fixo, para uma caracterização de formas baseada em região. No segundo descritor, o Tensor Scale Band (TSB), foram considerados histogramas das orientações relativas extraídos de bandas concêntricas, formando também um vetor de características de tamanho fixo, com uma função de distância de tempo linear. Resultados experimentais com diferentes bases de formas (MPEG-7 e MNIST) são apresentados para ilustrar e validar os métodos. TSS demonstra resultados comparáveis aos métodos estado da arte, que geralmente dependem de algoritmos custosos de otimização de correspondências. Já o TSB, com sua função de distância em tempo linear, se demonstra como uma solução adequada para grandes coleções de formas. / In this work, two new shape descriptors are proposed for tasks in Content-Based Image Retrieval (CBIR) and Shape Analysis tasks, which are built upon an extended tensor scale based on the Euclidean Distance Transform (EDT). First, the tensor scale algorithm is applied to extract shape attributes from its local structures (thickness, orientation, and anisotropy) as represented by the largest ellipse within a homogeneous region centered at each image pixel. In the new descriptors, the upper limit of the interval of local orientation of tensor scale ellipses is extended from 180º to 360º, to better discriminate the description of local structures. Then, the new descriptors are built based on different sampling approaches, aiming to summarize the most relevant features. In the first proposed descriptor, Tensor Scale Sector descriptor (TSS), the local distributions of relative orientations within circular sectors are used to compose a fixed-length feature vector, for a region-based shape characterization. For the second method, the Tensor Scale Band (TSB) descriptor, histograms of relative orientations are considered for each circular concentric band, to also compose a fixed-length feature vector, with linear time distance function for matching. Experimental results for different shape datasets (MPEG-7 and MNIST) are presented to illustrate and validate the methods. TSS can achieve high retrieval values comparable to state-of-the-art methods, which usually rely on time-consuming correspondence optimization algorithms, but uses a simpler and faster distance function, while the even faster linear complexity of TSB leads to a suitable solution for very large shape collections.

Com o crescente aumento no número de imagens geradas em mídias digitais surgiu a necessidade do desenvolvimento de novas técnicas de recuperação desses dados. Um critério de busca que pode ser utilizado na recuperação das imagens é o da dissimilaridade, no qual o usuário deseja recuperar as imagens semelhantes à uma imagem de consulta. Para a realização das consultas são empregados vetores de características extraídos das imagens e funções de distância para medir a dissimilaridade entre pares desses vetores. Infelizmente, a busca por conteúdo de imagens em consultas simples tende a gerar resultados que não correspondem ao interesse do usuário misturados aos resultados significativos encontrados, pois em geral há uma descontinuidade semântica entre as características extraídas automaticamente e a subjetividade da interpretação humana. Com o intuito de tratar esse problema, diversos métodos foram propostos para a diminuição da descontinuidade semântica. O foco principal desta tese é o desenvolvimento de métodos escaláveis para a redução da descontinuidade semântica em sistemas recuperação de imagens por conteúdo em tempo real. Nesta sentido, são apresentados: a formalização de consultas por similaridade que permitem a utilização de múltiplos centros de consulta em espaços métricos como base para métodos de realimentação de relevância; um método exato para otimização dessas consultas nesses espaços; e um modelo para tratamento da diversidade em consultas por similaridade e heurísticas para sua otimização / The increasing number of images captured in digital media fostered the developmet of new methods for the recovery of these images. Dissimilarity is a criteria that can be used for image retrieval, where the results are images that are similar to a given reference. The queries are based on feature vectors automatically extracted from the images and on distance functions to measure the dissimilarity between pair of vectors. Unfortunately, the search for images in simple queries may result in images that do not fulfill the user interest together with meaningful images, due to the semantic gap between the image features and to the subjectivity of the human interpretation. This problem leaded to the development of many methods to deal with the semantic gap. The focus of this thesis is the development of scalable methods aiming the semantic gap reduction in real time for content-based image retrieval systems. For this purpose, we present the formal definition of similarity queries based on multiple query centers in metric spaces to be used in relevance feedback methods, an exact method to optimize these queries and a model to deal with diversity in nearest neighbor queries including heuristics for its optimization

O hipocampo é uma estrutura cerebral que possui importância primordial para o sistema de memória humana. Alterações no seus tecidos levam a doenças neurodegenerativas, tais como: epilepsia, esclerose múltipla e demência, entre outras. Para medir a atrofia do hipocampo é necessário isolá-lo do restante do cérebro. A separação do hipocampo das demais partes do cérebro ajuda aos especialistas na análise e o entendimento da redução de seu volume e detecção de qualquer anomalia presente. A extração do hipocampo é principalmente realizada de modo manual, a qual é demorada, pois depende da interação do usuário. A segmentação automática do hipocampo é investigada como uma alternativa para contornar tais limitações. Esta dissertação de mestrado apresenta um novo método de segmentação automático, denominado Modelo de Nuvem de Similaridade (Similarity Cloud Model - SimCM). O processo de segmentação é dividido em duas etapas principais: i) localização por similaridade e ii) ajuste de nuvem. A primeira operação utiliza a nuvem para localizar a posição mais provável do hipocampo no volume destino. A segunda etapa utiliza a nuvem para corrigir o delineamento final baseada em um novo método de cálculo de readequação dos pesos das arestas. Nosso método foi testado em um conjunto de 235 MRI combinando imagens de controle e de pacientes com epilepsia. Os resultados alcançados indicam um rendimento superior tanto em efetividade (qualidade da segmentação) e eficiência (tempo de processamento), comparado com modelos baseados em grafos e com modelos Bayesianos. Como trabalho futuro, pretendemos utilizar seleção de características para melhorar a construção da nuvem e o delineamento dos tecidos / The hippocampus is a particular structure that plays a main role in human memory systems. Tissue modifications of the hippocampus lead to neurodegenerative diseases as epilepsy, multiple sclerosis, and dementia, among others. To measure hippocampus atrophy, it is crucial to get its isolated representation from the whole brain volume. Separating the hippocampus from the brain helps physicians in better analyzing and understanding its volume reduction, and detecting any abnormal behavior. The extraction of the hippocampus is dominated by manual segmentation, which is time consuming mainly because it depends on user interaction. Therefore, automatic segmentation of the hippocampus has being investigated as an alternative solution to overcome such limitations. This master dissertation presents a new automatic segmentation method called Similarity Cloud Model (SimCM) based on hippocampus feature extraction. The segmentation process consists of two main operations: i) localization by similarity, and ii) cloud adjustment. The first operation uses the cloud to localize the most probable position of the hippocampus in a target volume. The second process invokes the cloud to correct the final labeling, based on a new method for arc-weight re-adjustment. Our method has been tested in a dataset of 235 MRIs combining healthy and epileptic patients. Results indicate superior performance, in terms of effectiveness (segmentation quality) and efficiency (processing time), in comparison with similar graph-based and Bayesian-based models. As future work, we intend to use feature selection to improve cloud construction and tissue delineation