The objective of this work is to improve performance in fine-grained visual categorization (FGVC). In particular, we are interested in the large-scale classification between hundreds of different flower, bird, dog species. FGVC is challenging due to high intra-class variances caused by deformation, view angle, illumination and occlusion, and low inter-class variance since some categories only differ in detail that only experts notice. Applications include field guides, automatic image annotation, one-click shopping app and 3D reconstruction. At the start, we discuss the importance of foreground segmentation in FGVC, where we focus on the unsupervised segmentation of image training sets into fore- ground and background in order to improve image classification performance. To this end, we introduce a new scalable, alternation-based algorithm for co-segmentation, Bi-CoS, which is simpler than many of its predecessors, and yet has superior performance on standard benchmark image datasets. Next, we extend BiCos to a new model, Tri- CoS, that adds a class-discriminitiveness term directly into the segmentation objective. The new term aims at removing image regions that, although appearing as foreground, do not contribute to the discrimination between classes. We also propose a model that combines parts alignment and foreground segmentation into a unified convex framework. The model is called Symbiotic in that part discovery/localization is helped by segmentation and, conversely, the segmentation is helped by the detection (e.g. part layout). The joined system improves over what can be achieved with an analogous system that runs segmentation and part-localization independently. Finally, we built a new flower dataset consisting of 26,798 high quality images collected by ourselves and 187,559 images gathered from existing datasets. The construction of this dataset follows a strict biological taxonomy. We also evaluate the impact of using the Amazon Mechanical Turk (AMT) service for filtering fine-grained data.

Ces travaux de recherche ont pour ambition de contribuer à la problématique de la classification d’images de documents. Plus précisément, ces travaux tendent à répondre aux problèmes rencontrés par des sociétés de numérisation dont l’objectif est de mettre à disposition de leurs clients une version numérique des documents papiers accompagnés d’informations qui leurs sont relatives. Face à la diversité des documents à numériser, l’extraction d’informations peut s’avérer parfois complexe. C’est pourquoi la classification et l’indexation des documents sont très souvent réalisées manuellement. Ces travaux de recherche ont permis de fournir différentes solutions en fonction des connaissances relatives aux images que possède l’utilisateur ayant en charge l’annotation des documents.Le premier apport de cette thèse est la mise en place d’une méthode permettant, de manière interactive, à un utilisateur de classer des images de documents dont la nature est inconnue. Le second apport de ces travaux est la proposition d’une technique de recherche d’images de documents par l’exemple basée sur l’extraction et la mise en correspondance de points d’intérêts. Le dernier apport de cette thèse est l’élaboration d’une méthode de classification d’images de documents utilisant les techniques de sacs de mots visuels. / The aim of this research is to contribute to the document image classification problem. More specifically, these studies address digitizing company issues which objective is to provide the digital version of paper document with information relating to them. Given the diversity of documents, information extraction can be complex. This is why the classification and the indexing of documents are often performed manually. This research provides several solutions based on knowledge of the images that the user has. The first contribution of this thesis is a method for classifying interactively document images, where the content of documents and classes are unknown. The second contribution of this work is a new technique for document image retrieval by giving one example of researched document. This technique is based on the extraction and matching of interest points. The last contribution of this thesis is a method for classifying document images by using bags of visual words techniques.

Land cover classification using artificial neural networks Abstract This Diploma thesis deals with automatic classification of the satellite high spatial resolution image in the field of land cover. The first half of the work contains the theoretical information about remote sensing and classification methods. The biggest attention is given to the artificial neural networks. In practical part of Diploma thesis are these methods used for the classification of SPOT satellite image. Keywords: remote sensing, image classification, artificial neural networks, SPOT

Image classification is an active topic of computer vision research. This topic deals with the learning of patterns in order to allow efficient classification of visual information. However, most research efforts have focused on 2D image classification. In recent years, advances of 3D imaging enabled the development of applications and provided new research directions. In this thesis, we present methodologies and techniques for image classification using 3D image data. We conducted our research focusing on the attributes and limitations of depth information regarding possible uses. This research led us to the development of depth feature extraction methodologies that contribute to the representation of images thus enhancing the recognition efficiency. We proposed a new classification algorithm that adapts to the need of image representations by implementing a scale-based decision that exploits discriminant parts of representations. Learning from the design of image representation methods, we introduced our own which describes each image by its depicting content providing more discriminative image representation. We also propose a dictionary learning method that exploits the relation of training features by assessing the similarity of features originating from similar context regions. Finally, we present our research on deep learning algorithms combined with data and techniques used in 3D imaging. Our novel methods provide state-of-the-art results, thus contributing to the research of 3D image classification.

Doctor of Philosophy / Department of Computer Science / David A. Gustafson / In this paper we evaluate a general form of image classification algorithm based on dense SIFT sampling. This algorithm is present in some form in most state-of-the-art classification systems. However, in this algorithm, numerous parameters must be tuned, and current research provides little insight into effective parameter tuning. We explore the relationship between various parameters and classification performance. Many of our results suggest that there are basic modifications which would improve state-of-the-art algorithms. Additionally, we develop two novel concepts, sampling redundancy and semantic capacity, to explain our data. These concepts provide additional insight into the limitations and potential improvements of state-of-the-art algorithms.

It is estimated that less than ten percent of the world's species have been discovered and described. The main reason for the slow pace of new species description is that the science of taxonomy, as traditionally practiced, can be very laborious: taxonomists have to manually gather and analyze data from large numbers of specimens and identify the smallest subset of external body characters that uniquely diagnoses the new species as distinct from all its known relatives. The pace of data gathering and analysis can be greatly increased by the information technology. In this paper, we propose a content-based image retrieval system for taxonomic research. The system can identify representative body shape characters of known species based on digitized landmarks and provide statistical clues for assisting taxonomists to identify new species or subspecies. The experiments on a taxonomic problem involving species of suckers in the genera Carpiodes demonstrate promising results.

Segundo a Agência Nacional de Transportes Terrestres (ANTT) em seu Anuário Estatístico dos Transportes Terrestres AETT (2008), o Brasil em todo o seu território possui 211.678 quilômetros de rodovias pavimentadas. O valor de serventia do pavimento diminui com o passar do tempo por dois fatores principais: o tráfego e as intempéries (BERNUCCI et al., 2008). Monitorar a condição de uso de toda a extensão das rodovias brasileiras é tarefa dispendiosa e demorada. A investigação de novas técnicas que permitam o levantamento da condição dos pavimentos de forma ágil e automática é parte da pesquisa deste trabalho. Nos últimos anos, um número crescente de imagens de alta resolução espacial tem surgido no mercado mundial com o aparecimento dos novos satélites e sensores aeroembarcados de sensoriamento remoto. Da mesma forma, imagens multiespectrais e até mesmo hiperespectrais estão sendo disponibilizadas comercialmente e para pesquisa científica. Neste trabalho são utilizadas imagens hiperespectrais de sensor digital aeroembarcado. Uma metodologia para identificação automática dos pavimentos asfaltados e classificação das principais ocorrências dos defeitos do asfalto foi desenvolvida. A primeira etapa da metodologia é a identificação do asfalto na imagem, utilizando uma classificação híbrida baseada inicialmente em pixel e depois refinada por objetos foi possível a extração da informação de asfalto das imagens disponíveis. A segunda etapa da metodologia é a identificação e classificação das ocorrências dos principais defeitos nos pavimentos flexíveis que são observáveis nas imagens de alta resolução espacial. Esta etapa faz uso intensivo das novas técnicas de classificação de imagens baseadas em objetos. O resultado final é a geração de índices da condição do pavimento, a partir das imagens, que possam ser comparados com os indicadores da qualidade da superfície do pavimento já normatizados pelos órgãos competentes no país. / According to Statistical Survey of Land Transportation AETT (2008) of National Agency of Land Transportation (ANTT), Brazil has in its territory 211,678 kilometers of paved roads. The pavement Present Serviceability Ratio (PSR) value decreases over time by two main factors: traffic and weather (BERNUCCI et al., 2008). Monitor the condition of use of all Brazilian roads is expensive and time consuming task. The investigation of new techniques that allow a quick and automatic survey of pavement condition is part of this research. In recent years, an increasing number of images with high spatial resolution has emerged on the world market with the advent of new remote sensing satellites and airborne sensors. Similarly, multispectral and even hyperspectral imagery are become available commercially and for scientific research nowadays. Hyperspectral images from digital airborne sensor have been used in this work. A new methodology for automatic identification of asphalted pavement and also for classification of the main defects of the asphalt has been developed. The first step of the methodology is the identification of the asphalt in the image, using hybrid classification based on pixel initially and after improved by objects. Using this approach was feasible to extract asphalt information from the available images. The second step of the methodology is the identification and classification of the main defects of flexible pavement surface that are observable in high spatial resolution imagery. This step makes intensive use of new techniques for classification of images based on objects. The goal, is the generation of pavement surface condition index from the images that can be compared with quality index of pavement surface that are already regulated by the regulatory agency in the country.

In Image Classification, pictures of the same type of object can have very different pixel values. Traditional norm-based metrics therefore fail to identify objectsin the same category. Topology is a branch of mathematics that deals with homeomorphic spaces, by discarding length. With topology, we can discover patterns in the image that are invariant to rotation, translation and warping. Persistent Homology is a new approach in Applied Topology that studies the presence of continuous regions and holes in an image. It has been used successfully for image segmentation and classification [12]. However, current approaches in image classification require a grayscale image to generate the persistence modules. This means information encoded in colour channels is lost. This thesis investigates whether the information in the red, green and blue colour channels of an RGB image hold additional information that could help algorithms classify pictures. We apply two recent methods, one by Adams [2] and the other by Hofer [25], on the CUB-200-2011 birds dataset [40] andfind that Hofer’s method produces significant results. Additionally, a modified method based on Hofer that uses the RGB colour channels produces significantly better results than the baseline, with over 48 % of images correctly classified, compared to 44 % and with a more significant improvement at lower resolutions.This indicates that colour channels do provide significant new information and generating one persistence module per colour channel is a viable approach to RGB image classification.

Imagens multiespectrais estão presentes na grande maioria de dispositivos de imageamento atuais, desde câmeras pessoais até microscópios, telescópios e satélites. No entanto, grande parte dos trabalhos em análise de texturas e afins propõem abordagens monocromáticas, que muitas vezes consideram apenas níveis de cinza. Nesse contexto e considerando o aumento da capacidade dos computadores atuais, o uso da informação espectral deve ser considerada na construção de modelos melhores. Ultimamente redes neurais convolucionais profundas pré-treinadas tem sido usadas em imagens coloridas de 3 canais, porém são limitadas a apenas esse formato e computam muitas convoluções, o que demanda por hardware específico (GPU). Esses fatos motivaram esse trabalho, que propõem técnicas para a modelagem e caracterização de imagens multiespectrais baseadas em redes complexas, que tem se mostrado uma ferramenta eficiente em trabalhos anteriores e possui complexidade computacional similar à métodos tradicionais. São introduzidas duas abordagens para aplicação em imagens coloridas de três canais, denominadas Rede Multicamada (RM) e Rede Multicamada Direcionada (RMD). Esses métodos modelam todos os canais da imagem de forma conjunta, onde as redes possuem conexões intra e entre canais, de forma parecida ao processamento oponente de cor do sistema visual humano. Experimentos em cinco bases de textura colorida mostram a proposta RMD supera vários métodos da literatura no geral, incluindo redes convolucionais e métodos tradicionais integrativos. Além disso, as propostas demonstraram alta robustez a diferentes espaços de cor (RGB, LAB, HSV e I1I2I3), enquanto que outros métodos oscilam de base para base. Também é proposto um método para caracterizar imagens multiespectrais de muitos canais, denominado Rede Direcionada de Similaridade Angular (RDSA). Nessa proposta, cada pixel multiespectral é considerado como um vetor de dimensão equivalente à quantidade de canais da imagem e o peso das arestas representa sua similaridade do cosseno, apontando para o pixel de maior valor absoluto. Esse método é aplicado em um conjunto de imagens de microscopia por fluorescência de 32 canais, em um experimento para identificar variações na estrutura foliar do espécime Jacaranda Caroba submetidos à diferentes condições. O método RDSA obtém as maiores taxas de acerto de classificação nesse conjunto de dados, com 91, 9% usando o esquema de validação cruzada Leave-one-out e 90, 5(±1, 1)% com 10-pastas, contra 81, 8% e 84, 7(±2, 2) da rede convolucional VGG16. / Multispectral images are present in the vast majority of current imaging devices, from personal cameras to microscopes, telescopes and satellites. However, much of the work in texture analysis and the like proposes monochromatic approaches, which often consider only gray levels. In this context and considering the performance increase of current computers, the use of the spectral information must be considered in the construction of better models. Lately, pre-trained deep convolutional neural networks have been used in 3-channel color images, however they are limited to just this format and compute many convolutions, which demands specific hardware (GPU). These facts motivated this work, which propose techniques for the modeling and characterization of multispectral images based on complex networks, which has proved to be an efficient tool in previous works and has computational complexity similar to traditional methods. Two approaches are introduced for application in 3-channel color images, called Multilayer Network (RM) and Directed Multilayer Network (RMD). These methods model all channels of the image together, where the networks have intra- and inter-channel connections, similar to the opponent color processing of the human visual system. Experiments in five color texture datasets shows that the RMD proposal overcomes several methods of the literature in general, including convolutional networks and traditional integrative methods. In addition, the proposals have demonstrated high robustness to different color spaces (RGB, LAB, HSV and I1I2I3), while other methods oscillate from dataset to dataset. Moreover it is proposed a new method to characterize multispectral images of many channels, called Directed Network of Angular Similarity (RDSA). In this proposal, each multispectral pixel is considered as a vector of dimensions equivalent to the number of channels of the image and the weight of the edges represents its cosine similarity, pointing to the pixel of greatest absolute value. This method is applied to a set of fluorescence microscopy images of 32 channels in an experiment to identify variations in the leaf structure of the Jacaranda Caroba specimen under different conditions. The RDSA method obtains the highest classification rates in this dataset, with 91.9% with the Leave-one-out cross-validation scheme and 90.5(±1.1)% with 10-folds, against 81.8% and 84.7(±2.2) of the convolutional network VGG16.

This work presents my research about the use of Convolutional Neural Network (CNNs) for transfer learning through its application for colonic polyp classification and iris super-resolution. Traditionally, machine learning methods use the same feature space and the same distribution for training and testing the tools. Several problems in this approach can emerge as, for example, when the number of samples for training (especially in a supervised training) is limited. In the medical field, this problem is recurrent mainly because obtaining a database large enough with appropriate annotations for training is highly costly and may become impractical. Another problem relates to the distribution of textural features in a image database which may be too large such as the texture patterns of the human iris. In this case a single and specific training database might not get enough generalization to be applied to the entire domain. In this work we explore the use of texture transfer learning to surpass these problems for two applications: colonic polyp classification and iris super-resolution. The leading cause of deaths related to intestinal tract is the development of cancer cells (polyps) in its many parts. An early detection (when the cancer is still at an early stage) can reduce the risk of mortality among these patients. More specifically, colonic polyps (benign tumors or growths which arise on the inner colon surface) have a high occurrence and are known to be precursors of colon cancer development. Several studies have shown that automatic detection and classification of image regions which may contain polyps within the colon can be used to assist specialists in order to decrease the polyp miss rate. However, the classification can be a difficult task due to several factors such as the lack or excess of illumination, the blurring due to movement or water injection and the different appearances of polyps. Also, to find a robust and a global feature extractor that summarizes and represents all these pit-patterns structures in a single vector is very difficult and Deep Learning can be a good alternative to surpass these problems. One of the goals of this work is show the effectiveness of CNNs trained from scratch for colonic polyp classification besides the capability of knowledge transfer between natural images and medical images using off-the-shelf pretrained CNNs for colonic polyp classification. In this case, the CNN will project the target database samples into a vector space where the classes are more likely to be separable. The second part of this work dedicates to the transfer learning for iris super-resolution. The main goal of Super-Resolution (SR) is to produce, from one or more images, an image with a higher resolution (with more pixels) at the same time that produces a more detailed and realistic image being faithful to the low resolution image(s). Currently, most iris recognition systems require the user to present their iris for the sensor at a close distance. However, at present, there is a constant pressure to make that relaxed conditions of acquisitions in such systems could be allowed. In this work we show that the use of deep learning and transfer learning for single image super resolution applied to iris recognition can be an alternative for Iris Recognition of low resolution images. For this purpose, we explore if the nature of the images as well as if the pattern from the iris can influence the CNN transfer learning and, consequently, the results in the recognition process. / Diese Arbeit pr¨asentiert meine Forschung hinsichtlich der Verwendung von ”Transfer-Learning” (TL) in Kombination mit Convolutional Neural Networks (CNNs), um dadurch die Klassifikation von Dickdarmpolypen und die Qualit¨at von Iris Bildern (”Iris-Super-Resolution”) zu verbessern. Herk¨ommlicherweise verwenden Verfahren des maschinellen Lernens den gleichen Merkmalsraum und die gleiche Verteilung zum Trainieren und Testen der abgewendeten Methoden. Mehrere Probleme k¨onnen bei diesem Ansatz jedoch auftreten. Zum Beispiel ist es m¨ oglich, dass die Anzahl der zu trainierenden Daten (insbesondere in einem ”supervised training” Szenario) begrenzt ist. Im Speziellen im medizinischen Anwendungsfall ist man regelm¨aßig mit dem angesprochenen Problem konfrontiert, da die Zusammenstellung einer Datenbank, welche ¨ uber eine geeignete Anzahl an verwendbaren Daten verf ¨ ugt, entweder sehr kostspielig ist und/oder sich als ¨ uber die Maßen zeitaufw¨andig herausstellt. Ein anderes Problem betrifft die Verteilung von Strukturmerkmalen in einer Bilddatenbank, die zu groß sein kann, wie es im Fall der Verwendung von Texturmustern der menschlichen Iris auftritt. Dies kann zu dem Umstand f ¨ uhren, dass eine einzelne und sehr spezifische Trainingsdatenbank m¨oglicherweise nicht ausreichend verallgemeinert wird, um sie auf die gesamte betrachtete Dom¨ane anzuwenden. In dieser Arbeit wird die Verwendung von TL auf diverse Texturen untersucht, um die zuvor angesprochenen Probleme f ¨ ur zwei Anwendungen zu ¨ uberwinden: in der Klassifikation von Dickdarmpolypen und in Iris Super-Resolution. Die Hauptursache f ¨ ur Todesf¨alle im Zusammenhang mit dem Darmtrakt ist die Entwicklung von Krebszellen (Polypen) in vielen unterschiedlichen Auspr¨agungen. Eine Fr ¨uherkennung kann das Mortalit¨atsrisiko bei Patienten verringern, wenn sich der Krebs noch in einem fr ¨uhen Stadium befindet. Genauer gesagt, Dickdarmpolypen (gutartige Tumore oder Wucherungen, die an der inneren Dickdarmoberfl¨ache entstehen) haben ein hohes Vorkommen und sind bekanntermaßen Vorl¨aufer von Darmkrebsentwicklung. Mehrere Studien haben gezeigt, dass die automatische Erkennung und Klassifizierung von Bildregionen, die Polypen innerhalb des Dickdarms m¨oglicherweise enthalten, verwendet werden k¨onnen, um Spezialisten zu helfen, die Fehlerrate bei Polypen zu verringern. Die Klassifizierung kann sich jedoch aufgrund mehrerer Faktoren als eine schwierige Aufgabe herausstellen. ZumBeispiel kann das Fehlen oder ein U¨ bermaß an Beleuchtung zu starken Problemen hinsichtlich der Kontrastinformation der Bilder f ¨ uhren, wohingegen Unsch¨arfe aufgrund von Bewegung/Wassereinspritzung die Qualit¨at des Bildmaterials ebenfalls verschlechtert. Daten, welche ein unterschiedlich starkes Auftreten von Polypen repr¨asentieren, bieten auch dieM¨oglichkeit zu einer Reduktion der Klassifizierungsgenauigkeit. Weiters ist es sehr schwierig, einen robusten und vor allem globalen Feature-Extraktor zu finden, der all die notwendigen Pit-Pattern-Strukturen in einem einzigen Vektor zusammenfasst und darstellt. Um mit diesen Problemen ad¨aquat umzugehen, kann die Anwendung von CNNs eine gute Alternative bieten. Eines der Ziele dieser Arbeit ist es, die Wirksamkeit von CNNs, die von Grund auf f ¨ ur die Klassifikation von Dickdarmpolypen konstruiert wurden, zu zeigen. Des Weiteren soll die Anwendung von TL unter der Verwendung vorgefertigter CNNs f ¨ ur die Klassifikation von Dickdarmpolypen untersucht werden. Hierbei wird zus¨atzliche Information von nichtmedizinischen Bildern hinzugezogen und mit den verwendeten medizinischen Daten verbunden: Information wird also transferiert - TL entsteht. Auch in diesem Fall projiziert das CNN iii die Zieldatenbank (die Polypenbilder) in einen vorher trainierten Vektorraum, in dem die zu separierenden Klassen dann eher trennbar sind, daWissen aus den nicht-medizinischen Bildern einfließt. Der zweite Teil dieser Arbeit widmet sich dem TL hinsichtlich der Verbesserung der Bildqualit¨at von Iris Bilder - ”Iris- Super-Resolution”. Das Hauptziel von Super-Resolution (SR) ist es, aus einem oder mehreren Bildern gleichzeitig ein Bild mit einer h¨oheren Aufl¨osung (mit mehr Pixeln) zu erzeugen, welches dadurch zu einem detaillierteren und somit realistischeren Bild wird, wobei der visuelle Bildinhalt unver¨andert bleibt. Gegenw¨artig fordern die meisten Iris- Erkennungssysteme, dass der Benutzer seine Iris f ¨ ur den Sensor in geringer Entfernung pr¨asentiert. Jedoch ist es ein Anliegen der Industrie die bisher notwendigen Bedingungen - kurzer Abstand zwischen Sensor und Iris, sowie Verwendung von sehr teuren hochqualitativen Sensoren - zu ver¨andern. Diese Ver¨anderung betrifft einerseits die Verwendung von billigeren Sensoren und andererseits die Vergr¨oßerung des Abstandes zwischen Iris und Sensor. Beide Anpassungen f ¨ uhren zu Reduktion der Bildqualit¨at, was sich direkt auf die Erkennungsgenauigkeit der aktuell verwendeten Iris- erkennungssysteme auswirkt. In dieser Arbeit zeigen wir, dass die Verwendung von CNNs und TL f ¨ ur die ”Single Image Super-Resolution”, die bei der Iriserkennung angewendet wird, eine Alternative f ¨ ur die Iriserkennung von Bildern mit niedriger Aufl¨osung sein kann. Zu diesem Zweck untersuchen wir, ob die Art der Bilder sowie das Muster der Iris das CNN-TL beeinflusst und folglich die Ergebnisse im Erkennungsprozess ver¨andern kann.