Thesis (M.S.)--Kutztown University of Pennsylvania, 1991. / Source: Masters Abstracts International, Volume: 45-06, page: 2706. Abstract precedes thesis as [1] preliminary leaf. Typescript. Includes bibliographical references (leaves 66-67).

Image to Image translation has been triggered a huge interests among the different topics in deep learning recent years. It provides a mapping function to encode the noisy input images into a high dimensional signal and translate it to the desired output images. The mapping can be one to one, many to one or one to many. Due to the uncertainty from the mapping functions, when extend the methods in video field, the flickering problems emerges. Even a slight change among the frames may bring a obvious change in the output images. In this thesis, we provide a two-stream solution as CFNet for the flickering problems in video colorizations. Compared with the frame-by-frame methods by the previous work, CFNet has a great improvement in allevaiting the flickering problems in video colorizations, especially for the video clips with large objects and still background. Compared with the baseline with frame by frame methods, CFNet improved the PSNR from 27 to 30, which is a great progress.

Au cours de la dernière décennie, les technologies permettant la numérisation d'espaces urbains ont connu un développement rapide. Des campagnes d'acquisition de données couvrant des villes entières ont été menées en utilisant des scanners LiDAR (Light Detection And Ranging) installés sur des véhicules mobiles. Les résultats de ces campagnes d'acquisition laser, représentants les bâtiments numérisés, sont des nuages de millions de points pouvant également contenir un ensemble de photographies. On s'intéresse ici à l'amélioration du nuage de points à l'aide des données présentes dans ces photographies. Cette thèse apporte plusieurs contributions notables à cette amélioration. La position et l'orientation des images acquises sont généralement connues à l'aide de dispositifs embarqués avec le scanner LiDAR, même si ces informations de positionnement sont parfois imprécises. Pour obtenir un recalage précis d'une image sur un nuage de points, nous proposons un algorithme en deux étapes, faisant appel à l'information mutuelle normalisée et aux histogrammes de gradients orientés. Cette méthode permet d'obtenir une pose précise même lorsque les estimations initiales sont très éloignées de la position et de l'orientation réelles. Une fois ces images recalées, il est possible de les utiliser pour inférer la couleur de chaque point du nuage en prenant en compte la variabilité des points de vue. Pour cela, nous nous appuyons sur la minimisation d'une énergie prenant en compte les différentes couleurs associables à un point et les couleurs présentes dans le voisinage spatial du point. Bien entendu, les différences d'illumination lors de l'acquisition des données peuvent altérer la couleur à attribuer à un point. Notamment, cette couleur peut dépendre de la présence d'ombres portées amenées à changer avec la position du soleil. Il est donc nécessaire de détecter et de corriger ces dernières. Nous proposons une nouvelle méthode qui s'appuie sur l'analyse conjointe des variations de la réflectance mesurée par le LiDAR et de la colorimétrie des points du nuage. En détectant suffisamment d'interfaces ombre/lumière nous pouvons caractériser la luminosité de la scène et la corriger pour obtenir des scènes sans ombre portée. Le dernier problème abordé par cette thèse est celui de la densification du nuage de points. En effet la densité locale du nuage de points est variable et parfois insuffisante dans certaines zones. Nous proposons une approche applicable directement par la mise en oeuvre d'un filtre bilatéral joint permettant de densifier le nuage de points en utilisant les données des images / Recent years saw a rapid development of city digitization technologies. Acquisition campaigns covering entire cities are now performed using LiDAR (Light Detection And Ranging) scanners embedded aboard mobile vehicles. These acquisition campaigns yield point clouds, composed of millions of points, representing the buildings and the streets, and may also contain a set of images of the scene. The subject developed here is the improvement of the point cloud using the information contained in the camera images. This thesis introduces several contributions to this joint improvement. The position and orientation of acquired images are usually estimated using devices embedded with the LiDAR scanner, even if this information is inaccurate. To obtain the precise registration of an image on a point cloud, we propose a two-step algorithm which uses both Mutual Information and Histograms of Oriented Gradients. The proposed method yields an accurate camera pose, even when the initial estimations are far from the real position and orientation. Once the images have been correctly registered, it is possible to use them to color each point of the cloud while using the variability of the point of view. This is done by minimizing an energy considering the different colors associated with a point and the potential colors of its neighbors. Illumination changes can also change the color assigned to a point. Notably, this color can be affected by cast shadows. These cast shadows are changing with the sun position, it is therefore necessary to detect and correct them. We propose a new method that analyzes the joint variation of the reflectance value obtained by the LiDAR and the color of the points. By detecting enough interfaces between shadow and light, we can characterize the luminance of the scene and to remove the cast shadows. The last point developed in this thesis is the densification of a point cloud. Indeed, the local density of a point cloud varies and is sometimes insufficient in certain areas. We propose a directly applicable approach to increase the density of a point cloud using multiple images

Image segmentation is an essential tool to enhance the ability of computer systems to efficiently perform elementary cognitive tasks such as detection, recognition and tracking. In this thesis we concentrate on the investigation of two fundamental topics in the context of image segmentation: spectral clustering and seeded image segmentation. We introduce two new algorithms for those topics that, in summary, rely on Laplacian-based operators, spectral graph theory, and minimization of energy functionals. The effectiveness of both segmentation algorithms is verified by visually evaluating the resulting partitions against state-of-the-art methods as well as through a variety of quantitative measures typically employed as benchmark by the image segmentation community. Our spectral-based segmentation algorithm combines image decomposition, similarity metrics, and spectral graph theory into a concise and powerful framework. An image decomposition is performed to split the input image into texture and cartoon components. Then, an affinity graph is generated and weights are assigned to the edges of the graph according to a gradient-based inner-product function. From the eigenstructure of the affinity graph, the image is partitioned through the spectral cut of the underlying graph. Moreover, the image partitioning can be improved by changing the graph weights by sketching interactively. Visual and numerical evaluation were conducted against representative spectral-based segmentation techniques using boundary and partition quality measures in the well-known BSDS dataset. Unlike most existing seed-based methods that rely on complex mathematical formulations that typically do not guarantee unique solution for the segmentation problem while still being prone to be trapped in local minima, our segmentation approach is mathematically simple to formulate, easy-to-implement, and it guarantees to produce a unique solution. Moreover, the formulation holds an anisotropic behavior, that is, pixels sharing similar attributes are preserved closer to each other while big discontinuities are naturally imposed on the boundary between image regions, thus ensuring better fitting on object boundaries. We show that the proposed approach significantly outperforms competing techniques both quantitatively as well as qualitatively, using the classical GrabCut dataset from Microsoft as a benchmark. While most of this research concentrates on the particular problem of segmenting an image, we also develop two new techniques to address the problem of image inpainting and photo colorization. Both methods couple the developed segmentation tools with other computer vision approaches in order to operate properly. / Segmentar uma image é visto nos dias de hoje como uma prerrogativa para melhorar a capacidade de sistemas de computador para realizar tarefas complexas de natureza cognitiva tais como detecção de objetos, reconhecimento de padrões e monitoramento de alvos. Esta pesquisa de doutorado visa estudar dois temas de fundamental importância no contexto de segmentação de imagens: clusterização espectral e segmentação interativa de imagens. Foram propostos dois novos algoritmos de segmentação dentro das linhas supracitadas, os quais se baseiam em operadores do Laplaciano, teoria espectral de grafos e na minimização de funcionais de energia. A eficácia de ambos os algoritmos pode ser constatada através de avaliações visuais das segmentações originadas, como também através de medidas quantitativas computadas com base nos resultados obtidos por técnicas do estado-da-arte em segmentação de imagens. Nosso primeiro algoritmo de segmentação, o qual ´e baseado na teoria espectral de grafos, combina técnicas de decomposição de imagens e medidas de similaridade em grafos em uma única e robusta ferramenta computacional. Primeiramente, um método de decomposição de imagens é aplicado para dividir a imagem alvo em duas componentes: textura e cartoon. Em seguida, um grafo de afinidade é gerado e pesos são atribuídos às suas arestas de acordo com uma função escalar proveniente de um operador de produto interno. Com base no grafo de afinidade, a imagem é então subdividida por meio do processo de corte espectral. Além disso, o resultado da segmentação pode ser refinado de forma interativa, mudando-se, desta forma, os pesos do grafo base. Experimentos visuais e numéricos foram conduzidos tomando-se por base métodos representativos do estado-da-arte e a clássica base de dados BSDS a fim de averiguar a eficiência da metodologia proposta. Ao contrário de grande parte dos métodos existentes de segmentação interativa, os quais são modelados por formulações matemáticas complexas que normalmente não garantem solução única para o problema de segmentação, nossa segunda metodologia aqui proposta é matematicamente simples de ser interpretada, fácil de implementar e ainda garante unicidade de solução. Além disso, o método proposto possui um comportamento anisotrópico, ou seja, pixels semelhantes são preservados mais próximos uns dos outros enquanto descontinuidades bruscas são impostas entre regiões da imagem onde as bordas são mais salientes. Como no caso anterior, foram realizadas diversas avaliações qualitativas e quantitativas envolvendo nossa técnica e métodos do estado-da-arte, tomando-se como referência a base de dados GrabCut da Microsoft. Enquanto a maior parte desta pesquisa de doutorado concentra-se no problema específico de segmentar imagens, como conteúdo complementar de pesquisa foram propostas duas novas técnicas para tratar o problema de retoque digital e colorização de imagens.

[pt] Esta dissertação discute a aplicação de efeitos de colorização 3D a animações 2D produzidas pela técnica quadro-a-quadro. Utilizando algoritmos de processamento de imagens, desenhos 2D são preparados para receber técnicas de sombreamento evitando a transformação da cena para uma geometria 3D. Esta preparação se dá através da obtenção de mapas de normais que aproximam a geometria do desenho. O sombreamento é obtido através de um conjunto de técnicas de renderização foto-realistas e não-foto-realistas, que podem ser adaptadas para utilização de normais aproximadas. Visando amenizar o trabalho exaustivo de colorir cada desenho, um método baseado no relacionamento entre imagens é apresentado para colorir automaticamente cada quadro numa seqüência de desenhos. Este processo de colorização considera a necessidade de possíveis intervenções humanas para garantir a qualidade final de cada imagem da animação. Um estudo sobre aproximação de normais, técnicas de sombreamento, segmentação de imagens e rastreamento de objetos é amplamente discutido nesta dissertação. / [en] This dissertation discusses the 3D colorization effects usage over a 2d animation, which has been produced through frame-by-frame techniques. Normal vector maps approximates the drawing geometry and provide the ability to perform shading effects by applying digital image processing algorithms, avoiding 3D geometry scene transformation. A set of photorealistic and non-photorealistic renderization techniques, which can be adapted to normal approximation usage, is proposed in the colorization process. Also, a method based on interframe dependence is presented, aiming to reduce the thoroughgoing effort of colorizing each individual frame within an animation. This colorization process considers possible human interventions to ensure image´s result quality. Finally, this dissertation provides a comprehensive study regarding several topics, such as normal approximations, shading techniques, image segmentation and object tracking.

Image segmentation is an essential tool to enhance the ability of computer systems to efficiently perform elementary cognitive tasks such as detection, recognition and tracking. In this thesis we concentrate on the investigation of two fundamental topics in the context of image segmentation: spectral clustering and seeded image segmentation. We introduce two new algorithms for those topics that, in summary, rely on Laplacian-based operators, spectral graph theory, and minimization of energy functionals. The effectiveness of both segmentation algorithms is verified by visually evaluating the resulting partitions against state-of-the-art methods as well as through a variety of quantitative measures typically employed as benchmark by the image segmentation community. Our spectral-based segmentation algorithm combines image decomposition, similarity metrics, and spectral graph theory into a concise and powerful framework. An image decomposition is performed to split the input image into texture and cartoon components. Then, an affinity graph is generated and weights are assigned to the edges of the graph according to a gradient-based inner-product function. From the eigenstructure of the affinity graph, the image is partitioned through the spectral cut of the underlying graph. Moreover, the image partitioning can be improved by changing the graph weights by sketching interactively. Visual and numerical evaluation were conducted against representative spectral-based segmentation techniques using boundary and partition quality measures in the well-known BSDS dataset. Unlike most existing seed-based methods that rely on complex mathematical formulations that typically do not guarantee unique solution for the segmentation problem while still being prone to be trapped in local minima, our segmentation approach is mathematically simple to formulate, easy-to-implement, and it guarantees to produce a unique solution. Moreover, the formulation holds an anisotropic behavior, that is, pixels sharing similar attributes are preserved closer to each other while big discontinuities are naturally imposed on the boundary between image regions, thus ensuring better fitting on object boundaries. We show that the proposed approach significantly outperforms competing techniques both quantitatively as well as qualitatively, using the classical GrabCut dataset from Microsoft as a benchmark. While most of this research concentrates on the particular problem of segmenting an image, we also develop two new techniques to address the problem of image inpainting and photo colorization. Both methods couple the developed segmentation tools with other computer vision approaches in order to operate properly. / Segmentar uma image é visto nos dias de hoje como uma prerrogativa para melhorar a capacidade de sistemas de computador para realizar tarefas complexas de natureza cognitiva tais como detecção de objetos, reconhecimento de padrões e monitoramento de alvos. Esta pesquisa de doutorado visa estudar dois temas de fundamental importância no contexto de segmentação de imagens: clusterização espectral e segmentação interativa de imagens. Foram propostos dois novos algoritmos de segmentação dentro das linhas supracitadas, os quais se baseiam em operadores do Laplaciano, teoria espectral de grafos e na minimização de funcionais de energia. A eficácia de ambos os algoritmos pode ser constatada através de avaliações visuais das segmentações originadas, como também através de medidas quantitativas computadas com base nos resultados obtidos por técnicas do estado-da-arte em segmentação de imagens. Nosso primeiro algoritmo de segmentação, o qual ´e baseado na teoria espectral de grafos, combina técnicas de decomposição de imagens e medidas de similaridade em grafos em uma única e robusta ferramenta computacional. Primeiramente, um método de decomposição de imagens é aplicado para dividir a imagem alvo em duas componentes: textura e cartoon. Em seguida, um grafo de afinidade é gerado e pesos são atribuídos às suas arestas de acordo com uma função escalar proveniente de um operador de produto interno. Com base no grafo de afinidade, a imagem é então subdividida por meio do processo de corte espectral. Além disso, o resultado da segmentação pode ser refinado de forma interativa, mudando-se, desta forma, os pesos do grafo base. Experimentos visuais e numéricos foram conduzidos tomando-se por base métodos representativos do estado-da-arte e a clássica base de dados BSDS a fim de averiguar a eficiência da metodologia proposta. Ao contrário de grande parte dos métodos existentes de segmentação interativa, os quais são modelados por formulações matemáticas complexas que normalmente não garantem solução única para o problema de segmentação, nossa segunda metodologia aqui proposta é matematicamente simples de ser interpretada, fácil de implementar e ainda garante unicidade de solução. Além disso, o método proposto possui um comportamento anisotrópico, ou seja, pixels semelhantes são preservados mais próximos uns dos outros enquanto descontinuidades bruscas são impostas entre regiões da imagem onde as bordas são mais salientes. Como no caso anterior, foram realizadas diversas avaliações qualitativas e quantitativas envolvendo nossa técnica e métodos do estado-da-arte, tomando-se como referência a base de dados GrabCut da Microsoft. Enquanto a maior parte desta pesquisa de doutorado concentra-se no problema específico de segmentar imagens, como conteúdo complementar de pesquisa foram propostas duas novas técnicas para tratar o problema de retoque digital e colorização de imagens.

Cette thèse traite de problèmes liés à la couleur. En particulier, on s’intéresse à des problématiques communes à la colorisation d’images, de vidéos et au rehaussement de contraste. Si on considère qu’une image est composée de deux informations complémentaires, une achromatique (sans couleur) et l’autre chromatique (en couleur), les applications étudiées consistent à traiter une de ces deux informations en préservant sa complémentaire. En colorisation, la difficulté est de calculer une image couleur en imposant son niveau de gris. Le rehaussement de contraste vise à modifier l’intensité d’une image en préservant sa teinte. Ces problématiques communes nous ont conduits à étudier formellement la géométrie de l’espace RGB. On a démontré que les espaces couleur classiques de la littérature pour résoudre ces types de problème conduisent à des erreurs. Un algorithme, appelé spécification luminance-teinte, qui calcule une couleur ayant une teinte et une luminance données est décrit dans cette thèse. L’extension de cette méthode à un cadre variationnel a été proposée. Ce modèle a été utilisé avec succès pour rehausser les images couleur, en utilisant des hypothèses connues sur le système visuel humain. Les méthodes de l’état-de-l’art pour la colorisation d’images se divisent en deux catégories. La première catégorie regroupe celles qui diffusent des points de couleurs posés par l’utilisateur pour obtenir une image colorisée (colorisation manuelle). La seconde est constituée de celles qui utilisent une image couleur de référence ou une base d’images couleur et transfèrent les couleurs de la référence sur l’image en niveaux de gris (colorisation basée exemple). Les deux types de méthodes ont leurs avantages et inconvénients. Dans cette thèse, on propose un modèle variationnel pour la colorisation basée exemple. Celui-ci est étendu en une méthode unifiant la colorisation manuelle et basée exemple. Enfin, nous décrivons des modèles variationnels qui colorisent des vidéos tout en permettent une interaction avec l’utilisateur. / This thesis deals with problems related to color. In particular, we are interested inproblems which arise in image and video colorization and contrast enhancement. When considering color images composed of two complementary information, oneachromatic (without color) and the other chromatic (in color), the applications studied in this thesis are based on the processing one of these information while preserving its complement. In colorization, the challenge is to compute a color image while constraining its gray-scale channel. Contrast enhancement aims to modify the intensity channel of an image while preserving its hue.These joined problems require to formally study the RGB space geometry. In this work, it has been shown that the classical color spaces of the literature designed to solve these classes of problems lead to errors. An novel algorithm, called luminance-hue specification, which computes a color with a given hue and luminance is described in this thesis. The extension of this method to a variational framework has been proposed. This model has been used successfully to enhance color images, using well-known assumptions about the human visual system. The state-of-the-art methods for image colorization fall into two categories. The first category includes those that diffuse color scribbles drawn by the user (manual colorization). The second consists of those that benefits from a reference color image or a base of reference images to transfer the colors from the reference to the grayscale image (exemplar-based colorization). Both approach have their advantages and drawbacks. In this thesis, we design a variational model for exemplar-based colorization which is extended to a method unifying the manual colorization and the exemplar-based one. Finally, we describe two variational models to colorize videos in interaction with the user.

El cine de terror y sus subgéneros son de los géneros más taquilleros en el Perú, sin embargo, el cine comercial solo ha tenido en cuenta el generar ingresos ocasionando una necesidad por una cartelera que combine películas atractivas para el consumidor general con productos con narrativas potentes y visuales. La dirección de fotografía juega un papel importante en el desarrollo de una producción tanto atractiva como significativa artísticamente, generando impacto a través de elementos como el encuadre, la iluminación y la colonización. Darren Aronofksy es un cineasta del subgénero del thriller que ha logrado contar historias altamente visuales y perturbadoras que no solo generan reacciones emotivas sino sensoriales, siendo considerado dentro de lo que se denomina “el cine de los cuerpos”, que compromete tanto las emociones como sensaciones fisiológicas en el espectador a través de áreas como la dirección de fotografía. / Horror cinema and its sub-genres are some of the highest grossing genres in Peru, however, commercial cinema has only taken into account generating income, which led to a need for a billboard that combines attractive films for the mainstream consumer with products with powerful narratives and visuals. The direction of photography plays an important role in the development of both an attractive and artistically significant production, generating impact through elements such as framing, lighting and colorization. Darren Aronofksy is a filmmaker of the thriller subgenre who has managed to tell highly visual and disturbing stories that not only generate emotional but also sensory reactions, being considered within what is called "the cinema of bodies", which involves both emotions and physiological sensations in the viewer through areas such as the direction of photography. / Trabajo de investigación

This thesis deals with image colorization and image super-resolution using neural networks. It briefly explains neural networks principles and summarizes current approaches in this domain. It also describes the design, implementation and training of various neural network architectures. The best implemented architecture can colorize images, in particular, works well with outdoor areas. The architecture for image super-resolution with residual blocks that was trained with a perceptual loss function performs a double increase in image resolution (4x more pixels in total). Part of this thesis is also an implementation of a web application that uses trained models for image modification.

Casamento de padrões de pontos é um problema fundamental em reconhecimento de padrões. O objetivo é encontrar uma correspondência entre dois conjuntos de pontos, associados a características relevantes de objetos ou entidades, mapeando os pontos de um conjunto no outro. Este problema está associado a muitas aplicações, como por exemplo, reconhecimento de objetos baseado em modelos, imagens estéreo, registro de imagens, biometria, entre outros. Para encontrar um mapeamento, os objetos são codificados por representações abstratas, codificando as características relevantes consideradas na comparação entre pares de objetos. Neste trabalho, objetos são representados por grafos, codificando tanto as características `locais\' quanto as relações espaciais entre estas características. A comparação entre objetos é guiada por uma formulação de atribuição quadrática, que é um problema NP-difícil. Para estimar uma solução, duas técnicas de casamento entre grafos são propostas: uma baseada em grafos auxiliares, chamados de grafos deformados; e outra baseada em representações `esparsas\', campos aleatórios de Markov e propagação de crenças. Devido as suas respectivas limitações, as abordagens são adequadas para situações específicas, conforme mostrado neste documento. Resultados envolvendo as duas abordagens são ilustrados em quatro importantes aplicações: casamento de imagens de gel eletroforese 2D, segmentação interativa de imagens naturais, casamento de formas, e colorização assistida por computador. / Point set matching is a fundamental problem in pattern recognition. The goal is to match two sets of points, associated to relevant features of objects or entities, by finding a mapping, or a correspondence, from one set to another set of points. This issue arises in many applications, e.g. model-based object recognition, stereo matching, image registration, biometrics, among others. In order to find a mapping, the objects can be encoded by abstract representations, carrying relevant features which are taken into account to compare pairs of objects. In this work, graphs are adopted to represent the objects, encoding their `local\' features and the spatial relations between these features. The comparison of two given objects is guided by a quadratic assignment formulation, which is NP-hard. In order to estimate the optimal solution, two approximations techniques, via graph matching, are proposed: one is based on auxiliary graphs, called deformed graphs; the other is based on `sparse\' representations, Markov random fields and belief propagation. Due to their respective limitations, each approach is more suitable to each specific situation, as shown in this document. The quality of the two approaches is illustrated on four important applications: 2D electrophoresis gel matching, interactive natural image segmentation, shape matching, and computer-assisted colorization.