Neste trabalho são apresentadas, complementadas e melhoradas duas técnicas de restauração de imagens: uma abordando o problema de retoque digital/remoção de objetos enquanto a segunda é direcionada ao problema deneliminação de ruído. Em ambas as técnicas, a ideia é trabalhar com imagens contendo texturas e outras características de interesse para um observador humano como a preservação de padrões, bordas, estruturas e regiões de natureza oscilatória. A técnica descrita sobre retoque digital de imagens combina difusão anisotrópica, síntese de texturas, busca dinâmica e um novo termo empregado no mecanismo de atribuição da ordem de prioridade durante o processo de reconstrução. Assim, dada uma imagem com regiões a serem recompostas, uma técnica de difusão anisotrópica é aplicada à imagem afim de se obter um mapa de saliência contendo bordas, estruturas e demais informações de baixa frequência da imagem. Na sequência, um mecanismo de prioridade baseado em um novo termo de confiabilidade regularizado é calculado a partir da combinação do mapa anteriormente gerado com a equação do transporte. Tal mecanismo é utilizado para determinar a ordem de preenchimento das partes faltantes da imagem. Para essa tarefa, a abordagem apresentada utiliza uma nova medida de similaridade entre blocos de pixels(amostrados dinamicamente para acelerar o processo), afim de encontrar os melhores candidatos a serem alocados nas regiões danificadas. A técnica destinada à remoção de ruídos alia a teoria da difusão anisotrópica, técnicas de análise harmônica e modelos numéricos de discretização de EDPs não-lineares em uma equação diferencial parcial regularizada, a qual atua de forma incisiva em regiões mais homogêneas da imagem e de forma mais suave em regiões caracterizadas como textura e bordas, preservando, assim, essas regiões. Além da natureza anisotrópica, a EDP procura recompor partes texturizadas perdidas no processo de eliminação de ruído através da aplicação de técnicas robustas de análise harmônica. Uma validação teórica e experimental para esta EDP e um estudo do ajuste paramétrico do método de eliminação de ruído baseado nesta EDP foram realizados neste trabalho. A eficiência e a performance das técnicas propostas são atestadas por meio das análises experimentais quantitativas e qualitativas com outras abordagens clássicas da literatura. / In this work two techniques of image restoration are presented, complemented and improved: one approaching the problem of image inpainting/object removal problem while the second one dealing with the image denoising problem. In both cases, the core idea is to process images containing textures and other features perceptible to a human observer such as patterns, contours, structures and oscillatory information. The image inpainting technique combines anisotropic diffusion, texture synthesis, dynamic search and a mechanism to set the order of priority during the image completion process. More precisely, given an image and target region to be inpainted, an anisotropic diffusion technique is applied in order to generate a saliency map containing edges, structures and other low frequency parts of the image. Next, apriority mechanism based on a new biased confidence term is computed from the map previously generated with the transport equation to define the level of priority of the pixels during the filling procedure. To accomplish this task, the presented approach employs a novel measure of similarity wich measures the distance between blocks of pixels (sampled dynamically to speed up the process) in order to find the best candidates to be allocated in the damaged regions. The technique devoted to denoising an image combines the theory of anisotropic diffusion, harmonic analysis techniques and numerical models into a regularized partial differential equation, which diffuses the pixels more incisively on homogeneous regions of the image while still seeking to attenuate regions formed by textures and patterns, thus preserving those information. Moreover, the proposed PDE aims at recovering texturized regions which have been degraded during the denoising process by employing harmonic analysis tools. A theoretical and experimental validation for this EDP and a study of the parametric adjustment of the image denoising method based on this EDP were performed in this work. The effectivenss and performance of the proposed approaches are attested through a comprehensive set of comparisons against other representative techniques in the literature.

{La première partie de cette thèse est dédiée à la modélisation d'images ou de vidéos considérés comme des fonds sur lesquels on s'attache à détecter des anomalies. Notre analyse de la littérature de la détection d'anomalie sur une seule image nous a fait identifier cinq différentes familles d'hypothèses structurelles sur le fond. Nous proposons de nouveaux algorithmes pour les problèmes de détection d'anomalie sur seule image, de détection de petites cibles sur un fond en mouvement, de détection de changements sur des images satellitaires SAR (Synthetic Aperture Radar) et de détection de nuages dans des séquences d'images de satellite optique.Dans une seconde partie, nous étudions deux autres applications de la modélisation de fond. Pour le débruitage vidéo, nous cherchons pour chaque patch de la vidéo, des patchs similaires le long de la séquence vidéo, et fournissons à un réseau de neurones convolutif les pixels centraux de ces patchs. Le modèle de fond est caché dans les poids du réseau de neurones. Cette méthode s'avère être la plus performante des méthodes par réseau de neurones comparées. Nous étudions également la synthèse de texture à partir d'un exemple. Dans ce problème, des échantillons de texture doivent être générés à partir d'un seul exemple servant de référence. Notre étude distingue les familles d'algorithmes en fonction du type de modèle adopté. Dans le cas des méthodes par réseau de neurones, nous proposons une amélioration corrigeant les artefacts de bord.Dans une troisième partie, nous proposons des implémentations temps-réel GPU de l'interpolation B-spline et de plusieurs algorithmes de débruitage d'images et de vidéo: NL-means, BM3D et VBM3D. La rapidité des implémentations proposées permet leur utilisation dans des scénarios temps-réel, et elles sont en cours de transfert vers l'industrie. / The first part of this thesis is dedicated to the modeling of image or video backgrounds, applied to anomaly detection. In the case of anomaly detection on a single image, our analysis leads us to find five different families of structural assumptions on the background. We propose new algorithms for single-image anomaly detection, small target detection on moving background, change detection on satellite SAR (Synthetic Aperture Radar) images and cloud detection on a sequence of satellite optical images.In the second part, we study two further applications of background modeling. To perform video denoising we search, for every video patch, similar patches in the video sequence, and feed their central pixels to a convolutional neural network (CNN). The background model in this case is hidden in the CNN weights. In our experiments, the proposed method is the best performing of the compared CNN-based methods. We also study exemplar-based texture synthesis. In this problem texture samples have to be generated based on only one reference sample. Our survey classifies the families of algorithms for this task according to their model assumptions. In addition, we propose improvements to fix the border behavior issues that we pointed out in several deep learning based methods.In the third part, we propose real-time GPU implementations for B-spline interpolation and for several image and video denoising algorithms: NL-means, BM3D and VBM3D. The speed of the proposed implementations enables their use in real-time scenarios, and they are currently being transitioned to industry.

This master's thesis focuses on the dynamic presentation of still photography using a depth map. This text presents an algorithm that describes the process of creating a spatial model which is used to render input photography so that the movement of virtual camera creates parallax effect due to depth in image. The thesis also presents an approach how to infill the missing data in the model. It is suggested that a guided texture synthesis is used for this problem by using rendering outputs of the model themselves as guides. Additional information in model allows the virtual camera to move more freely. The final result of the camera movement can be saved to simple video sequence which can be used for presenting the input photography.

Three Dimensional Video (3DV) has become increasingly popular with the success of 3D cinema. Moreover, emerging display technology offers an immersive experience to the viewer without the necessity of any visual aids such as 3D glasses. 3DV applications, Three Dimensional Television (3DTV) and Free Viewpoint Television (FTV) are auspicious technologies for living room environments by providing immersive experience and look around facilities. In order to provide such an experience, these technologies require a number of camera views captured from different viewpoints. However, the capture and transmission of the required number of views is not a feasible solution, and thus view rendering is employed as an efficient solution to produce the necessary number of views. Depth-image-based rendering (DIBR) is a commonly used rendering method. Although DIBR is a simple approach that can produce the desired number of views, inherent artifacts are major issues in the view rendering. Despite much effort to tackle the rendering artifacts over the years, rendered views still contain visible artifacts. This dissertation addresses three problems in order to improve 3DV quality: 1) How to improve the rendered view quality using a direct approach without dealing each artifact specifically. 2) How to handle disocclusions (a.k.a. holes) in the rendered views in a visually plausible manner using inpainting. 3) How to reduce spatial inconsistencies in the rendered view. The first problem is tackled by an edge-aided rendering method that uses a direct approach with one-dimensional interpolation, which is applicable when the virtual camera distance is small. The second problem is addressed by using a depth-based inpainting method in the virtual view, which reconstructs the missing texture with background data at the disocclusions. The third problem is undertaken by a rendering method that firstly inpaint occlusions as a layered depth image (LDI) in the original view, and then renders a spatially consistent virtual view. Objective assessments of proposed methods show improvements over the state-of-the-art rendering methods. Visual inspection shows slight improvements for intermediate views rendered from multiview videos-plus-depth, and the proposed methods outperforms other view rendering methods in the case of rendering from single view video-plus-depth. Results confirm that the proposed methods are capable of reducing rendering artifacts and producing spatially consistent virtual views. In conclusion, the view rendering methods proposed in this dissertation can support the production of high quality virtual views based on a limited number of input views. When used to create a multi-scopic presentation, the outcome of this dissertation can benefit 3DV technologies to improve the immersive experience.

Thèse numérisée par la Division de la gestion de documents et des archives de l'Université de Montréal.

This thesis explores synthesis by example as a paradigm for rendering real-world phenomena. In particular, phenomena that can be visually described as texture are considered. We exploit, for synthesis, the self-repeating nature of the visual elements constituting these texture exemplars. Techniques for unconstrained as well as constrained/controllable synthesis of both image and video textures are presented. For unconstrained synthesis, we present two robust techniques that can perform spatio-temporal extension, editing, and merging of image as well as video textures. In one of these techniques, large patches of input texture are automatically aligned and seamless stitched with each other to generate realistic looking images and videos. The second technique is based on iterative optimization of a global energy function that measures the quality of the synthesized texture with respect to the given input exemplar. We also present a technique for controllable texture synthesis. In particular, it allows for generation of motion-controlled texture animations that follow a specified flow field. Animations synthesized in this fashion maintain the structural properties like local shape, size, and orientation of the input texture even as they move according to the specified flow. We cast this problem into an optimization framework that tries to simultaneously satisfy the two (potentially competing) objectives of similarity to the input texture and consistency with the flow field. This optimization is a simple extension of the approach used for unconstrained texture synthesis. A general framework for example-based synthesis and rendering is also presented. This framework provides a design space for constructing example-based rendering algorithms. The goal of such algorithms would be to use texture exemplars to render animations for which certain behavioral characteristics need to be controlled. Our motion-controlled texture synthesis technique is an instantiation of this framework where the characteristic being controlled is motion represented as a flow field.

Thèse numérisée par la Division de la gestion de documents et des archives de l'Université de Montréal

Cette thèse s’inscrit dans le contexte de la synthèse d’images texturées. Dans l’objectif d’assurer une reproduction fidèle des motifs et des variations d’orientations d’une texture initiale, un algorithme de synthèse de texture à deux étapes « structure/texture » est proposé. Il s’agit, dans une première étape, de réaliser la synthèse d’une couche de structure caractérisant la géométrie de l’exemplaire et représentée par un champ de tenseurs de structure et, dans une deuxième étape, d’utiliser le champ de structure résultant pour contraindre la synthèse d’une couche de texture portant des variations plus locales. Une réduction du temps d’exécution est ensuite développée, fondée notamment sur l’utilisation de pyramides Gaussiennes et la parallélisation des calculs mis en oeuvre.Afin de démontrer la capacité de l’algorithme proposé à reproduire fidèlement l’aspect visuel des images texturées considérées, la méthode est testée sur une variété d’échantillons de texture et évaluée objectivement à l’aide de statistiques du 1er et du 2nd ordre du champ d’intensité et d’orientation. Les résultats obtenus sont de qualité supérieure ou équivalente à ceux obtenus par des algorithmes de la littérature. Un atout majeur de l’approche proposée est son aptitude à synthétiser des textures avec succès dans de nombreuses situations où les algorithmes existants ne parviennent pas à reproduire les motifs à grande échelle.L’approche de synthèse structure/texture proposée est étendue à la synthèse de texture couleur. La synthèse de texture 3D est ensuite abordée et, finalement, une extension à la synthèse de texture de forme spécifiée par une texture imposée est mise en oeuvre, montrant la capacité de l’approche à générer des textures de formes arbitraires en préservant les caractéristiques de la texture initiale. / This work is a part of the texture synthesis context. Aiming to ensure a faithful reproduction of the patterns and variations of orientations of the input texture, a two-stage structure/texture synthesis algorithm is proposed. It consists of synthesizing the structure layer showing the geometry of the exemplar and represented by the structure tensor field in the first stage, and using the resulting tensor field to constrain the synthesis of the texture layer holding more local variations, in the second stage. An acceleration method based on the use of Gaussian pyramids and parallel computing is then developed.In order to demonstrate the ability of the proposed algorithm to faithfully reproduce the visual aspect of the considered textures, the method is tested on various texture samples and evaluated objectively using statistics of 1st and 2nd order of the intensity and orientation field. The obtained results are of better or equivalent quality than those obtained using the algorithms of the literature. A major advantage of the proposed approach is its capacity in successfully synthesizing textures in many situations where traditional algorithms fail to reproduce the large-scale patterns.The structure/texture synthesis approach is extended to color texture synthesis. 3D texture synthesis is then addressed and finally, an extension to the synthesis of specified form textures using an imposed texture is carried out, showing the capacity of the approach in generating textures of arbitrary forms while preserving the input texture characteristics.

Dans le cadre de la classification de sons,cette thèse construit des représentations du signal qui vérifient des propriétés d’invariance et de variabilité inter-classe. D’abord, nous étudions le scattering temps- fréquence, une représentation qui extrait des modulations spectrotemporelles à différentes échelles. Enclassification de sons urbains et environnementaux, nous obtenons de meilleurs résultats que les réseaux profonds à convolutions et les descripteurs à court terme. Ensuite, nous introduisons le scattering en spirale, une représentation qui combine des transformées en ondelettes selon le temps, selon les log-fréquences, et à travers les octaves. Le scattering en spirale suit la géométrie de la spirale de Shepard, qui fait un tour complet à chaque octave. Nous étudions les sons voisés avec un modèle source-filtre non stationnaire dans lequel la source et le filtre sont transposés au cours du temps, et montrons que le scattering en spirale sépare et linéarise ces transpositions. Le scattering en spirale améliore lesperformances de l’état de l’art en classification d’instruments de musique. Outre la classification de sons, le scattering temps-fréquence et le scattering en spirale peuvent être utilisés comme des descripteurspour la synthèse de textures audio. Contrairement au scattering temporel, le scattering temps-fréquence est capable de capturer la cohérence de motifs spectrotemporels en bioacoustique et en parole, jusqu’à une échelle d’intégration de 500 ms environ. À partir de ce cadre d’analyse-synthèse, une collaboration artscience avec le compositeur Florian Hecker / This dissertation addresses audio classification by designing signal representations which satisfy appropriate invariants while preserving inter-class variability. First, we study time-frequencyscattering, a representation which extract modulations at various scales and rates in a similar way to idealized models of spectrotemporal receptive fields in auditory neuroscience. We report state-of-the-artresults in the classification of urban and environmental sounds, thus outperforming short-term audio descriptors and deep convolutional networks. Secondly, we introduce spiral scattering, a representationwhich combines wavelet convolutions along time, along log-frequency, and across octaves. Spiral scattering follows the geometry of the Shepard pitch spiral, which makes a full turn at every octave. We study voiced sounds with a nonstationary sourcefilter model where both the source and the filter are transposed through time, and show that spiral scattering disentangles and linearizes these transpositions. Furthermore, spiral scattering reaches state-of-the-art results in musical instrument classification ofsolo recordings. Aside from audio classification, time-frequency scattering and spiral scattering can be used as summary statistics for audio texture synthesis. We find that, unlike the previously existing temporal scattering transform, time-frequency scattering is able to capture the coherence ofspectrotemporal patterns, such as those arising in bioacoustics or speech, up to anintegration scale of about 500 ms. Based on this analysis-synthesis framework, an artisticcollaboration with composer Florian Hecker has led to the creation of five computer music

In computer graphics, textures are used to create detail along geometric surfaces. They are less computationally expensive than geometry, but this efficiency is traded for greater memory demands, especially with large output resolutions. Research has shown, that textures can be synthesized from low-resolution exemplars, reducing overall runtime memory cost and enabling applications, like remixing existing textures to create new, visually similar representations. In many modern applications, textures are not limited to simple images, but rather represent geometric detail in different ways, that describe how lights interacts at a certain point on a surface. Physically Based Rendering (PBR) is a technique, that employs complex lighting models to create effects like self-shadowing, realistic reflections or subsurface scattering. A set of multiple textures is used to describe what is called a material. In this thesis, example-based texture synthesis is extented to physical lighting models to create a physically based material synthesizer. It introduces a framework that is capable of utilizing multiple texture maps to synthesize new representations from existing material exemplars. The framework is then tested with multiple exemplars from different texture categories, to prospect synthesis performance in terms of quality and computation time. The synthesizer works in uv space, enabling to re-use the same exemplar material at runtime with different uv maps, reducing memory cost, whilst increasing visual varienty and minimizing repetition artifacts. The thesis shows, that this can be done effectively, without introducing inconsitencies like seams or discontiuities under dynamic lighting scenarios.:1. Context and Motivation 2. Introduction 2.1. Terminology: What is a Texture? 2.1.1. Classifying Textures 2.1.2. Characteristics and Appearance 2.1.3. Advanced Analysis 2.2. Texture Representation 2.2.1. Is there a theoretical Limit for Texture Resolution? 2.3. Texture Authoring 2.3.1. Texture Generation from Photographs 2.3.2. Computer-Aided Texture Generation 2.4. Introduction to Physically Based Rendering 2.4.1. Empirical Shading and Lighting Models 2.4.2. The Bi-Directional Reflectance Distribution Function (BRDF) 2.4.3. Typical Texture Representations for Physically Based Models 3. A brief History of Texture Synthesis 3.1. Algorithm Categories and their Developments 3.1.1. Pixel-based Texture Synthesis 3.1.2. Patch-based Texture Synthesis 3.1.3. Texture Optimization 3.1.4. Neural Network Texture Synthesis 3.2. The Purpose of example-based Texture Synthesis Algorithms 4. Framework Design 4.1. Dividing Synthesis into subsequent Stages 4.2. Analysis Stage 4.2.1. Search Space 4.2.2. Guidance Channel Extraction 4.3. Synthesis Stage 4.3.1. Synthesis by Neighborhood Matching 4.3.2. Validation 5. Implementation 5.1. Modules and Components 5.2. Image Processing 5.2.1. Image Representation 5.2.2. Filters and Guidance Channel Extraction 5.2.3. Search Space and Descriptors 5.2.4. Neighborhood Search 5.3. Implementing Synthesizers 5.3.1. Unified Synthesis Interface 5.3.2. Appearance Space Synthesis: A Hierarchical, Parallel, Per-Pixel Synthesizer 5.3.3. (Near-) Regular Texture Synthesis 5.3.4. Extented Appearance Space: A Physical Material Synthesizer 5.4. Persistence 5.4.1. Codecs 5.4.2. Assets 5.5. Command Line Sandbox 5.5.1. Providing Texture Images and Material Dictionaries 6. Experiments and Results 6.1. Test Setup 6.1.1. Metrics 6.1.2. Result Visualization 6.1.3. Limitations and Conventions 6.2. Experiment 1: Analysis Stage Performance 6.2.1. Influence of Exemplar Resolution 6.2.2. Influence of Exemplar Maps 6.3. Experiment 2: Synthesis Performance 6.3.1. Influence of Exemplar Resolution 6.3.2. Influence of Exemplar Maps 6.3.3. Influence of Sample Resolution 6.4. Experiment 3: Synthesis Quality 6.4.1. Influence of Per-Level Jitter 6.4.2. Influence of Exemplar Maps and Map Weights 7. Discussion and Outlook 7.1. Contributions 7.2. Further Improvements and Research 7.2.1. Performance Improvements 7.2.2. Quality Improvements 7.2.3. Methology 7.2.4. Further Problem Fields