[pt] A degradação da qualidade do áudio pode ter muitas causas. Para aplicações musicais, esta fragmentação pode levar a experiências altamente desagradáveis. Algoritmos de restauração podem ser empregados para reconstruir partes do áudio de forma semelhante à reconstrução da imagem, em uma abordagem chamada Audio Inpainting. Os métodos atuais de última geração para Audio Inpainting cobrem cenários limitados, com janelas de intervalo bem definidas e pouca variedade de gêneros musicais. Neste trabalho, propomos um método baseado em aprendizado profundo para Audio Inpainting acompanhado por um conjunto de dados com condições de fragmentação aleatórias que se aproximam de situações reais de deficiência. O conjunto de dados foi coletado utilizando faixas de diferentes gêneros musicais, o que proporciona uma boa variabilidade de sinal. Nosso melhor modelo melhorou a qualidade de todos os gêneros musicais, obtendo uma média de 13,1 dB de PSNR, embora tenha funcionado melhor para gêneros musicais nos quais os instrumentos acústicos são predominantes. / [en] Audio quality degradation can have many causes. For musical applications, this fragmentation may lead to highly unpleasant experiences. Restoration algorithms may be employed to reconstruct missing parts of the audio in a similar way as for image reconstruction - in an approach called audio inpainting. Current state-of-theart methods for audio inpainting cover limited scenarios, with well-defined gap windows and little variety of musical genres. In this work, we propose a Deep-Learning-based (DLbased) method for audio inpainting accompanied by a dataset with random fragmentation conditions that approximate real impairment situations. The dataset was collected using tracks from different music genres to provide a good signal variability. Our best model improved the quality of all musical genres, obtaining an average of 13.1 dB of PSNR, although it worked better for musical genres in which acoustic instruments are predominant.

Made available in DSpace on 2015-03-05T13:58:27Z (GMT). No. of bitstreams: 0 Previous issue date: 26 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / Inpainting Digital é uma técnica recente que permite completar a falta de informação em imagens, seja por falha ou por remoção intencional de alguma área ou objeto. Uma das atribuições importantes do inpainting digital é de que deve ser capaz de alterar uma imagem, de forma que não seja simples perceber que esta alteração foi feita; caracteriza uma modificação indetectável. Os métodos para determinar como esta falta de informação será preenchida variam desde a criação do primeiro modelo de inpainting digital. Contudo, sempre deve ser mantida uma coerência no preenchimento, que fará com que a região preenchida automaticamente aparente como parte da imagem verdadeira. As técnicas atuais tratam este preenchimento como uma propagação da estrutura da área que está ao redor da região a ser preenchida, trabalhando diretamente no domímio das cores, utilizando abordagens como Variação Total e Equações Diferenciais Parciais. Neste trabalho, é feito o uso de transformada Wavelet para a aplicação de inpainting digita / Digital Inpainting is a recent techinique that allows the filling of missing information in images. One important attribute of a digital inpainting technique is the ability of altering an image in such a way that it is not simple for the human observer to detect the modification, characterizing an undetectable modification. The strategies for filling missing parts vary since the first inpainting model, but one thing that remains is the fact that the filled area must be coherent with the original part of the image. Current techniques handle the filling as a structure propagation problem, working directly in the image color domain, and based on concepts like Total Variation or Partial Diferential Equations. In this work, we present a digital inpainting model that works exclusively in Wavelet domain,filling the target area with a texture synthesis mechanism using the properties of the Wavelet Transform

Par opposition à la Réalité Augmentée qui consiste à ajouter des éléments virtuels à un environnement réel, la Réalité Diminuée consiste à supprimer des éléments réels d'un environnement. Le but est d'effectuer un rendu visuel d'une scène 3D où les éléments "effacés" ne sont plus présents : la difficulté consiste à créer une image de sorte que la diminution ne soit pas perceptible par l'utilisateur. Il faut donc venir compléter la scène initialement cachée par ces éléments, en effectuant une opération d'inpainting qui prenne en compte la géométrie de la pièce, sa texture (structurée ou non), et la luminosité ambiante de l'environnement. Par exemple, l’œil humain est sensible à la régularité d'une texture. L'un des objectifs d'Innersense, entreprise spécialisée dans l'aménagement virtuel d’intérieurs, est de développer un produit capable d'enlever des éléments présents dans une pièce d'intérieur. Une fois la suppression virtuelle des meubles existants effectuée , il sera alors possible d'ajouter des meubles virtuels dans l'espace laissé vacant. L'objectif de cette thèse CIFRE est donc de mettre en place un scénario de réalité diminuée pouvant être exécuté sur un système mobile (tablette IOS ou Android) qui génère des images photo-réalistes de la scène diminuée. Pour cela, à partir d’un modèle géométrique de la pièce d'intérieur que l'on veut altérer, nous adaptons et améliorons des procédures d'effacement d'éléments d'une image appelées inpainting dans une image 2D. Ensuite, nous appliquons ces techniques dans le contexte 3D intérieur pour tenir compte de la géométrie de la scène. Enfin, nous analysons la luminosité pour augmenter le réalisme des zones complétées.Dans cette thèse, nous rappelons d'abord les différents travaux académiques et les solutions industrielles existantes. Nous évoquons leurs avantages et leurs limites. Nous abordons ensuite les différentes techniques d'inpainting existantes pour introduire notre première contribution qui propose d'adapter une des méthodes de l’état de l’art pour prendre en compte de la structure du motif de la texture. La problématique de la luminosité est ensuite abordée en proposant un processus qui traite séparément la texture et la variation de la luminosité. Nous présentons ensuite une troisième contribution qui propose un critère de confiance basé sur des considérations radiométriques pour sélectionner une information selon sa qualité dans le processus d'inpainting. Nous proposons une dernière contribution basée sur la complétion de texture de modèles 3D non planaires reconstruits à partir de peu d’images et donc présentant une texture incomplète. Enfin, nous montrons les applications développées grâce à ces travaux dans le contexte des scènes d'intérieur considérées par Innersense / In contrast to Augmented Reality, which consists in adding virtual elements to a real environment,Diminished Reality consists in removing real elements from an environment. The goal is to visuallyrender a 3D scene where the "deleted" elements are no longer present: the difficulty is to createan image so that the processing is not perceptible to the user. It is therefore necessary tocomplete the scene initially hidden by these elements, by performing an inpainting operation thattakes into account the geometry of the part, its texture (structured or not), and the ambientbrightness of the environment. For example, the human eye is sensitive to the regularity of atexture. One of the objectives of Innersense, a company specializing in virtual interior design, is todevelop a product that can remove elements from an interior room. Once the virtual removal ofexisting furniture has been completed, it will then be possible to add virtual furniture in the vacantspace. The objective of this CIFRE thesis is therefore to set up a scenario of diminished realitythat can be executed on a mobile system (IOS or Android tablet) that generates photorealisticimages of the diminished scene. To do this, based on a geometric model of the interior part thatwe want to alter, we adapt and improve procedures for erasing elements of an image calledinpainting in a 2D image. Then, we apply these techniques in the 3D indoor context to take intoaccount the geometry of the scene. Finally, we analyze the brightness to increase the realism ofthe completed areas. In this thesis, we first review the various academic works and existingindustrial solutions. We discuss their advantages and limitations. We then discuss the differentexisting inpainting techniques to introduce our first contribution which proposes to adapt one of thestate of the art methods to take into account the structure of the texture pattern. The problem ofbrightness is then discussed by proposing a process that deals separately with texture andvariation of brightness. We then present a third contribution that proposes a confidence criterionbased on radiometric considerations to select information according to its quality in the inpaintingprocess. We propose a last contribution based on the texture completion of non-planar 3D modelsreconstructed from few images and therefore presenting an incomplete texture. Finally, we showthe applications developed through this work in the context of the interior scenes considered byInnersense.

[pt] Nos últimos anos tem ocorrido um notável desenvolvimento de técnicas de Image Inpainting, entretanto transpor esse conhecimento para aplicações em vídeo tem se mostrado um desafio. Além dos desafios inerentes a tarefa de Video Inpainting (VI), utilizar essa técnica requer um trabalho prévio de anotação da área que será reconstruída. Se a aplicação do método for para remover um objeto ao longo de um vídeo, então a anotação prévia deve ser uma máscara da área deste objeto frame a frame. A tarefa de propagar a anotação de um objeto ao longo de um vídeo é conhecida como Video Object Segmentation (VOS) e já existem técnicas bem desenvolvidas para solucionar este problemas. Assim, a proposta desse trabalho é aplicar técnicas de VOS para gerar insumo para um algoritmo de VI. Neste trabalho iremos analisar o impacto de utilizar anotações preditas no resultado final de um modelo de VI. / [en] In recent years there has been a remarkable development of Image Inpainting techniques, but using this knowledge in video application is still a challenge. Besides the inherent challenges of the Video Inpainting (VI) task, applying this technique requires a previous job of labeling the area that should be reconstructed. If this method is used to remove an object from the video, then the annotation should be a mask of this object s area frame by frame. The task of propagating an object mask in a video is known as Video Object Segmentation (VOS) and there are already well developed techniques to solve this kind of task. Therefore, this work aims to apply VOS techniques to create the inputs for an VI algorithm. In this work we shall analyse the impact in the result of a VI algorithm when we use a predicted annotation as the input.

Image inpainting is the process of recovering missing or deteriorated data within the digital images and videos in a plausible way. It has become an important topic in the area of image processing, which leads to the understanding of the textural and structural information within the images. Image inpainting has many different applications, such as image/video restoration, text/object removal, texture synthesis, and transmission error concealment. In recent years, many algorithms have been developed to solve the image inpainting problem, which can be roughly grouped into four categories, partial differential equation-based inpainting, exemplar-based inpainting, transform domain inpainting, and hybrid image inpainting. However, the existing algorithms do not work well when the missing region to be inpainted is large, and when there are textural and structural information needed to be recovered. To address this inpainting problem, we propose multiple algorithms, 1) perceptually aware image inpainting based on the perceptual-fidelity aware mean squared error metric, 2) image inpainting using nonlocal texture matching and nonlinear filtering, and 3) multiresolution exemplar-based image inpainting. The experimental results show that our proposed algorithms outperform other existing algorithms with respect to both qualitative analysis and observer studies when inpainting the missing regions of images.

This report is a master thesis, written by an engineering physics and electrical engineering student at Luleå University of Technology.The desires of this project was to remove dislocations from wrapped phase maps using sparse reconstructive methods. Dislocations is an error that can appear in phase maps due to improper filtering or inadequate sampling. Dislocations makes it impossible to correctly unwrap the phasemap.The report contains a mathematical description of a sparse reconstructive method. The sparse reconstructive method is based on KSVDbox which was created by R. Rubinstein and is free for download and use. The KSVDbox is a MATLAB implementation of a dictionary learning algorithm called K-SVD with Orthogonal Matching Pursuit and a sparse reconstructive algorithm. A guide for adapting the toolbox for inpainting is included, with a couple of examples on natural images which supports the suggested adaptation. For experimental purposes a set of simulated wrapped phase maps with and without disloca-tions were created. These simulated phase maps are based on work by P. Picart. The MATLAB implementation that was used to generate these test images can be found in the appendix of this report such that they can easily be generated by anyone who has the interest to do so. Finally the report leads to an outline of five different experiments that was designed to test the KSVDbox for the processing of dislocations. Each one of these experiments uses a different dictionary. These experiments are due to inpainting with, 1. A dictionary based on Discrete Cosine Transform. 2. An adaptive dictionary, where the dictionary learning algorithm has been shown what thearea in the phase map that was damaged by dislocations should look like. 3. An adaptive dictionary, where the dictionary learning algorithm has been allowed to trainon the phase map that with damages. This is done such that areas with dislocations areignored. 4. An adaptive dictionary, where training is done on a separate image that has been designedto contain general phase patterns. 5. An adaptive dictionary, that results from concatenating the dictionaries used in experiment 3 and 4. The first three experiments are complimented with experiments done on a natural image for comparison purposes.The results show that sparse reconstructive methods, when using the scheme used in this work, is unsuitable for processing of dislocations in phase maps. This is most likely because the reconstructive method has difficulties in acquiring a high contrast reconstruction and there is nothing in the algorithm that causes the inpainting from any direction to match with the inpainting from other directions.

This work applies sparse representations and nonlinear image processing to two inverse imaging problems. The first problem involves image restoration, where the aim is to reconstruct an unknown high-quality image from a low-quality observed image. Sparse representations of images have drawn a considerable amount of interest in recent years. The assumption that natural signals, such as images, admit a sparse decomposition over a redundant dictionary leads to efficient algorithms for handling such sources of data. The standard sparse representation, however, does not consider the intrinsic geometric structure present in the data, thereby leading to sub-optimal results. Using the concept that a signal is block sparse in a given basis —i.e., the non-zero elements occur in clusters of varying sizes — we present a novel and efficient algorithm for learning a sparse representation of natural images, called graph regularized block sparse dictionary (GRBSD) learning. We apply the proposed method towards two image restoration applications: 1) single-Image super-resolution, where we propose a local regression model that uses learned dictionaries from the GRBSD algorithm for super-resolving a low-resolution image without any external training images, and 2) image inpainting, where we use GRBSD algorithm to learn a multiscale dictionary to generate visually plausible pixels to fill missing regions in an image. Experimental results validate the performance of the GRBSD learning algorithm for single-image super-resolution and image inpainting applications. The second problem addressed in this work involves image enhancement for detection and segmentation of objects in images. We exploit the concept that even though data from various imaging modalities have high dimensionality, the data is sufficiently well described using low-dimensional geometrical structures. To facilitate the extraction of objects having such structure, we have developed general structure enhancement methods that can be used to detect and segment various curvilinear structures in images across different applications. We use the proposed method to detect and segment objects of different size and shape in three applications: 1) segmentation of lamina cribrosa microstructure in the eye from second-harmonic generation microscopy images, 2) detection and segmentation of primary cilia in confocal microscopy images, and 3) detection and segmentation of vehicles in wide-area aerial imagery. Quantitative and qualitative results show that the proposed methods provide improved detection and segmentation accuracy and computational efficiency compared to other recent algorithms.

Diminished reality, as a sub-topic of augmented reality where digital information is overlaid on an environment, is the perceived removal of an object from an environment. Previous approaches to diminished reality used digital replacement techniques, inpainting, and multi-view homographies. However, few used a virtual representation of the real environment, limiting their domains to planar environments. This thesis provides a framework to achieve real-time diminished reality on an augmented reality headset. Using state-of-the-art hardware, we combine a virtual representation of the real environment with inpainting to remove existing objects from complex environments. Our work is found to be competitive with previous results, with a similar qualitative outcome under the limitations of available technology. Additionally, by implementing new texturing algorithms, a more detailed representation of the real environment is achieved.

Background: Audio signal reconstruction using machine/deep learning algorithms has been explored much more in the recent years, and it has many applications in digital signal processing. There are many research works on audio reconstruction with linear interpolation, phase coding, tone insertion techniques combined with AI models. However, there is no research work on reconstructing audio signals with the fusion of Steganoflage (an adaptive approach to image steganography)  and AI models. Thus, in our thesis work, we focus on audio reconstruction combining Steganoflage and AI models. Objectives: This thesis aims to explore the possible enhancement of audio reconstruction using machine/deep learning models fusing Steganoflage technique. Furthermore, the suitable models implemented with the fusion of Steganoflage are analyzed and compared based on the performance metrics. Methods: We have conducted a systematic literature review followed by an experiment method to answer our research questions. The models implemented in the thesis are the results from a systematic literature review (SLR). In the experiments, we have fused the RF (Random Forest), SVR (Support Vector Regression), and LSTM (Long Short-Term Memory) models with Steganoflage for possible enhancement of reconstruction of lost audio signals. Then, the models were trained to estimate the possible approximate reconstructed signals. Finally, we observed the performance of the models and compared the reconstructed audio signals with the original signals (ground-truth) with four different performance metrics: Pearson linear correlation, PSNR, WPSNR, and SSIM. Results: The results from the SLR show that for machine learning models, RF and SVR models were mainly used for signals reconstructions and works well with time-series data. For deep learning models, recurrent neural network LSTM was the first choice as the survey of literature demonstrated that the model is suitable for time series forecasting. From the experiments, we found that the performance of LSTM model was better than RF and SVR models. Moreover, the reconstruction of audio signals from dropped short single region was better than that for multiple regions. Conclusions: We conclude that the Steganoflage, when fused with machine/deep learning models, enhances the lost audio signal reconstruction. Moreover, we also conclude that the LSTM model is more accurate than RF and SVR models in reconstructing the lost audio signals for a single drop region on both short and long gaps. However, we also observed that the audio reconstruction for multiple drops needs improvements considering long gaps. Furthermore, improvements can be made by exploring newer AI methods/optimization to enhance the reconstructed audio signals.

This diploma thesis discusses methods of dictionary learning to inpaint missing sections in the audio signal. There was theoretically analyzed and practically used algorithms K-SVD and INK-SVD for dictionary learning. These dictionaries have been applied to the reconstruction of audio signals using OMP (Orthogonal Matching Pursuit). Furthermore, there was proposed an algorithm for selecting the stationary segments and their subsequent use as training data for K-SVD and INK-SVD. In the practical part of thesis have been observed efficiency with training set selection from whole signal compared with algorithm for stationary segmentation used. The influence of mutual coherence on the quality of reconstruction with incoherent dictionary was also studied. With created scripts for multiple testing in Matlab, there was performed comparison of these methods on genre distinct songs.