This project studies the removal of non-anatomical objects from medical images. During tumor ablation procedures, the ablation probes appear in the image, hindering the performance of segmentation, registration, and dose computation algorithms. These algorithms can also be affected by artifacts and noise generated by body implants. Image inpainting methods allow the completion of the missing or distorted regions, generating realistic structures coherent with the rest of the image. During the last decade, the study of image inpainting methods has accelerated due to advances in deep learning and the increase in the consumption of multimedia content. Models applying generative adversarial networks have excelled at the task of image synthesis. However, there has not been much study done on medical image inpainting. In this project, a new inpainting method is proposed for recovering missing information from medical images. This method consists of a two-stage model, where a coarse network is followed by a refinement network, both of which are U-Nets. The refinement network is trained together with a discriminator, providing adversarial learning. The model is trained on a dataset of CT images of the liver and, in order the mimic the areas where information is missing, regular and irregular shaped masks are applied. The trained models are compared both quantitatively and qualitatively. Due to the lack of standards and accurate metrics in image inpainting tasks, results cannot be easily compared to current approaches. However, qualitative analysis of the inpainted images shows promising results. In addition, this project identifies the Frechet Inception Distance as a more valid metric than older metrics commonly used for evaluation of image inpainting models. In conclusion, this project provides an inpainting model for medical images, which could be used during tumor ablation procedures and for noise and artifact elimination. Future research could include implementing a 3D model to provide more coherent results for inpainting patients - a stack of images - instead of single images. / I detta projekt undersöks metoder för avlägsnande av icke-anatomiska föremål från medicinska bilder. Bilder tagna under ablationsbehandling av tumörer innehåller själva ablationsnålen, denna kan hindra segmenterings-, registrerings-och dosberäknings-algoritmer för att uppnå önska resultat. Dessa algoritmer kan också påverkas av artefakter och brus som genereras av olika metallimplantat. Bildifyllningsmetoder gör det möjligt att ersätta regioner som saknar eller innehåller inkorrekt bilddata, med realistiska strukturer som är sammanhängande med resten av bilden. Under det senaste decenniet har intresset för metoder för bildifyllning accelererat på grund av framsteg inom djupinlärning och ökad konsumtion av multimediainnehåll. Modeller som använder generative adversarial networks har utmärkt sig i bildsynteseringsuppgifter. Det har dock inte gjorts så många studier gällande bildifyllning av medicinska bilder. I detta projekt föreslås en ny bildifyllningsmetod för att återställa regioner med inkorrekt information i medicinska bilder. Denna metod består av ett tvåstegsnätverk, där ett första nätverk följs av ett förfiningsnätverk, båda av typen U-net. Förfiningsnätverk tränas tillsammans med ett diskriminatornätverk. Modellen tränas på ett dataset av CT-bilder av levern. För att efterlikna de områden där information saknas, applicerades masker av olika former. De färdigtränade modellerna jämfördes både kvantitativt och kvalitativt. På grund av bristen på standarder och noggranna mätvärden för bildifyllningsmetoder, kan resultaten inte enkelt jämföras med existerande metoder. Men kvalitativ analys av de målade bilderna visar ganska lovande resultat. Modellen presterar som bäst i områden inte innehåller komplexa strukturer. Sammanfattningsvis har en fungerande bildifyllningsmetod för medicinska bilder skapats och som kan användas vid tumörablation och för eliminering av bildartefakter. Framtida forskning kan inkludera implementering av en 3D-modell för att ge mer sammanhängande resultat.

The proliferation of multimedia data, including digital images and videos, has led to an increase in their misuse, such as the unauthorized sharing of sensitive content, the spread of fake news, and the dissemination of misleading propaganda. To address these issues, the research field of multimedia forensics has developed tools to distinguish genuine multimedia from fakes and identify the sources of those who share sensitive content. However, the accuracy and reliability of multimedia forensics tools are threatened by recent technological advancements in new multimedia processing software and camera devices. For example, source attribution involves attributing an image or video to a specific camera device, which is crucial for addressing privacy violations, cases of revenge porn, and instances of child pornography. These tools exploit forensic traces unique to each camera’s manufacturing process, such as Photo Response Non-Uniformity (PRNU). Nevertheless, image and video processing transformations can disrupt the consistency of PRNU, necessitating the development of new methods for its recovery. Conversely, to distinguish genuine multimedia from fakes, AI-based image and video forgery localization methods have also emerged. However, they constantly face challenges from new, more sophisticated AI-forgery techniques and are hindered by factors like AI-aided post-processing and, in the case of videos, lower resolutions, and stronger compression. This doctoral study investigates the relevance of exploiting temporal information during the parameters estimation used to reverse complex spatial transformations for source attribution, and video forgery localization in low-resolution H.264 post-processed inpainted videos. Two novel methods will be presented that model the set of parameters involved in reversing in-camera and out-camera complex spatial transformations applied to images and videos as time series, improving source attribution accuracy and computational efficiency. Regarding video inpainting localization, a novel dataset of videos inpainted and post-processed with Temporal Consistency Networks will be introduced, and we will present our solution to improve video inpainting localization by taking into account spatial and temporal inconsistencies at dense optical flow level. The research presented in this dissertation has resulted in several publications that contribute to the field of multimedia forensics, addressing challenges related to source attribution and video forgery localization.

Dans ce manuscrit, nous avons étudié l’extension de l’analyse multi-résolution aux images couleurs par des transformées sur graphe. Dans ce cadre, nous avons déployé trois stratégies d’analyse différentes. En premier lieu, nous avons défini une transformée basée sur l’utilisation d’un graphe perceptuel dans l’analyse à travers la transformé en ondelettes spectrale sur graphe. L’application en débruitage d’image met en évidence l’utilisation du SVH dans l’analyse des images couleurs. La deuxième stratégie consiste à proposer une nouvelle méthode d’inpainting pour des images couleurs. Pour cela, nous avons proposé un schéma de régularisation à travers les coefficients d’ondelettes de la TOSG, l’estimation de la structure manquante se fait par la construction d’un graphe des patchs couleurs à partir des moyenne non locales. Les résultats obtenus sont très encourageants et mettent en évidence l’importance de la prise en compte du SVH. Dans la troisième stratégie, nous proposons une nouvelleapproche de décomposition d’un signal défini sur un graphe complet. Cette méthode est basée sur l’utilisation des propriétés de la matrice laplacienne associée au graphe complet. Dans le contexte des images couleurs, la prise en compte de la dimension couleur est indispensable pour pouvoir identifier les singularités liées à l’image. Cette dernière offre de nouvelles perspectives pour une étude approfondie de son comportement. / In our work, we studied the extension of the multi-resolution analysis for color images by using transforms on graphs. In this context, we deployed three different strategies of analysis. Our first approach consists of computing the graph of an image using the psychovisual information and analyzing it by using the spectral graph wavelet transform. We thus have defined a wavelet transform based on a graph with perceptual information by using the CIELab color distance. Results in image restoration highlight the interest of the appropriate use of color information. In the second strategy, we propose a novel recovery algorithm for image inpainting represented in the graph domain. Motivated by the efficiency of the wavelet regularization schemes and the success of the nonlocal means methods we construct an algorithm based on the recovery of information in the graph wavelet domain. At each step the damaged structure are estimated by computing the non local graph then we apply the graph wavelet regularization model using the SGWT coefficient. The results are very encouraging and highlight the use of the perceptual informations. In the last strategy, we propose a new approach of decomposition for signals defined on a complete graphs. This method is based on the exploitation of of the laplacian matrix proprieties of the complete graph. In the context of image processing, the use of the color distance is essential to identify the specificities of the color image. This approach opens new perspectives for an in-depth study of its behavior.

O problema de reconstrucão digital de imagens (Image Inpainting) possui diversas abordagens para sua resolução. Uma possibilidade consiste na sua modelagem como um problema de otimizacão contínua (lasso). Na presente dissertacão aplica-se o método do gradiente espectral projetado a esse problema. Desenvolve-se inteiramente a modelagem do problema assim como a implementacão computacional do método de otimização que o resolve. Resultados computacionais demonstram a qualidade do método para um conjunto de imagens digitais / The image inpainting problem has several resolution approaches. One possibility consists in its modeling as a continuous optimization problem. In the present dissertation we apply the spectral projected gradient method to this problem. We develop the whole modeling of the problem as well as the computational implementation of the optimization method to solve it. Computational results show the quality of the method for a set of digital images

Neste trabalho são apresentadas, complementadas e melhoradas duas técnicas de restauração de imagens: uma abordando o problema de retoque digital/remoção de objetos enquanto a segunda é direcionada ao problema deneliminação de ruído. Em ambas as técnicas, a ideia é trabalhar com imagens contendo texturas e outras características de interesse para um observador humano como a preservação de padrões, bordas, estruturas e regiões de natureza oscilatória. A técnica descrita sobre retoque digital de imagens combina difusão anisotrópica, síntese de texturas, busca dinâmica e um novo termo empregado no mecanismo de atribuição da ordem de prioridade durante o processo de reconstrução. Assim, dada uma imagem com regiões a serem recompostas, uma técnica de difusão anisotrópica é aplicada à imagem afim de se obter um mapa de saliência contendo bordas, estruturas e demais informações de baixa frequência da imagem. Na sequência, um mecanismo de prioridade baseado em um novo termo de confiabilidade regularizado é calculado a partir da combinação do mapa anteriormente gerado com a equação do transporte. Tal mecanismo é utilizado para determinar a ordem de preenchimento das partes faltantes da imagem. Para essa tarefa, a abordagem apresentada utiliza uma nova medida de similaridade entre blocos de pixels(amostrados dinamicamente para acelerar o processo), afim de encontrar os melhores candidatos a serem alocados nas regiões danificadas. A técnica destinada à remoção de ruídos alia a teoria da difusão anisotrópica, técnicas de análise harmônica e modelos numéricos de discretização de EDPs não-lineares em uma equação diferencial parcial regularizada, a qual atua de forma incisiva em regiões mais homogêneas da imagem e de forma mais suave em regiões caracterizadas como textura e bordas, preservando, assim, essas regiões. Além da natureza anisotrópica, a EDP procura recompor partes texturizadas perdidas no processo de eliminação de ruído através da aplicação de técnicas robustas de análise harmônica. Uma validação teórica e experimental para esta EDP e um estudo do ajuste paramétrico do método de eliminação de ruído baseado nesta EDP foram realizados neste trabalho. A eficiência e a performance das técnicas propostas são atestadas por meio das análises experimentais quantitativas e qualitativas com outras abordagens clássicas da literatura. / In this work two techniques of image restoration are presented, complemented and improved: one approaching the problem of image inpainting/object removal problem while the second one dealing with the image denoising problem. In both cases, the core idea is to process images containing textures and other features perceptible to a human observer such as patterns, contours, structures and oscillatory information. The image inpainting technique combines anisotropic diffusion, texture synthesis, dynamic search and a mechanism to set the order of priority during the image completion process. More precisely, given an image and target region to be inpainted, an anisotropic diffusion technique is applied in order to generate a saliency map containing edges, structures and other low frequency parts of the image. Next, apriority mechanism based on a new biased confidence term is computed from the map previously generated with the transport equation to define the level of priority of the pixels during the filling procedure. To accomplish this task, the presented approach employs a novel measure of similarity wich measures the distance between blocks of pixels (sampled dynamically to speed up the process) in order to find the best candidates to be allocated in the damaged regions. The technique devoted to denoising an image combines the theory of anisotropic diffusion, harmonic analysis techniques and numerical models into a regularized partial differential equation, which diffuses the pixels more incisively on homogeneous regions of the image while still seeking to attenuate regions formed by textures and patterns, thus preserving those information. Moreover, the proposed PDE aims at recovering texturized regions which have been degraded during the denoising process by employing harmonic analysis tools. A theoretical and experimental validation for this EDP and a study of the parametric adjustment of the image denoising method based on this EDP were performed in this work. The effectivenss and performance of the proposed approaches are attested through a comprehensive set of comparisons against other representative techniques in the literature.

The subject of this thesis is a class of fast inpainting methods (image or video) based on the idea of filling the inpainting domain in successive shells from its boundary inwards. Image pixels (or video voxels) are filled by assigning them a color equal to a weighted average of either their already filled neighbors (the ``direct'' form of the method) or those neighbors plus additional neighbors within the current shell (the ``semi-implicit'' form). In the direct form, pixels (voxels) in the current shell may be filled independently, but in the semi-implicit form they are filled simultaneously by solving a linear system. We focus in this thesis mainly on the image inpainting case, where the literature contains several methods corresponding to the {\em direct} form of the method - the semi-implicit form is introduced for the first time here. These methods effectively differ only in the order in which pixels (voxels) are filled, the weights used for averaging, and the neighborhood that is averaged over. All of them are very fast, but at the same time all of them leave undesirable artifacts such as ``kinking'' (bending) or blurring of extrapolated isophotes. This thesis has two main goals. First, we introduce new algorithms within this class, which are aimed at reducing or eliminating these artifacts, and also target a specific application - the 3D conversion of images and film. The first part of this thesis will be concerned with introducing 3D conversion as well as Guidefill, a method in the above class adapted to the inpainting problems arising in 3D conversion. However, the second and more significant goal of this thesis is to study these algorithms as a class. In particular, we develop a mathematical theory aimed at understanding the origins of artifacts mentioned. Through this, we seek is to understand which artifacts can be eliminated (and how), and which artifacts are inevitable (and why). Most of the thesis is occupied with this second goal. Our theory is based on two separate limits - the first is a {\em continuum} limit, in which the pixel width →0, and in which the algorithm converges to a partial differential equation. The second is an asymptotic limit in which h is very small but non-zero. This latter limit, which is based on a connection to random walks, relates the inpainted solution to a type of discrete convolution. The former is useful for studying kinking artifacts, while the latter is useful for studying blur. Although all the theoretical work has been done in the context of image inpainting, experimental evidence is presented suggesting a simple generalization to video. Finally, in the last part of the thesis we explore shell-based video inpainting. In particular, we introduce spacetime transport, which is a natural generalization of the ideas of Guidefill and its predecessor, coherence transport, to three dimensions (two spatial dimensions plus one time dimension). Spacetime transport is shown to have much in common with shell-based image inpainting methods. In particular, kinking and blur artifacts persist, and the former of these may be alleviated in exactly the same way as in two dimensions. At the same time, spacetime transport is shown to be related to optical flow based video inpainting. In particular, a connection is derived between spacetime transport and a generalized Lucas-Kanade optical flow that does not distinguish between time and space.

The generation of urban digital elevation models from satellite images using stereo reconstruction techniques poses several challenges due to its precision requirements. In this thesis we study three problems related to the reconstruction of urban models using stereo images in a low baseline disposition. They were motivated by the MISS project, launched by the CNES (Centre National d'Etudes Spatiales), in order to develop a low baseline acquisition model. The first problem is the restoration of irregularly sampled images and image fusion using a band limited interpolation model. A novel restoration algorithm is proposed, which incorporates the image formation model as a set of local constraints, and uses of a family of regularizers that allow to control the spectral behavior of the solution. Secondly, the problem of interpolating sparsely sampled images is addressed using a self-similarity prior. The related problem of image inpainting is also considered, and a novel framework for exemplar-based image inpainting is proposed. This framework is then extended to consider the interpolation of sparsely sampled images. The third problem is the regularization and interpolation of digital elevation models imposing geometric restrictions. The geometric restrictions come from a reference image. For this problem three different regularization models are studied: an anisotropic minimal surface regularizer, the anisotropic total variation and a new piecewise affine interpolation algorithm. / La generación de modelos urbanos de elevación a partir de imágenes de satélite mediante técnicas de reconstrucción estereoscópica presenta varios retos debido a sus requisitos de precisión. En esta tesis se estudian tres problemas vinculados a la generación de estos modelos partiendo de pares estereoscópicos adquiridos por satélites en una configuración con baseline pequeño. Estos problemas fueron motivados por el proyecto MISS, lanzado por el CNES (Centre National d'Etudes Spatiales) con el objetivo de desarrollar las técnicas de reconstrucción para imágenes adquiridas con baseline pequeños. El primer problema es la restauración de imágenes muestreadas irregularmente y la fusión de imágenes usando un modelo de interpolación de banda limitada. Se propone un nuevo método de restauración, el cual usa una familia de regularizadores que permite controlar el decaimiento espectral de la solución e incorpora el modelo de formación de imagen como un conjunto de restricciones locales. El segundo problema es la interpolación de imágenes muestreadas en forma dispersa usando un prior de auto similitud, se considera también el problema relacionado de inpainting de imágenes. Se propone un nuevo framework para inpainting basado en ejemplares, el cual luego es extendido a la interpolación de imágenes muestreadas en forma dispersa. El tercer problema es la regularización e interpolación de modelos digitales de elevación imponiendo restricciones geométricas las cuales se extraen de una imagen de referencia. Para este problema se estudian tres modelos de regularización: un regularizador anisótropo de superficie mínima, la variación total anisótropa y un nuevo algoritmo de interpolación afín a trozos.

Orientador: Maurílio Boaventura / Banca: Adilson Gonzaga / Banca: Eliana Xavier Linhares de Andrade / Resumo: Neste trabalho propõe-se um filtro mediano seletivo e um filtro híbrido para eliminação de ruído impulsivo em imagens digitais monocromáticas. O primeiro é baseado em uma modificação do filtro mediano por meio de um detector de ruído impulsivo. O segundo é obtido combinando-se o filtro mediano seletivo com um modelo de retoque digital. A remoção de ruído impulsivo em uma imagem colorida é realizada por meio da extensão dos filtros propostos para cada canal de cor da imagem. Os experimentos realizados indicam que os métodos propostos são eficazes na restauração de imagens com grandes densidades de ruído. / Abstract: In this paper a selective median filter and a hybrid filter for removing impulsive noise in digital grayscale images are proposed. The first is a median filter modification based on impulsive noise detector. The second is obtained by combining the selective median filter with a digital inpainting model. The noise removal in color image is obtained by extending the proposed filters for each color channel of the image. The experiments indicated that the proposed methods are powerful in restoring images with high densities noise. / Mestre

O problema de reconstrucão digital de imagens (Image Inpainting) possui diversas abordagens para sua resolução. Uma possibilidade consiste na sua modelagem como um problema de otimizacão contínua (lasso). Na presente dissertacão aplica-se o método do gradiente espectral projetado a esse problema. Desenvolve-se inteiramente a modelagem do problema assim como a implementacão computacional do método de otimização que o resolve. Resultados computacionais demonstram a qualidade do método para um conjunto de imagens digitais / The image inpainting problem has several resolution approaches. One possibility consists in its modeling as a continuous optimization problem. In the present dissertation we apply the spectral projected gradient method to this problem. We develop the whole modeling of the problem as well as the computational implementation of the optimization method to solve it. Computational results show the quality of the method for a set of digital images

Image completion is a process of removing an area from a photograph and replacing it with suitable data. Earlier methods either search for this relevant data within the image itself, or extends the search to some form of additional data, usually some form of database. Methods that search for suitable data within the image itself has problems when no suitable data can be found in the image. Methods that extend their search has in earlier work either used some form of database with labeled images or a massive database with photos from the Internet. For the labels in a database to be useful they typically needs to be entered manually, which is a very time consuming process. Methods that uses databases with millions of images from the Internet has issues with copyrighted images, storage of the photographs and computation time. This work shows that a small database of the user’s own private, or professional, photos can be used to improve the quality of image completions. A photographer today typically take many similar photographs on similar scenes during a photo session. Therefore a smaller number of images are needed to find images that are visually and structurally similar, than when random images downloaded from the internet are used. Thus, this approach gains most of the advantages of using additional data for the image completions, while at the same time minimizing the disadvantages. It gains a better ability to find suitable data without having to process millions of irrelevant photos.