Cette thèse présente des outils de vidéo inpainting permettant de reconstruire de manière efficace les zones perdues d'une séquence vidéo. Deux catégories d'approches sont particulièrement étudiées. Dans une première étape les approches basées sur l'exemple sont considérées. Différentes contributions ont été proposées. Une application des méthodes de neighbor embedding pour l'approximation des pixels perdus dans un exemple est d'abord considérée en utilisant deux méthodes de réduction de dimensionnalité: la factorisation de matrice non négative (FMN) et le locally linear embedding (LLE). La méthode d'inpainting proposée a été ensuite adaptée à l'application de dissimulation d'erreurs en utilisant une étape de pré-traitement d'estimation des vecteurs de mouvement perdus. Une approche multisolution a également été considérée pour réduire la complexité. Les évaluations expérimentales de cette approche démontrent son efficacité dans les applications de suppression d'objets et de dissimulation des erreurs. Une deuxième catégorie de méthodes de vidéo inpaintinting a été par la suite étudiée en utilisant une approche basée sur l'optimisation globale d'une fonction d'énergie exprimant la cohérence spatio-temporelle de la région reconstruite. Enfin, le problème d'inpainting des vidéos capturées par des caméras en mouvement a été étudié. L'alignement des images en utilisant une homographie par région montre de meilleure performances que les méthodes classiques d'alignement par optimisation d'une homography par pixel. / This thesis presents video inpainting tools to efficiently recover space-time holes in different kinds of video sequences. Two categories of video inpainting approaches are particularly studied. The first category concerns exemplar-based approach. Several contributions have been proposed for this approach. Neighbor embedding techniques have been proposed for patch sampling using two data dimensionality reductions methods: non-negative matrix factorization (NMF) and locally linear embedding (LLE). An analysis of similarity metrics for patches matching have then been proposed based on both subjective and objective tests. The proposed framework have been also adapted to the error concealment application by using a preprocessing step of motion estimation. A multiresolution approach has been considered to reduce the computational time of the method. The experimental evaluations demonstrate the effectiveness of the proposed video inpainting approach in both object removal and error concealment applications. The video inpainting problem has been also solved using a second approach based on the optimization of a well-defined cost function expressing the global consistency of the recovered regions. The camera moving videos has later been takled by using a region-based homography. The neighboring frames in the sequence are aligned based on segmented planar regions. This method has been shown to give better performance compared to classical optimization-based homography.

[pt] Nos últimos anos tem ocorrido um notável desenvolvimento de técnicas de Image Inpainting, entretanto transpor esse conhecimento para aplicações em vídeo tem se mostrado um desafio. Além dos desafios inerentes a tarefa de Video Inpainting (VI), utilizar essa técnica requer um trabalho prévio de anotação da área que será reconstruída. Se a aplicação do método for para remover um objeto ao longo de um vídeo, então a anotação prévia deve ser uma máscara da área deste objeto frame a frame. A tarefa de propagar a anotação de um objeto ao longo de um vídeo é conhecida como Video Object Segmentation (VOS) e já existem técnicas bem desenvolvidas para solucionar este problemas. Assim, a proposta desse trabalho é aplicar técnicas de VOS para gerar insumo para um algoritmo de VI. Neste trabalho iremos analisar o impacto de utilizar anotações preditas no resultado final de um modelo de VI. / [en] In recent years there has been a remarkable development of Image Inpainting techniques, but using this knowledge in video application is still a challenge. Besides the inherent challenges of the Video Inpainting (VI) task, applying this technique requires a previous job of labeling the area that should be reconstructed. If this method is used to remove an object from the video, then the annotation should be a mask of this object s area frame by frame. The task of propagating an object mask in a video is known as Video Object Segmentation (VOS) and there are already well developed techniques to solve this kind of task. Therefore, this work aims to apply VOS techniques to create the inputs for an VI algorithm. In this work we shall analyse the impact in the result of a VI algorithm when we use a predicted annotation as the input.

Digital inpainting is the technique of filling in the missing regions of an image or a video using information from surrounding area. This technique has found widespread use in applications such as restoration, error recovery, multimedia editing, and video privacy protection. This dissertation addresses three significant challenges associated with the existing and emerging inpainting algorithms and applications. The three key areas of impact are 1) Structure completion for image inpainting algorithms, 2) Fast and efficient object based video inpainting framework and 3) Perceptual evaluation of large area image inpainting algorithms. One of the main approach of existing image inpainting algorithms in completing the missing information is to follow a two stage process. A structure completion step, to complete the boundaries of regions in the hole area, followed by texture completion process using advanced texture synthesis methods. While the texture synthesis stage is important, it can be argued that structure completion aspect is a vital component in improving the perceptual image inpainting quality. To this end, we introduce a global structure completion algorithm for completion of missing boundaries using symmetry as the key feature. While existing methods for symmetry completion require a-priori information, our method takes a non-parametric approach by utilizing the invariant nature of curvature to complete missing boundaries. Turning our attention from image to video inpainting, we readily observe that existing video inpainting techniques have evolved as an extension of image inpainting techniques. As a result, they suffer from various shortcoming including, among others, inability to handle large missing spatio-temporal regions, significantly slow execution time making it impractical for interactive use and presence of temporal and spatial artifacts. To address these major challenges, we propose a fundamentally different method based on object based framework for improving the performance of video inpainting algorithms. We introduce a modular inpainting scheme in which we first segment the video into constituent objects by using acquired background models followed by inpainting of static background regions and dynamic foreground regions. For static background region inpainting, we use a simple background replacement and occasional image inpainting. To inpaint dynamic moving foreground regions, we introduce a novel sliding-window based dissimilarity measure in a dynamic programming framework. This technique can effectively inpaint large regions of occlusions, inpaint objects that are completely missing for several frames, change in size and pose and has minimal blurring and motion artifacts. Finally we direct our focus on experimental studies related to perceptual quality evaluation of large area image inpainting algorithms. The perceptual quality of large area inpainting technique is inherently a subjective process and yet no previous research has been carried out by taking the subjective nature of the Human Visual System (HVS). We perform subjective experiments using eye-tracking device involving 24 subjects to analyze the effect of inpainting on human gaze. We experimentally show that the presence of inpainting artifacts directly impacts the gaze of an unbiased observer and this in effect has a direct bearing on the subjective rating of the observer. Specifically, we show that the gaze energy in the hole regions of an inpainted image show marked deviations from normal behavior when the inpainting artifacts are readily apparent.