Shadows are ubiquitous in image and video, and their removal is of interest in both Computer Vision and Graphics. In this thesis, four methods for interactive shadow removal from single images are presented. Their improvements are made in user interaction, quality and robustness of shadow removal. We also show our state-of-the-art ground truth data set with variable scene categories for shadow removal and applications for shadow editing and its extension to video data processing.

Shadow removal is an important problem for both artists and algorithms. Previous methods handle some shadows well but, because they rely on the shadowed data, perform poorly in cases with severe degradation. Image-completion algorithms can completely replace severely degraded shadowed regions, and perform well with smaller-scale textures, but often fail to reproduce larger-scale macrostructure that may still be visible in the shadowed region. This paper provides a general framework that leverages degraded (e.g., shadowed) data to guide the image completion process by extending the objective function commonly used in current state-of-the-art image completion energy-minimization methods. This approach achieves realistic shadow removal even in cases of severe degradation and could be extended to other types of localized degradation.

Human activity recognition is one of the intensively studied areas in computer vision. Most existing works do not assume video resolution to be a problem due to general applications of interests. However, with continuous concerns about global security and emerging needs for intelligent video analysis tools, activity recognition from low-resolution and low-quality videos has become a crucial topic for further research. In this dissertation, We present a series of approaches which are developed specifically to address the related issues regarding low-level image preprocessing, single person activity recognition, and human-vehicle interaction reasoning from low-resolution surveillance videos. Human cast shadows are one of the major issues which adversely effect the performance of an activity recognition system. This is because human shadow direction varies depending on the time of the day and the date of the year. To better resolve this problem, we propose a shadow removal technique which effectively eliminates a human shadow cast from a light source of unknown direction. A multi-cue shadow descriptor is employed to characterize the distinctive properties of shadows. Our approach detects, segments, and then removes shadows. We propose two different methods to recognize single person actions and activities from low-resolution surveillance videos. The first approach adopts a joint feature histogram based representation, which is the concatenation of subspace projected gradient and optical flow features in time. However, in this problem, the use of low-resolution, coarse, pixel-level features alone limits the recognition accuracy. Therefore, in the second work, we contributed a novel mid-level descriptor, which converts an activity sequence into simultaneous temporal signals at body parts. With our representation, activities are recognized through both the local video content and the short-time spectral properties of body parts' movements. We draw the analogies between activity and speech recognition and show that our speech-like representation and recognition scheme improves recognition performance in several low-resolution datasets. To complete the research on this subject, we also tackle the challenging problem of recognizing human-vehicle interactions from low-resolution aerial videos. We present a temporal logic based approach which does not require training from event examples. At the low-level, we employ dynamic programming to perform fast model fitting between the tracked vehicle and the rendered 3-D vehicle models. At the semantic-level, given the localized event region of interest (ROI), we verify the time series of human-vehicle spatial relationships with the pre-specified event definitions in a piecewise fashion. Our framework can be generalized to recognize any type of human-vehicle interaction from aerial videos. / text

An improved object tracking algorithm based Kalman filtering is developed in this thesis. The algorithm uses a median filter and morphological operations during tracking. The problem created by object shadows is identified and the primary focus is to incorporate shadow detection and removal to improve tracking multiple objects in complex scenes. It is shown that the Kalman filter, without the improvements, fails to remove shadows that connect different objects. The application of the median filter helps the separation of different objects and thus enables the tracking of multiple objects individually. The performances of the Kalman filter and the improved tracking algorithm were tested on a highway video sequence of moving cars and it is shown that the proposed algorithm yields better performance in the presence of shadows.

Shadows in indoor scenarios are usually characterized with multiple light sources that produce complex shadow patterns of a single object. Without removing shadow, the foreground object tends to be erroneously segmented. The inconsistent hue and intensity of shadows make automatic removal a challenging task. In this thesis, a dynamic thresholding and transfer learning-based method for removing shadows is proposed. The method suppresses light shadows with a dynamically computed threshold and removes dark shadows using an online learning strategy that is built upon a base classifier trained with manually annotated examples and refined with the automatically identified examples in the new videos. Experimental results demonstrate that despite variation of lighting conditions in videos our proposed method is able to adapt to the videos and remove shadows effectively. The sensitivity of shadow detection changes slightly with different confidence levels used in example selection for classifier retraining and high confidence level usually yields better performance with less retraining iterations.

Submitted by PPG Engenharia El?trica (engenharia.pg.eletrica@pucrs.br) on 2018-06-08T13:54:41Z No. of bitstreams: 1 RICARDO_VINICIUS_DOTH_DIS.pdf: 9281309 bytes, checksum: d26fbf7274d4c8eb7158a2d987437b1b (MD5) / Approved for entry into archive by Sheila Dias (sheila.dias@pucrs.br) on 2018-06-18T12:49:23Z (GMT) No. of bitstreams: 1 RICARDO_VINICIUS_DOTH_DIS.pdf: 9281309 bytes, checksum: d26fbf7274d4c8eb7158a2d987437b1b (MD5) / Made available in DSpace on 2018-06-18T12:58:59Z (GMT). No. of bitstreams: 1 RICARDO_VINICIUS_DOTH_DIS.pdf: 9281309 bytes, checksum: d26fbf7274d4c8eb7158a2d987437b1b (MD5) Previous issue date: 2018-03-27 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior - CAPES / Wide Area Motion Imagery (WAMI) systems acquire large area aerial images in real time to provide accurate situational awareness information from a region (BLASCH et al., 2014). This system is applied for urban aerial monitoring. Unfavorable environmental conditions, such as shadow regions, are factors that increase system complexity by compromising the effectiveness of tracking algorithms and human visual interpretation (PORTER; FRASER; HUSH, 2010). Several techniques of shadow removal in aerial images have been developed, however due to the characteristics of the shadow and aerial image, a specific method to evaluate and compare the removal is unknown. The main objective of this study is to develop a method to evaluate shadow removal algorithms in aerial images acquired by the WAMI system. This work proposes a radiometric approach modifying the illumination in a controlled environment, simulating an aerial scene, acquiring images with and without the presence of shadows. The image with shadows is processed by the evaluated shadow removal algorithm, with the ideal output being the shadow free image. Shadow detection is evaluated using the confusion matrix concept. Shadow removal is evaluated using the structural similarity index (SSIM). As a result the reduced scale aerial scene model is presented to generate shadow and freeshadow images and the evaluation of 3 shadow removal methods using the data sets of images obtained from the scale model applying the methodology developed. / Sistemas WAMI (Wide Area Motion Imagery) adquirem imagens a?reas de grandes ?reas em tempo real para prover informa??es precisas de uma determinada regi?o (BLASCH et al., 2014). Este sistema ? aplicado para monitoramento a?reo urbano. Condi??es ambientais desfavor?veis, como ?reas sombreadas, s?o fatores que aumentam a complexidade do sistema comprometendo a efic?cia de algoritmos de rastreamento e a interpreta??o visual humana (PORTER; FRASER; HUSH, 2010). Diversas t?cnicas de remo??o de sombra em imagens a?reas foram desenvolvidas, no entanto devido ?s caracter?sticas da sombra e da imagem a?rea ? desconhecido um m?todo espec?fico para avaliar e comparar a remo??o de sombras em imagens a?reas. O objetivo principal deste estudo ? desenvolver um m?todo para avaliar algoritmos de remo??o de sombra em imagens a?reas adquiridas pelo sistema WAMI. Este trabalho prop?e uma abordagem radiom?trica modificando a ilumina??o em um ambiente controlado, simulando uma cena a?rea, adquirindo imagens com e sem sombras. A imagem com sombra ? processada pelo algoritmo de remo??o de sombra avaliado, sendo a imagem sem sombra o resultado ideal a ser alcan?ado. A detec??o de sombra ? avaliada utilizando o conceito de matriz de confus?o (error matrix). A remo??o de sombra ? avaliada utilizando o ?ndice de similaridade estrutural entre duas imagens (SSIM). Foram desenvolvidos o modelo de cena a?rea em escala reduzida para gerar imagens com e sem sombra e a avalia??o de 3 m?todos de remo??o de sombras utilizando os data sets de imagens obtidas do modelo em escala aplicando a metodologia descrita.

Submitted by PPG Engenharia El?trica (engenharia.pg.eletrica@pucrs.br) on 2017-11-08T12:48:45Z No. of bitstreams: 1 Guilherme_Silva_Dissertacao.pdf: 5806780 bytes, checksum: dd97b70975650b889aaa277b7e6f2b19 (MD5) / Approved for entry into archive by Caroline Xavier (caroline.xavier@pucrs.br) on 2017-11-17T12:39:26Z (GMT) No. of bitstreams: 1 Guilherme_Silva_Dissertacao.pdf: 5806780 bytes, checksum: dd97b70975650b889aaa277b7e6f2b19 (MD5) / Made available in DSpace on 2017-11-17T12:50:25Z (GMT). No. of bitstreams: 1 Guilherme_Silva_Dissertacao.pdf: 5806780 bytes, checksum: dd97b70975650b889aaa277b7e6f2b19 (MD5) Previous issue date: 2017-08-14 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior - CAPES

Ces dernières années, la biométrie a fait l’objet d’une grande attention en raison du besoin sans cesse croissant d’authentification d’identité, notamment pour sécuriser de plus en plus d’applications enlignes. Parmi divers traits biométriques, le visage offre des avantages compétitifs sur les autres, e.g., les empreintes digitales ou l’iris, car il est naturel, non-intrusif et facilement acceptable par les humains. Aujourd’hui, les techniques conventionnelles de reconnaissance faciale ont atteint une performance quasi-parfaite dans un environnement fortement contraint où la pose, l’éclairage, l’expression faciale et d’autres sources de variation sont sévèrement contrôlées. Cependant, ces approches sont souvent confinées aux domaines d’application limités parce que les environnements d’imagerie non-idéaux sont très fréquents dans les cas pratiques. Pour relever ces défis d’une manière adaptative, cette thèse porte sur le problème de reconnaissance faciale non contrôlée, dans lequel les images faciales présentent plus de variabilités sur les éclairages. Par ailleurs, une autre question essentielle vise à profiter des informations limitées de 3D pour collaborer avec les techniques basées sur 2D dans un système de reconnaissance faciale hétérogène. Pour traiter les diverses conditions d’éclairage, nous construisons explicitement un modèle de réflectance en caractérisant l’interaction entre la surface de la peau, les sources d’éclairage et le capteur de la caméra pour élaborer une explication de la couleur du visage. A partir de ce modèle basé sur la physique, une représentation robuste aux variations d’éclairage, à savoir Chromaticity Invariant Image (CII), est proposée pour la reconstruction des images faciales couleurs réalistes et sans ombre. De plus, ce processus de la suppression de l’ombre en niveaux de couleur peut être combiné avec les techniques existantes sur la normalisation d’éclairage en niveaux de gris pour améliorer davantage la performance de reconnaissance faciale. Les résultats expérimentaux sur les bases de données de test standard, CMU-PIE et FRGC Ver2.0, démontrent la capacité de généralisation et la robustesse de notre approche contre les variations d’éclairage. En outre, nous étudions l’usage efficace et créatif des données 3D pour la reconnaissance faciale hétérogène. Dans un tel scénario asymétrique, un enrôlement combiné est réalisé en 2D et 3D alors que les images de requête pour la reconnaissance sont toujours les images faciales en 2D. A cette fin, deux Réseaux de Neurones Convolutifs (Convolutional Neural Networks, CNN) sont construits. Le premier CNN est formé pour extraire les descripteurs discriminants d’images 2D/3D pour un appariement hétérogène. Le deuxième CNN combine une structure codeur-décodeur, à savoir U-Net, et Conditional Generative Adversarial Network (CGAN), pour reconstruire l’image faciale en profondeur à partir de son homologue dans l’espace 2D. Plus particulièrement, les images reconstruites en profondeur peuvent être également transmise au premier CNN pour la reconnaissance faciale en 3D, apportant un schéma de fusion qui est bénéfique pour la performance en reconnaissance. Notre approche a été évaluée sur la base de données 2D/3D de FRGC. Les expérimentations ont démontré que notre approche permet d’obtenir des résultats comparables à ceux de l’état de l’art et qu’une amélioration significative a pu être obtenue à l’aide du schéma de fusion. / In recent years, biometrics have received substantial attention due to the evergrowing need for automatic individual authentication. Among various physiological biometric traits, face offers unmatched advantages over the others, such as fingerprints and iris, because it is natural, non-intrusive and easily understandable by humans. Nowadays conventional face recognition techniques have attained quasi-perfect performance in a highly constrained environment wherein poses, illuminations, expressions and other sources of variations are strictly controlled. However these approaches are always confined to restricted application fields because non-ideal imaging environments are frequently encountered in practical cases. To adaptively address these challenges, this dissertation focuses on this unconstrained face recognition problem, where face images exhibit more variability in illumination. Moreover, another major question is how to leverage limited 3D shape information to jointly work with 2D based techniques in a heterogeneous face recognition system. To deal with the problem of varying illuminations, we explicitly build the underlying reflectance model which characterizes interactions between skin surface, lighting source and camera sensor, and elaborate the formation of face color. With this physics-based image formation model involved, an illumination-robust representation, namely Chromaticity Invariant Image (CII), is proposed which can subsequently help reconstruct shadow-free and photo-realistic color face images. Due to the fact that this shadow removal process is achieved in color space, this approach could thus be combined with existing gray-scale level lighting normalization techniques to further improve face recognition performance. The experimental results on two benchmark databases, CMU-PIE and FRGC Ver2.0, demonstrate the generalization ability and robustness of our approach to lighting variations. We further explore the effective and creative use of 3D data in heterogeneous face recognition. In such a scenario, 3D face is merely available in the gallery set and not in the probe set, which one would encounter in real-world applications. Two Convolutional Neural Networks (CNN) are constructed for this purpose. The first CNN is trained to extract discriminative features of 2D/3D face images for direct heterogeneous comparison, while the second CNN combines an encoder-decoder structure, namely U-Net, and Conditional Generative Adversarial Network (CGAN) to reconstruct depth face image from its counterpart in 2D. Specifically, the recovered depth face images can be fed to the first CNN as well for 3D face recognition, leading to a fusion scheme which achieves gains in recognition performance. We have evaluated our approach extensively on the challenging FRGC 2D/3D benchmark database. The proposed method compares favorably to the state-of-the-art and show significant improvement with the fusion scheme.