Esta dissertação apresenta uma solução prática para o problema de reiluminação de imagens para objetos com geometria arbitrária. Nossa técnica baseia-se no que chamamos de shading proxies (versões deformadas de modelos 3D que aproximam o objeto a ser reiluminado) para guiar o processo de reiluminação. Nosso método é flexível e robusto, podendo reiluminar fotografias, pinturas, e desenhos de diferentes objetos de maneira eficaz. Além de reiluminação, nossa técnica pode ser usada para estimar mapas de normais e profundidade, bem como realizar decomposição intrínsica de imagens, e transferir iluminação para desenhos delineados. Uma avaliação preliminar mostra que nossa técnica produz resultados convincentes, e usuários novatos podem reiluminar imagens facilmente em poucos minutos. / We present a practical solution to the problem of single-image relighting of objects with arbitrary shapes. It is based on a shading-ratio image obtained from the original and target lighting applied to shading proxies (warped versions of 3-D models that approximate the objects to be relit). Our approach is flexible and robust, being applicable to objects with non-uniform albedos. We demonstrate its effectiveness by relighting a large number of photographs, paintings, and drawings containing a variety of objects of different materials. In addition to relighting, our technique can estimate smooth normal and depth maps from pictures, as well as perform intrinsic-image decomposition. Preliminary evaluation has shown that our technique produces convincing results, and novice users can relight images in just a couple of minutes.

Esta dissertação apresenta uma solução prática para o problema de reiluminação de imagens para objetos com geometria arbitrária. Nossa técnica baseia-se no que chamamos de shading proxies (versões deformadas de modelos 3D que aproximam o objeto a ser reiluminado) para guiar o processo de reiluminação. Nosso método é flexível e robusto, podendo reiluminar fotografias, pinturas, e desenhos de diferentes objetos de maneira eficaz. Além de reiluminação, nossa técnica pode ser usada para estimar mapas de normais e profundidade, bem como realizar decomposição intrínsica de imagens, e transferir iluminação para desenhos delineados. Uma avaliação preliminar mostra que nossa técnica produz resultados convincentes, e usuários novatos podem reiluminar imagens facilmente em poucos minutos. / We present a practical solution to the problem of single-image relighting of objects with arbitrary shapes. It is based on a shading-ratio image obtained from the original and target lighting applied to shading proxies (warped versions of 3-D models that approximate the objects to be relit). Our approach is flexible and robust, being applicable to objects with non-uniform albedos. We demonstrate its effectiveness by relighting a large number of photographs, paintings, and drawings containing a variety of objects of different materials. In addition to relighting, our technique can estimate smooth normal and depth maps from pictures, as well as perform intrinsic-image decomposition. Preliminary evaluation has shown that our technique produces convincing results, and novice users can relight images in just a couple of minutes.

Esta dissertação apresenta uma solução prática para o problema de reiluminação de imagens para objetos com geometria arbitrária. Nossa técnica baseia-se no que chamamos de shading proxies (versões deformadas de modelos 3D que aproximam o objeto a ser reiluminado) para guiar o processo de reiluminação. Nosso método é flexível e robusto, podendo reiluminar fotografias, pinturas, e desenhos de diferentes objetos de maneira eficaz. Além de reiluminação, nossa técnica pode ser usada para estimar mapas de normais e profundidade, bem como realizar decomposição intrínsica de imagens, e transferir iluminação para desenhos delineados. Uma avaliação preliminar mostra que nossa técnica produz resultados convincentes, e usuários novatos podem reiluminar imagens facilmente em poucos minutos. / We present a practical solution to the problem of single-image relighting of objects with arbitrary shapes. It is based on a shading-ratio image obtained from the original and target lighting applied to shading proxies (warped versions of 3-D models that approximate the objects to be relit). Our approach is flexible and robust, being applicable to objects with non-uniform albedos. We demonstrate its effectiveness by relighting a large number of photographs, paintings, and drawings containing a variety of objects of different materials. In addition to relighting, our technique can estimate smooth normal and depth maps from pictures, as well as perform intrinsic-image decomposition. Preliminary evaluation has shown that our technique produces convincing results, and novice users can relight images in just a couple of minutes.

Previsualization is a class of techniques for creating approximate previews of a movie sequence in order to visualize a scene prior to shooting it on the set. Often these techniques are used to convey the artistic direction of the story in terms of cinematic elements, such as camera movement, angle, lighting, dialogue, and character motion. Essentially, a movie director uses previsualization (previs) to convey movie visuals as he sees them in his "minds-eye". Traditional methods for previs include hand-drawn sketches, Storyboards, scaled models, and photographs, which are created by artists to convey how a scene or character might look or move. A recent trend has been to use 3D graphics applications such as video game engines to perform previs, which is called 3D previs. This type of previs is generally used prior to shooting a scene in order to choreograph camera or character movements. To visualize a scene while being recorded on-set, directors and cinematographers use a technique called On-set previs, which provides a real-time view with little to no processing. Other types of previs, such as Technical previs, emphasize accurately capturing scene properties but lack any interactive manipulation and are usually employed by visual effects crews and not for cinematographers or directors. This dissertation's focus is on creating a new method for interactive visualization that will automatically capture the on-set lighting and provide interactive manipulation of cinematic elements to facilitate the movie maker's artistic expression, validate cinematic choices, and provide guidance to production crews. Our method will overcome the drawbacks of the all previous previs methods by combining photorealistic rendering with accurately captured scene details, which is interactively displayed on a mobile capture and rendering platform. This dissertation describes a new hardware and software previs framework that enables interactive visualization of on-set post-production elements. A three-tiered framework, which is the main contribution of this dissertation is; 1) a novel programmable camera architecture that provides programmability to low-level features and a visual programming interface, 2) new algorithms that analyzes and decomposes the scene photometrically, and 3) a previs interface that leverages the previous to perform interactive rendering and manipulation of the photometric and computer generated elements. For this dissertation we implemented a programmable camera with a novel visual programming interface. We developed the photometric theory and implementation of our novel relighting technique called Symmetric lighting, which can be used to relight a scene with multiple illuminants with respect to color, intensity and location on our programmable camera. We analyzed the performance of Symmetric lighting on synthetic and real scenes to evaluate the benefits and limitations with respect to the reflectance composition of the scene and the number and color of lights within the scene. We found that, since our method is based on a Lambertian reflectance assumption, our method works well under this assumption but that scenes with high amounts of specular reflections can have higher errors in terms of relighting accuracy and additional steps are required to mitigate this limitation. Also, scenes which contain lights whose colors are a too similar can lead to degenerate cases in terms of relighting. Despite these limitations, an important contribution of our work is that Symmetric lighting can also be leveraged as a solution for performing multi-illuminant white balancing and light color estimation within a scene with multiple illuminants without limits on the color range or number of lights. We compared our method to other white balance methods and show that our method is superior when at least one of the light colors is known a priori.

We enhance photographs shot in dark environments by combining a picture taken with the available light and one taken with the flash. We preserve the ambiance of the original lighting and insert the sharpness from the flash image. We use the bilateral filter to decompose the images into detail and large scale. We reconstruct the image using the large scale of the available lighting and the detail of the flash. We detect and correct flash shadows. This combines the advantages of available illumination and flash photography. / Singapore-MIT Alliance (SMA)

Nous introduisons un algorithme de décomposition intrinsèque multi-vue qui permet de ré-éclairer une scène extérieure en utilisant quelques images en entrée. Plusieurs applications comme l’architecture, jeux et films exigent de manipuler un modèle 3D d’une scène. Cependant, la modification de telles scènes est limitée par les conditions d’éclairage de capture. Notre méthode estime les images intrinsèques prises dans des conditions d’éclairage identiques avec des ombres. Nous utilisons conjointement une reconstruction 3D automatique et la direction du soleil pour obtenir la décomposition de chaque image en calques de réflectance et d’éclairage malgré l’inexactitude des données du modèle 3D. Notre approche est basée sur deux idées principales. Tout d’abord, nous raffinons l’estimation des paramètres de notre modèle de formation d’image en combinant la simulation d’éclairage 3D avec des méthodes d’optimisation basée image. Deuxièmement, nous utilisons ce modèle pour exprimer la réflectance en fonction de valeur de visibilité discrète pour l’ombre et la lumière, ce qui nous permet d’introduire un classificateur d’ombre robuste pour des paires de points dans une scène. Nos calques intrinsèques sont de qualité suffisante pour manipuler les images d’entrée. Nous déplaçons les ombres portées en créant une géométrie qui préserve les silhouettes d’ombre. Notre méthode est compatible avec les approches de rendu basé image et réduit les coûts de création de contenu 3D. Enfin, nous présentons une étude sur les limites du modèle de réflectance diffus et la difficulté d’appliquer les approches existantes dans le cadre de reconstruction 3D multi vue où les données sont imprécises. / We present a multi-view intrinsic decomposition algorithm that allows relighting of an outdoor scene using just a few photographs as input. Several applications such as architecture, games and movies require a 3D model of a scene. However, editing such scenes is limited by the lighting condition at the time of capture. Our method computes intrinsic images photos taken under the same lighting condition with existing cast shadows by the sun. We use an automatic 3D reconstruction from these photos and the sun direction as input and decompose each image into reflectance and shading layers, despite the inaccuracies and missing data of the 3D model. Our approach is based on two key ideas. First, we progressively improve the accuracy of the parameters of our image formation model by performing iterative estimation and combining 3D lighting simulation with 2D image optimization methods. Second we use the image formation model to express reflectance as a function of discrete visibility values for shadow and light, which allows us to introduce a robust shadow classifier for pairs of points in a scene. Our multi-view intrinsic decomposition is of sufficient quality for relighting of the input images. We create shadow-caster geometry which preserves shadow silhouettes and using the intrinsic layers, we can perform multi-view relighting with moving cast shadows. Our method allows image-based rendering with changing illumination conditions and reduces the cost of creating 3D content for applications. Finally, we present an initial study on the limitation of diffuse reflectance models for these computations.

Creation of visually-pleasing images has always been one of the main goals of computer graphics. Two important components are necessary to achieve this goal --- artists who design visual aspects of an image (such as materials or lighting) and sophisticated algorithms that render the image. Traditionally, rendering has been of greater interest to researchers, while the design part has always been deemed as secondary. This has led to many inefficiencies, as artists, in order to create a stunning image, are often forced to resort to the traditional, creativity-baring, pipelines consisting of repeated rendering and parameter tweaking. Our work shifts the attention away from the rendering problem and focuses on the design. We propose to combine non-physical editing with real-time feedback and provide artists with efficient ways of designing complex visual aspects such as global illumination or all-frequency shadows. We conform to existing pipelines by inserting our editing components into existing stages, hereby making editing of visual aspects an inherent part of the design process. Many of the examples showed in this work have been, until now, extremely hard to achieve. The non-physical aspect of our work enables artists to express themselves in more creative ways, not limited by the physical parameters of current renderers. Real-time feedback allows artists to immediately see the effects of applied modifications and compatibility with existing workflows enables easy integration of our algorithms into production pipelines.

This thesis introduces methods for adding user-guided depth-aware effects to images captured with a consumer-grade stereo camera with minimal user interaction. In particular, we present methods for highlighted depth-of-field, haze, depth-of-field, and image relighting. Unlike many prior methods for adding such effects, we do not assume prior scene models or require extensive user guidance to create such models, nor do we assume multiple input images. We also do not require specialized camera rigs or other equipment such as light-field camera arrays, active lighting, etc. Instead, we use only an easily portable and affordable consumer-grade stereo camera. The depth is calculated from a stereo image pair using an extended version of PatchMatch Stereo designed to compute not only image disparities but also normals for visible surfaces. We also introduce a pipeline for rendering multiple effects in the order they would occur physically. Each can be added, removed, or adjusted in the pipeline without having to reapply subsequent effects. Individually or in combination, these effects can be used to enhance the sense of depth or structure in images and provide increased artistic control. Our interface also allows editing the stereo pair together in a fashion that preserves stereo consistency, or the effects can be applied to a single image only, thus leveraging the advantages of stereo acquisition even to produce a single photograph.

L'objectif de cette thèse est de tirer parti d'une illumination totalement ou partiellement contrôlée pour enrichir l'acquisition vidéo de contenus tel que la reconstruction de la forme et de l'apparence. Aujourd'hui de nombreux travaux ont tenté d'atteindre cet objectif. Certains utilisent une illumination contrôlée et séquentielle pour obtenir des reconstructions de haute qualité de la forme et de la réflectance. En revanche, ces méthodes requièrent des dispositifs coûteuses et/ou ne fonctionnent pas en temps réel. Dans cette thèse, nous visions un système d'acquisition à bas coût, rapide et mobile, qui se veut non-seulement le moins intrusif possible mais aussi simple d'utilisation. La première contribution présentée dans cette thèse est une application de la méthode bien connue, intitulée stéréo photométrie, à la vidéo. De plus, comme une fréquence de trame élevée est nécessaire à une telle application, nous proposons une méthode permettant l'utilisation d'une illumination séquentielle avec des caméras rapides de type "electronic rolling shutter". Malgré les résultats intéressants obtenus, la qualité des reconstructions de l'apparence et de la forme n'étaient pas à la hauteur de nos espérances. De plus, la stéréo photométrie est une méthode qui, de nature, n'est pas très adaptée aux applications visées dans cette thèse. Pour notre seconde contribution, nous proposons une méthode de reconstruction de la forme (géométrie) ainsi que de la réflectance diffuse à partir d'une image (d'une séquence) en utilisant un système de capture hybride composé d'un capteur de profondeur (Kinect), d'une caméra grand public et d'un flash. L'objectif est de montrer qu'en combinant une acquisition RGB-D (image couleur + profondeur) avec illumination séquentielle, on peut obtenir une reconstruction qualitative de la forme et de la réflectance d'une scène dans le cas où l'éclairage n'est pas connu. Un couple d'images est capturé : une image non flashée (image sous une illumination ambiante) et une image flashée. Une image dont l'illumination ne provient que du flash (image flash pure) peut être calculée en soustrayant l'image non flashée de l'image flashée. Nous proposons un nouvel algorithme temps réel, qui, basé sur un modèle local d'illumination de notre flash et de l'image flash pure, améliore l'information de forme fournie par le capteur de profondeur tout en retrouvant les informations de réflectance diffuse. Notre dernière contribution concerne la composition automatique d'éclairage. L'éclairage est un élément clé de la photographie. Les professionnels travaillent régulièrement avec des systèmes d'éclairage complexes afin de capturer directement des images esthétiques. Récemment, certains photographes ont tenté une nouvelle approche : plutôt que photographier une scène directement sous un éclairage complexe, ils capturent la scène sous plusieurs éclairages simples, permettant ainsi un post-traitement permettant combiner les différentes illuminations de la scène. Cette approche apporte une nouvelle dimensionnalité intéressante au post-traitement. Cependant la combinaison des images requiert des compétences en matière de photographie, et l'acquisition sous différentes conditions d'éclairage n'en est pas moins fastidieuse. Nous proposons une méthode totalement automatisée, qui, à partir d'un modèle 3D (forme et albedo) reconstruit à partir de capture d'une scène réelle, produit virtuellement les images correspondant aux différentes conditions d'éclairages. Ensuite, ces images sont combinées automatiquement, à l'aide d'un algorithme génétique, pour correspondre à un style d'éclairage fourni par l'utilisateur sous forme d'une image cible de son choix. / The objective of this thesis is to take advantage of controlled illumination to enrich a video acquisition with shape and reflectance reconstructions. Today, a lot of works have tried to meet this objective. Some of them take advantage of sequential controlled illumintation to recover high quality shape and reflectance, however they either require a costly and very cumbersome fixed setup, and/or do not run in real-time. Our aim is a low cost, fast, mobile and simple acquisition setup which has to be the less intrusive possible so as to provide a greater ease of use. The first contribution of this thesis focuses on the application of the well known photometric stereo method to a video acquisition. Moreover, as a high frame rate is required by such an application, a method using sequential illumination with high frame rate cameras (electronic rolling shutter cameras) is also considered. Despite the interesting results provided by photometric stereo, we found that this latter did not provide enough qualitative results. Moreover, by its nature, photometric stereo is not really suitable for the range of applications targeted. We propose, as a second contribution, a method for recovering the shape (geometry) and the diffuse reflectance from an image (or video) using a hybrid setup consisting of a depth sensor (Kinect), a consumer camera and a partially controlled illumination (using a flash). The objective is to show how combining RGB-D acquisition with a sequential illumination is useful for shape and reflectance recovery. A pair of two images are captured : one non flashed (image under ambient illumination) and a flashed one. A pure flash image is computed by subtracting the non flashed image from the flashed image. We propose a novel and near real-time algorithm, based on a local illumination model of our flash and the pure flash image, to enhance geometry (from the noisy depth map) and recover reflectance information. Finally, our last contribution concerns an automatic method for light compositing, using rendered images. Lighting is a key element in photography. Professional photographers often work with complex lighting setups to directly capture an image close to the targeted one. Some photographers reversed this traditional workflow. Indeed, they capture the scene under several lighting conditions, then combine the captured images to get the expected one. Acquiring such a set of images is a tedious task and combining them requires some skill in photography. We propose a fully automatic method, that renders, based on a 3D reconstructed model (shape and albedo), a set of images corresponding to several lighting conditions. The resulting images are combined using a genetic optimization algorithm to match the desired lighting provided by the user as an image.