In this thesis we explore the concepts and components which can be used as individual building blocks for producing immersive virtual reality (VR) content from a single RGB-D sensor. We identify the properties of immersive VR videos and propose a system composed of a foreground/background separator, a dynamic scene re-constructor and a shape completer. We initially explore the foreground/background separator component in the context of video summarization. More specifically, we examined how to extract trajectories of moving objects from video sequences captured with a static camera. We then present a new approach for video summarization via minimization of the spatial-temporal projections of the extracted object trajectories. New evaluation criterion are also presented for video summarization. These concepts of foreground/background separation can then be applied towards VR scene creation by extracting relative objects of interest. We present an approach for the dynamic scene re-constructor component using a single moving RGB-D sensor. By tracking the foreground objects and removing them from the input RGB-D frames we can feed the background only data into existing RGB-D SLAM systems. The result is a static 3D background model where the foreground frames are then super-imposed to produce a coherent scene with dynamic moving foreground objects. We also present a specific method for extracting moving foreground objects from a moving RGB-D camera along with an evaluation dataset with benchmarks. Lastly, the shape completer component takes in a single view depth map of an object as input and "fills in" the occluded portions to produce a complete 3D shape. We present an approach that utilizes a new data minimal representation, the additive depth map, which allows traditional 2D convolutional neural networks to accomplish the task. The additive depth map represents the amount of depth required to transform the input into the "back depth map" which would exist if there was a sensor exactly opposite of the input. We train and benchmark our approach using existing synthetic datasets and also show that it can perform shape completion on real world data without fine-tuning. Our experiments show that our data minimal representation can achieve comparable results to existing state-of-the-art 3D networks while also being able to produce higher resolution outputs.

Recent advancements in computer vision have led to the emergence of Neural Radiance Fields (NeRFs), a powerful tool for reconstructing photorealistic 3D scenes, even in dynamic settings. However, these methods struggle when dealing with human subjects, especially when the subject is partially obscured or not completely observable, resulting in inaccurate reconstructions of geometries and textures. To address this issue, this thesis evaluates state-of-the-art human modeling using implicit representations with partial observability of the subject. We then propose and test several novel methods to improve the generalization of these models, including the use of symmetry and Signed Distance Function (SDF) driven losses and leveraging prior knowledge from multiple subjects via a pre-trained model. Our results demonstrate that our proposed methods significantly improve the accuracy of the reconstructions, even in challenging ”in-the-wild” situations, both quantitatively and qualitatively. Our approach opens new opportunities for applications such as asset generation for video games and movies and improved simulations for autonomous driving scenarios from abundant in-the-wild monocular videos. In summary, our research presents a significant improvement to the state-of-the-art human modeling using implicit representations, with important implications for 3D Computer Vision (CV) and Neural Rendering and its applications in various industries. / De senaste framstegen inom datorseende har lett till uppkomsten av Neural Radiance Fields (NeRFs), ett kraftfullt verktyg för att rekonstruera fotorealistiska 3D-scener, även i dynamiska miljöer. Dessa metoder brister dock vid hantering av människor, särskilt när människan är delvis skymd eller inte helt observerbar, vilket resulterar i felaktiga rekonstruktioner av geometrier och texturer. För att ta itu med denna fråga, utvärderar denna avhandling toppmodern mänsklig modellering med hjälp av implicita representationer med partiell observerbarhet av ämnet. Vidare föreslår, samt testar vi, flertalet nya metoder för att förbättra generaliseringen av dessa modeller, inklusive användningen av symmetri och SDF-drivna förluster och utnyttjandet av förkunskaper från flera individer via en förtränad modell. Resultaten visar att våra föreslagna metoder avsevärt förbättrar rekonstruktionernas noggrannhet, även i utmanande ”in-the-wild” situationer, både kvantitativt och kvalitativt. Vårt tillvägagångssätt skapar nya möjligheter för applikationer som tillgångsgenerering för videospel och filmer och förbättrade simuleringar för scenarier för autonom körning från rikliga monokulära videor. Sammanfattningsvis, presenterar vår forskning en betydande förbättring av toppmodern modelleringen med hjälp av implicita representationer, med viktiga implikationer för 3D CV och neural rendering och dess tillämpningar i olika industrier.

Various Image Based Rendering (IBR) techniques have been proposed to reconstruct scenes from its images. Voxel-based IBR algorithms reconstruct Lambertian scenes well, but fail for specular objects due to limitations of their consistency checks. We show that the conventional consistency techniques fail due to the large variation in reflected color of the surface for different viewing positions. We present a new consistency approach that can predict this variation in color and reconstruct specular objects present in the scene. We also present an evaluation of our technique by comparing it with three other consistency methods.

This thesis deals with the problem of three dimensional scene reconstruction from multiple camera images. This is a well established problem in computer vision and has been significantly researched. In recent years some excellent results have been achieved, however existing algorithms often fall short of many biological systems in terms of robustness and generality. The aim of this research was to develop improved algorithms for reconstructing 3D scenes, with a focus on accurate system modelling and correctly dealing with occlusions. With scene reconstruction the objective is to infer scene parameters describing the 3D structure of the scene from the data given by camera images. This is an illposed inverse problem, where an exact solution cannot be guaranteed. The use of a statistical approach to deal with the scene reconstruction problem is introduced and the differences between maximum a priori (MAP) and minimum mean square estimate (MMSE) considered. It is discussed how traditional stereo matching can be performed using a volumetric scene model. An improved model describing the relationship between the camera data and a discrete model of the scene is presented. This highlights some of the common causes of modelling errors, enabling them to be dealt with objectively. The problems posed by occlusions are considered. Using a greedy algorithm the scene is progressively reconstructed to account for visibility interactions between regions and the idea of a complete scene estimate is established. Some simple and improved techniques for reliably assigning opaque voxels are developed, making use of prior information. Problems with variations in the imaging convolution kernel between images motivate the development of a pixel dissimilarity measure. Belief propagation is then applied to better utilise prior information and obtain an improved global optimum. A new volumetric factor graph model is presented which represents the joint probability distribution of the scene and imaging system. By utilising the structure of the local compatibility functions, an efficient procedure for updating the messages is detailed. To help convergence, a novel approach of accentuating beliefs is shown. Results demonstrate the validity of this approach, however the reconstruction error is similar or slightly higher than from the Greedy algorithm. To simplify the volumetric model, a new approach to belief propagation is demonstrated by applying it to a dynamic model. This approach is developed as an alternative to the full volumetric model because it is less memory and computationally intensive. Using a factor graph, a volumetric known visibility model is presented which ensures the scene is complete with respect to all the camera images. Dynamic updating is also applied to a simpler single depth-map model. Results show this approach is unsuitable for the volumetric known visibility model, however, improved results are obtained with the simple depth-map model.

This thesis deals with the reconstruction of the scene using two or more images. It describes the whole reconstruction process consisting of detecting points in images, finding the appropriate geometry between images and resulting projection of these points into the space of scene. The thesis also includes a description of the application, which demonstrates the described methods.

The causes of errors in the angle of impact calculation were investigated including the surface type, falling velocity and the method used to fit an ellipse to a bloodstain. As had been cited previously the angle of impact was generally underestimated, especially at acute angles and the reason for this was determined to be due to an overestimation of the length of a bloodstain. The surface type was found to significantly affect the accuracy of an angle of impact calculation and as the falling velocity increased, the angle of impact calculation became more accurate. High-speed photography was used to further investigate the formation of bloodstains on surfaces. It was found that the formation of the bloodstain varied depending on the surface type and the angle of the surface. Bloodstain pattern analysis involves the application of scientific techniques to reconstruct events that resulted in a bloodstain pattern. The position of the blood source in three-dimensional space is a fundamental element of this application. Currently little is known about the methods used by bloodstain pattern analysts to select bloodstains when determining the region of origin. Fourteen analysts worldwide were surveyed in order to ascertain this information. It was found that the methods used were variable and were often not based on scientific research. Research was therefore undertaken into bloodstain selection and in particular, which bloodstains should be selected for a region of origin analysis. As a result of these experiments, two sets of selection criteria were established, one for use when the region of origin is being calculated manually and one for when directional analysis is being used.

The causes of errors in the angle of impact calculation were investigated including the surface type, falling velocity and the method used to fit an ellipse to a bloodstain. As had been cited previously the angle of impact was generally underestimated, especially at acute angles and the reason for this was determined to be due to an overestimation of the length of a bloodstain. The surface type was found to significantly affect the accuracy of an angle of impact calculation and as the falling velocity increased, the angle of impact calculation became more accurate. High-speed photography was used to further investigate the formation of bloodstains on surfaces. It was found that the formation of the bloodstain varied depending on the surface type and the angle of the surface. Bloodstain pattern analysis involves the application of scientific techniques to reconstruct events that resulted in a bloodstain pattern. The position of the blood source in three-dimensional space is a fundamental element of this application. Currently little is known about the methods used by bloodstain pattern analysts to select bloodstains when determining the region of origin. Fourteen analysts worldwide were surveyed in order to ascertain this information. It was found that the methods used were variable and were often not based on scientific research. Research was therefore undertaken into bloodstain selection and in particular, which bloodstains should be selected for a region of origin analysis. As a result of these experiments, two sets of selection criteria were established, one for use when the region of origin is being calculated manually and one for when directional analysis is being used.

La modélisation géométrique et la sémantisation de scènes intérieures à partir d'échantillon de points et un sujet de recherche qui prend de plus en plus d'importance. Cependant, le traitement d'un ensemble volumineux de données est rendu difficile d'une part par le nombre élevé d'objets parasitant la scène et d'autre part par divers défauts d'acquisitions comme par exemple des données manquantes ou un échantillonnage de la scène non isotrope. Cette thèse s'intéresse de près à de nouvelles méthodes permettant de modéliser géométriquement un nuage de point non structuré et d’y donner de la sémantique. Dans le chapitre 2, nous présentons deux méthodes permettant de transformer le nuage de points en un ensemble de formes. Nous proposons en premier lieu une méthode d'extraction de lignes qui détecte des segments à partir d'une coupe horizontale du nuage de point initiale. Puis nous introduisons une méthode par croissance de régions qui détecte et renforce progressivement des régularités parmi les formes planaires. Dans la première partie du chapitre 3, nous proposons une méthode basée sur de l'analyse statistique afin de séparer de la structure de la scène les objets la parasitant. Dans la seconde partie, nous présentons une méthode d'apprentissage supervisé permettant de classifier des objets en fonction d'un ensemble de formes planaires. Nous introduisons dans le chapitre 4 une méthode permettant de modéliser géométriquement le volume d'une pièce (sans meubles). Une formulation énergétique est utilisée afin de labelliser les régions d’une partition générée à partir de formes élémentaires comme étant intérieur ou extérieur de manière robuste au bruit et aux données. / Geometric modeling and semantization of indoor scenes from sampled point data is an emerging research topic. Recent advances in acquisition technologies provide highly accurate laser scanners and low-cost handheld RGB-D cameras for real-time acquisition. However, the processing of large data sets is hampered by high amounts of clutter and various defects such as missing data, outliers and anisotropic sampling. This thesis investigates three novel methods for efficient geometric modeling and semantization from unstructured point data: Shape detection, classification and geometric modeling. Chapter 2 introduces two methods for abstracting the input point data with primitive shapes. First, we propose a line extraction method to detect wall segments from a horizontal cross-section of the input point cloud. Second, we introduce a region growing method that progressively detects and reinforces regularities of planar shapes. This method utilizes regularities common to man-made architecture, i.e. coplanarity, parallelism and orthogonality, to reduce complexity and improve data fitting in defect-laden data. Chapter 3 introduces a method based on statistical analysis for separating clutter from structure. We also contribute a supervised machine learning method for object classification based on sets of planar shapes. Chapter 4 introduces a method for 3D geometric modeling of indoor scenes. We first partition the space using primitive shapes detected from permanent structures. An energy formulation is then used to solve an inside/outside labeling of a space partitioning, the latter providing robustness to missing data and outliers.

Image based modeling using visual images has undergone major development during the earlier parts of the 21th century. In this thesis a system for automated uncalibrated scene reconstruction using infrared images is implemented and tested. An automated reconstruction system could serve to simplify thermal inspection or as a demonstration tool. Thermal images will in general have lower resolution, less contrast and less high frequency content as compared to visual images. These characteristics of infrared images further complicates feature extraction and matching, key steps in the reconstruction process. In order to remedy the complication preprocessing methods are suggested and tested as well. Infrared modeling will also impose additional demands on the reconstruction as it is of importance to maintain thermal accuracy of the images in the product. Three main results are obtained from this thesis. Firstly, it is possible to obtain camera calibration and pose as well as a sparse point cloud reconstruction from an infrared image sequence using the suggested implementation. Secondly, correlation of thermal measurements from the images used to reconstruct three dimensional coordinates is presented and analyzed. Lastly, from the preprocessing evaluation it is concluded that the tested methods are not suitable. The methods will increase computational cost while improvements in the model are not proportional. / Bildbaserad modellering med visuella bilder har genomgått en stor utveckling under de tidigare delarna av 2000-talet. Givet en sekvens bestående av vanliga tvådimensionella bilder på en scen från olika perspektiv så är målet att rekonstruera en tredimensionell modell. I denna avhandling implementeras och testas ett system för automatiserad okalibrerad scenrekonstruktion från infraröda bilder. Okalibrerad rekonstruktion refererar till det faktum att parametrar för kameran, såsom fokallängd och fokus, är okända och enbart bilder används som indata till systemet. Ett stort användingsområde för värmekameror är inspektion. Temperaturskillnader i en bild kan indikera till exempel dålig isolering eller hög friktion. Om ett automatiserat system kan skapa en tredimensionell modell av en scen så kan det bidra till att förenkla inspektion samt till att ge en bättre överblick. Värmebilder kommer generellt att ha lägre upplösning, mindre kontrast och mindre högfrekvensinnehåll jämfört med visuella bilder. Dessa egenskaper hos infraröda bilder komplicerar extraktion och matchning av punkter i bilderna vilket är viktiga steg i rekonstruktionen. För att åtgärda komplikationen förbehandlas bilderna innan rekonstruktionen, ett urval av metoder för förbehandling har testats. Rekonstruktion med värmebilder kommer också att ställa ytterligare krav på rekonstruktionen, detta eftersom det är viktigt att bibehålla termisk noggrannhet från bilderna i modellen. Tre huvudresultat erhålls från denna avhandling. För det första är det möjligt att beräkna kamerakalibrering och position såväl som en gles rekonstruktion från en infraröd bildsekvens, detta med implementationen som föreslås i denna avhandling. För det andra presenteras och analyseras korrelationen för temperaturmätningar i bilderna som används för rekonstruktionen. Slutligen så visar den testade förbehandlingen inte en förbättring av rekonstruktionen som är propotionerlig med den ökade beräkningskomplexiteten.

No description available.