<p>3D character modeling is a crucial process of asset creation in the entertainment industry, particularly for animation and games. A fully automated pipeline via sketch-based 3D modeling (SBM) is an emerging possibility, but development is stalled by unrefined outputs and a lack of character-centered tools. This thesis proposes an improved method for constructing 3D character models with minimal user input, using only two sketch inputs  i.e., a front and side unshaded sketch. The system implements a deep convolutional neural network (CNN), a type of deep learning algorithm extending from artificial intelligence (AI), to process the input sketch and generate multi-view depth, normal and confidence maps that offer more information about the 3D surface. These are then fused into a 3D point cloud, which is a type of object representation for 3D space. This point cloud is converted into a 3D mesh via an occupancy network, involving another CNN, for a more precise 3D representation. This reconstruction step contends with non-deep learning approaches such as  Poisson reconstruction. The proposed system is evaluated for character generation on standardized quantitative metrics (i.e., Chamfer Distance [CD], Earth Mover’s Distance [EMD], F-score and Intersection of Union [IoU]), and compared to the base framework trained on the same character sketch and model database. This implementation offers a  significant improvement in the accuracy of vertex positions for the reconstructed character models. </p>

<p>Accurate 3D landscape models of cities or mountains have wide applications in mission</p> <p>planning, navigation, geological studies, etc. Lidar scanning using drones can provide high</p> <p>accuracy 3D landscape models, but the data is more expensive to collect as the area of</p> <p>each scan is limited. Thanks to recent maturation of Very-High-Resolution (VHR) optical</p> <p>imaging on satellites, people nowadays have access to stereo images that are collected on a</p> <p>much larger area than Lidar scanning. My research addresses unique challenges in satellite</p> <p>stereo, including stereo rectification with pushbroom sensors, dense stereo matching using</p> <p>image pairs with varied appearance, e.g. sun angles and surface plantation, and rasterized</p> <p>digital surface model (DSM) generation. The key contributions include the Continuous 3D-</p> <p>Label Semi-Global Matching (CoSGM) and a large scale dataset for satellite stereo processing</p> <p>and DSM evaluation.</p>

<p>    </p> <p>Image data remains an important tool for post-event building assessment and documentation. After each natural hazard event, significant efforts are made by teams of engineers to visit the affected regions and collect useful image data. In general, a global positioning system (GPS) can provide useful spatial information for localizing image data. However, it is challenging to collect such information when images are captured in places where GPS signals are weak or interrupted, such as the indoor spaces of buildings. An inability to document the images’ locations would hinder the analysis, organization, and documentation of these images as they lack sufficient spatial context. This problem becomes more urgent to solve for the inspection mission covering a large area, like a community. To address this issue, the objective of this research is to generate a tool to automatically process the image data collected during such a mission and provide the location of each image. Towards this goal, the following tasks are performed. First, I develop a methodology to localize images and link them to locations on a structural drawing (Task 1). Second, this methodology is extended to be able to process data collected from a large scale area, and perform indoor localization for images collected on each of the indoor floors of each individual building (Task 2). Third, I develop an automated technique to render the damage condition decision of buildings by fusing the image data collected within (Task 3). The methods developed through each task have been evaluated with data collected from real world buildings. This research may also lead to automated assessment of buildings over a large scale area. </p>

<p>In this dissertation, we dedicate ourselves to applying deep-learning-based computer vision algorithms to industrial applications in 2D and 3D image processing. More specifically, we present the application of deep-learning-based image processing algorithms to the following three topics: RGB-image-based shipping box defect detection, RGB-image-based patients' bed-side status monitoring, and an RGBD-image-based 3D surface video conferencing system. These projects cover 2D image detection of static objects in industrial scenarios, 2D detection of dynamic human images in bedroom environments, and accurate 3D reconstruction of dynamic humanoid objects in video conferencing. In each of these projects, we proposed ready-to-deploy pipelines combining deep learning and traditional computer vision algorithms to improve the overall performance of industrial products. In each chapter, we describe in detail how we utilize, modify, and enhance the architecture of convolutional neural networks, including the training techniques using data acquisition, image annotation, synthetic datasets, and other schemes. In the relevant sections, we also present how post-processing steps with image processing algorithms can improve the overall effectiveness of the algorithm. We hope that our work demonstrates the versatility and advantages of deep neural networks in both 2D and 3D computer vision applications.</p>

This thesis covers the topic of depth- assisted Structure from Motion (SfM). When performing classic SfM, the goal is to reconstruct a 3D scene using only a set of unstructured RGB images. What is attempted to be achieved in this thesis is adding the depth dimension to the problem formulation, and consequently create a system that can receive a set of RGBD images. The problem has been addressed by modifying an already existing SfM pipeline and in particular, its Bundle Adjustment (BA) process. Comparisons between the modified framework and the baseline framework resulted in conclusions regarding the impact of the modifications. The results show mainly two things. First of all, the accuracy of the framework is increased in most situations. The difference is the most significant when the captured scene only is covered from a small sector. However, noisy data can cause the modified pipeline to decrease in performance. Secondly, the run time of the framework is significantly reduced. A discussion of how to modify other parts of the pipeline is covered in the conclusion of the report. / Följande examensarbete behandlar ämnet djupassisterad Struktur genom Rörelse (eng. SfM). Vid klassisk SfM är målet att återskapa en 3D scen, endast med hjälp av en sekvens av oordnade RGB bilder. I djupassiterad SfM adderas djupinformationen till problemformulering och följaktligen har ett system som kan motta RGBD bilder skapats. Problemet har lösts genom att modifiera en befintlig SfM- mjukvara och mer specifikt dess Buntjustering (eng. BA). Resultatet från den modifierade mjukvaran jämförs med resultatet av originalutgåvan för att dra slutsatser rådande modifikationens påverkan på prestandan. Resultaten visar huvudsakligen två saker. Först och främst, den modifierade mjukvaran producerar resultat med högre noggrannhet i de allra flesta fall. Skillnaden är som allra störst när bilderna är tagna från endast en liten sektor som omringar scenen. Data med brus kan dock försämra systemets prestanda aningen jämfört med orginalsystemet. För det andra, så minskar exekutionstiden betydligt. Slutligen diskuteras hur mjukvaran kan vidareutvecklas för att ytterligare förbättra resultaten.

Indoor environment modeling has become a relevant topic in several applications fields including Augmented, Virtual and Mixed Reality. Furthermore, with the Digital Transformation, many industries have moved toward this technology trying to generate detailed models of an environment allowing the viewers to navigate through it or mapping surfaces to insert virtual elements in a real scene. Therefore, this Thesis project has been conducted with the purpose to review well- established deterministic methods for 3D scene reconstruction and researching the state- of- the- art, such as machine learning- based approaches, and a possible implementation on mobile devices. Initially, we focused on the well- established methods such as Structure from Motion (SfM) that use photogrammetry to estimate camera poses and depth using only RGB images. Lastly, the research has been centered on the most innovative methods that make use of machine learning to predict depth maps and camera poses from a video stream. Most of the methods reviewed are completely unsupervised and are based on a combination of two subnetwork, the disparity network (DispNet) for the depth estimation and pose network (PoseNet) for camera pose estimation. Despite the fact that the results in outdoor application show high quality depth map and and reliable odometry, there are still some limitations for the deployment of this technology in indoor environment. Overall, the results are promising. / Modellering av inomhusmiljö har blivit ett relevant ämne inom flera applikationsområden, inklusive Augmented, Virtual och Mixed Reality. Dessutom, med den digitala transformationen, har många branscher gått mot denna teknik som försöker generera detaljerade modeller av en miljö som gör det möjligt för tittarna att navigera genom den eller kartlägga ytor för att infoga virtuella element i en riktig scen. Därför har detta avhandlingsprojekt genomförts med syftet att granska väletablerade deterministiska metoder för 3Dscenrekonstruktion och undersöka det senaste inom teknik, såsom maskininlärningsbaserade metoder och en möjlig implementering på mobil. Inledningsvis fokuserade vi på de väletablerade metoderna som Structure From Motion (SfM) som använder fotogrammetri för att uppskatta kameraställningar och djup med endast RGBbilder. Slutligen har forskningen varit inriktad på de mest innovativa metoderna som använder maskininlärning för att förutsäga djupkartor och kameraposer från en videoström. De flesta av de granskade metoderna är helt utan tillsyn och baseras på en kombination av två undernätverk, skillnadsnätverket (DispNet) för djupuppskattning och posenätverk (PoseNet) för kameraposestimering. Trots att resultaten i utomhusanvändning visar djupkarta av hög kvalitet och tillförlitlig vägmätning, finns det fortfarande vissa begränsningar för användningen av denna teknik i inomhusmiljön, men ändå är resultaten lovande.

In den letzten Jahren sind immer mehr erschwingliche Kamera-Sensoren mit einer zunehmenden Vielfalt optischer Abbildungsfunktionen verfügbar geworden. Low-Cost-Optiken können aufgrund höherer Toleranzen und unterschiedlicher optischer Materialien von der gewünschten Lochkamera Metrik abweichen. Weitwinkel- und Fischaugenobjektive, verzerrende katadioptrische Objektive (spiegelnd und refraktiv) und andere ungewöhnliche Objektive weichen von der Annahme des Modells einer Lochkamera mit einer Brennweite ab. Actionkameras können die gesamte Umgebung mit zwei Objektiven abbilden, diese entsprechen meist nicht mehr dem Lochkameramodell. Kameras werden auch für Messaufgaben hinter zusätzlichen optischen Elementen eingesetzt. Die vorliegende Arbeit erweitert die ersten Erkenntnisse im Bereich der differenzierbaren (glatten) Kameramodelle ohne Einschränkungen. Viele existierende Modelle sind auf bestimmte Objektivtypen spezialisiert. In dieser Arbeit werden mehrere solcher allgemeinen Modelle eingeführt, ohne dass eine global feste Brennweite und spezielle Anforderungen an die Symmetrie der Abbildung erforderlich sind. Eine Einführung alternativer Fehlermetriken im Objektraum bringt auch enorme Rechenvorteile, da eine Abbildungsrichtung analytisch berechnet und viele der Berechnungsergebnisse konstant gehalten werden können. Zur Initialisierung solcher Modelle wird in dieser Arbeit eine generische lineare Kamera vorgestellt. Das wesentliche Merkmal dabei ist eine künstliche Transformation in höhere Dimensionen, welche mit linearen Verfahren weiterverwendet werden. Sie modellieren bereits nichtlineare Verzerrungen und Asymmetrien. Eine Multikamera-Kalibrierungssoftware wird ebenfalls beschrieben und implementiert. Das Ergebnis der Arbeit ist ein theoretischer Rahmen für glatte Kameramodelle im Objektraum selbst – anstelle der Abbildung in den Bildraum – mit mehreren konkreten Modellvorschlägen, Implementierungen und dem angepassten und erweiterten Kalibrierungsprozess. / In recent years, more and more affordable camera sensors with an increasing variety of optical imaging features have become available. Low-cost optics may deviate from the desired pinhole metric due to higher tolerances and different optical materials. Wide-angle and fisheye lenses, distorting catadioptric lenses (specular and refractive) and other unusual lenses deviate from the single focal pinhole camera model assumption, which is sometimes intentional. Action cameras can map the entire environment using two lenses, these usually no longer correspond to the pinhole camera model. Cameras are also used for measuring tasks behind additional optical elements – with unforeseeable deviations in the line of sight. The present work expands the first findings in the field of differentiable (smooth) camera models without constraints. Many existing models specialise in certain types of lenses. In this work, several such general models are introduced without requiring fixed global focal length and symmetry requirements. An introduction of alternative error metrics in the object space also gives enormous computational advantages, since one imaging direction can be calculated analytically and many of the calculation results can be kept constant. For the generation of meaningful starting values of such models, this work introduces a generic linear camera. The essential feature of is an artificial transformation into higher dimensions. These transformed coordinates can then continue to be used with linear methods. They already model non-linear distortions and asymmetries. A multi-camera calibration software that efficiently implements these models is also described and implemented. The result of the work is a theoretical framework for smooth camera models in the object space itself - instead of the established mapping into the image space - with several concrete model proposals, implementations and the adapted and extended calibration process.

An Hand einer Reihe Ontogenesestadien konnte die Entwicklung ausgewählter Bereiche des Schädels von Macroscelides proboscideus dargestellt werden. Die Befunde der Craniogenese vom Macroscelides proboscideus wurden mit Elephantulus spec., anderen Macroscelidea und Orycteropus afer verglichen. In der vorliegenden Arbeit wurden bisher nicht beschriebene Synapomorpien der Macroscelidea dargestellt, die einen Beitrag zur Erstellung des Grundmusters dieser Gruppe leisten. Es konnten folgende Befunde erhoben werden: Bei Macroscelides proboscideus wurde ein bisher nur bei Elephantulus brachyrhynchus und Elephantulus myurus beschriebener Knorpel gefunden: diese Cartilago lateralis dient bei Macroscelides proboscideus als rostraler Ursprung des Musculus maxillo-labialis ventralis. Wird dieser Befund für weitere Vertreter der Sengis bestätigt, könnte es sich um eine Autapomorphie der Macroscelidea handeln. Ein fehlender Processus postorbitalis des Frontale scheint für Macroscelides proboscideus abgeleitet zu sein. Das Foramen subopticum, das bei den Orycteropodidae fehlt, ist eine Apomorphie der Sengis. Die eindeutig basale Position der Macroscelidea innerhalb des Stammbaums der Afrotheria, die auf wenigen Fossilien gestützte phylogenetische Entwicklung, die bis heute nicht geklärten intrafamiliären Beziehungen innerhalb der Macroscelididae und die fehlenden cranialen Apomorphien der Afrotheria lassen viel Raum für weitergehende vergleichende Untersuchungen. Es wurden histologische Schnittserien digital fotografiert und als Grundlage für eine computergestützte 3D-Rekonstruktion verwendet, so dass ein digitales 3D-Modell eines Stadiums der Sengis für Vergleichszwecke vorliegt. / Sengis (formerly known as elephant shrews) represent a monophyletic group within Afrotheria. An ontogenetic study on cranial development was conducted by histological slicing 7 individuals of Macroscelides proboscideus and Elephantulus spec. Through the development of an computer aided 3D- computer model following aspects could be found: The Cartilago lateralis, a separate cartilage holding the rostral beginning of the Musculus maxillo-labialis ventralis could be described for two species of Sengis and might turn out to be a cranial autapomophie for Macroscelidea. The in Oryctepodiadae missing Foramen subopticum is present in all Sengis. The basal position of Sengis within Afrotheria and the still missing cranial Apomorphies for the Afrotheria leave much space for further discussions.

Despite advances in ground-based Mobile Mapping System (MMS), automatic feature reconstruction seems far from being reached. In this thesis, we focus on 3D roadmark reconstruction from images acquired by road looking cameras of a MMS stereo-rig in dense urban context. A new approach is presented, that uses 3D geometric knowledge of roadmarks and provides a centimetric 3D accuracy with a low level of generalisation. Two classes of roadmarks are studied: zebra-crossing and dashed-lines. The general strategy consists in three main steps. The first step provides 3D linked-edges. Edges are extracted in the left and right images. Then a matching algorithm that is based on dynamic programming optimisation matches the edges between the two images. A sub-pixel matching is computed by post processing and 3D linked-edges are provided by classical photogrammetric triangulation. The second step uses the known specification of roadmarks to perform a signature based filtering of 3D linked-edges. This step provides hypothetical candidates for roadmark objects. The last step can be seen as a validation step that rejects or accepts the candidates. The validated candidates are finely reconstructed. The adopted model consists of a quasi parallelogram for each strip of zebra-crossing or dashed-line. Each strip is constrained to be flat but the roadmark as a whole is not planar. The method is evaluated on a set of 150 stereo-pairs acquired in a real urban area under normal traffic conditions. The results show the validity of the approach in terms of robustness, completeness and geometric accuracy. The method is robust and deals properly with partially occluded roadmarks as well as damaged or eroded ones. The detection rate reaches 90% and the 3D accuracy is about 2-4 cm. Finally an application of reconstructed roadmarks is presented. They are used in georeferencing of the system. Most of the MMSs use direct georeferencing devices such as GPS/INS for their localisation. However in urban areas masks and multi-path errors corrupt the measurements and provide only 50 cm accuracy. In order to improve the localisation quality, we aim at matching ground-based images with calibrated aerial images of the same area. For this purpose roadmarks are used as matching objects. The validity of this method is demonstrated on a zebra-crossing example

La scoliose idiopathique de l’adolescent est une déformation 3D du rachis. La littérature comporte une multitude d’études sur la prédiction de l’évolution et l’identification de facteurs de risque de progression. Pour l’instant les facteurs de risque établis sont l’amplitude de la déformation, la maturité squelettique et le type de courbure. Plusieurs autres champs ont été explorés comme les aspects génétiques, biochimiques, mécaniques, posturaux et topographiques, sans vraiment apporter beaucoup de précision à la prédiction de l’évolution. L’avancement de la technologie permet maintenant de générer des reconstructions 3D du rachis à l’aide des radiographies standard et d’obtenir des mesures de paramètres 3D. L’intégration de ces paramètres 3D dans un modèle prédictif représente une avenue encore inexplorée qui est tout à fait logique dans le contexte de cette déformation 3D du rachis. L’objectif général de cette thèse est de développer un modèle de prédiction de l’angle de Cobb à maturité squelettique à partir de l’information disponible au moment de la première visite, soit l’angle de Cobb initial, le type de courbure, l’âge osseux et des paramètres 3D du rachis. Dans une première étude, un indice d’âge osseux a été développé basé sur l’ossification de l’apophyse iliaque et sur le statut du cartilage triradié. Cet indice comporte 3 stades et le second stade, qui est défini par un cartilage triradié fermé avec maximum 1/3 d’ossification de l’apophyse iliaque, représente le moment pendant lequel la progression de la scoliose idiopathique de l’adolescent est la plus rapide. Une seconde étude rétrospective a permis de mettre en évidence le potentiel des paramètres 3D pour améliorer la prédiction de l’évolution. Il a été démontré qu’à la première visite il existe des différences pour 5 paramètres 3D du rachis entre un groupe de patients qui sera éventuellement opéré et un groupe qui ne progressera pas. Ces paramètres sont : la moyenne da la cunéiformisation 3D des disques apicaux, la rotation intervertébrale à la jonction inférieure de la courbure, la torsion, le ratio hauteur/largeur du corps vertébral de T6 et de la colonne complète. Les deux dernières études sont basées sur une cohorte prospective de 133 patients avec une scoliose idiopathique de l’adolescent suivi dès leur première visite à l’hôpital jusqu’à maturité squelettique. Une première étude a permis de mettre en évidence les différences morphologiques à la première visite entre les patients ayant progresser de plus ou moins de 6°. Des différences ont été mise en évidence pour la cyphose, l’angle de plan de déformation maximal, la rotation ntervertébrale l’apex, la torsion et plusieurs paramètres de «slenderness». Ensuite une seconde étude a permis de développer un modèle prédictif basé sur un modèle linéaire général en incluant l’indice d’âge osseux développé dans la première étude, le type de courbure, l’amplitude de l’angle de Cobb à la première visite, l’angle de déformation du plan maximale, la cunéiformisation 3D des disques T3-T4, T8-­T9, T11-­T12 et la somme des cunéiformisation 3D de tous les disques thoraciques et lombaires. Le coefficient de détermination multiple pour cette modélisation est de 0.715. Le modèle prédictif développé renforce l’importance de considérer la scoliose idiopathique dans les trois dimensions et il permettra d’optimiser la prédiction de l’évolution au moment de la première visite. / Prediction of curve progression remains a challenge for the clinicians at the first visit for a patient with adolescent idiopathic scoliosis. Prediction of progression is based on curve type, curve magnitude and skeletal or chronological age. The failure to accurately predict the risk of progression can lead to inadequate treatment, as well as unnecessary medical visits and radiographs. Three-­dimensional evaluation is currently more popular in the scoliosis research community either for classification or treatment planning. The global objective of this thesis was to develop a predictive model of the final Cobb angle in adolescent idiopathic scoliosis based on 3D spine parameters and on skeletal age, curve type and curve magnitude. The first study was to develop a skeletal maturity system based on ossification of iliac apophysis and triradiate cartilage status, two details available on routine radiographs. A 3 stages system was develop with the second stage being associated to the rapid acceleration phase of the deformation. The second stage corresponds to a closed triradiate cartilage and a maximum of 1/3 of the iliac apophysis ossification. The second study was a retrospective case control study. It showed the potential for 3D parameters to help predict the evolution. 3D spinal parameters at first visit of a group of patient eventually treated with surgery and a group who had no progression revealed statistical differences for: mean 3D wedging of the apical disks, lower junctional vertebral axial rotation, torsion and T6 and whole spine height/width ratio were all significantly affected. The last study was a prospective cohort study based on an adolescent idiopathic scoliosis group followed from first visit to skeletal maturity. A general linear model was developed based on 3D spinal parameters and on curve type, Cobb angle at first visit and with the 3 stages skeletal maturity system developed in the first study. The predictive model obtained has a determination coefficient of 0,715. Included 3D parameters predictors were: the angle of the plane of maximal curvature, the 3D wedging of T3-­T4, T8-­T9 and T11-­T12 disks, and the sum of 3D wedging of all thoracic and lumbar disks. The predictive model developed reinforced the importance of considering adolescent idiopathic scoliosis in a three-­‐dimensional point of view. This model will also optimise the prediction of evolution at first visit.