The ability to reconstruct 3D scenes of environments is of great interest in a number of fields such as autonomous driving, surveillance, and virtual reality. However, traditional methods often rely on multiple cameras or sensor-based depth measurements to accurately reconstruct 3D scenes. In this thesis we propose an alternative, deep learning-based approach to 3D scene reconstruction for objects of interest, using nothing but single RGB images. We evaluate our approach using the Deep Object Pose Estimation (DOPE) neural network for object detection and pose estimation, and the NVIDIA Deep learning Dataset Synthesizer for synthetic data generation. Using two unique objects, our results indicate that it is possible to reconstruct 3D scenes from single RGB images within a few centimeters of error margin.

京都大学 / 新制・課程博士 / 博士(医学) / 甲第24477号 / 医博第4919号 / 新制||医||1062(附属図書館) / 京都大学大学院医学研究科医学専攻 / (主査)教授 中本 裕士, 教授 波多野 悦朗, 教授 万代 昌紀 / 学位規則第4条第1項該当 / Doctor of Medical Science / Kyoto University / DFAM

3D modelling is a technology in massive demand now and can potentially become a key factor for enabling subsequent technological evolutions such as metaverses, digital twins, and virtual reality. Current 3D modellings include high-precision 3D human body modelling and rapid modelling through single or multiple monocular photos. However, some problems persist in both modellings. The modelling based on high-precision equipment has low practicability, few applicable scenarios, and high cost. Modelling through monocular photos, on the other hand, has low accuracy and is sensitive to noisy data. And both modellings generate static 3D models. Therefore, to realize the model's dynamic effect in various fields while retaining fast modelling, we propose a system that recovers a 3D model from a single photo to fuse skeleton animation extracted from videos, for a realization of the Digital Twin (DT). DT is defined as "digital replications of living as well as non-living entities that enable data to be seamlessly transmitted between the physical and virtual worlds". Rigging is setting up the skeleton-based animation to combine the 3D model and skeleton animation. Traditional rigging method is time-consuming and non-reusable, since rigging is often done manually or semi-automatically. In this thesis, we propose an automatic rigging method to achieve a loose coupling fusion of one-to-many or many-to-one 3D models and skeletal animations. Our rigging method is fast and efficient, and only needs a single photo as input.

As more data sources have become abundantly available, an increased interest in 3D reconstruction has emerged in the image processing academic community. Applications for 3D reconstruction of urban and residential buildings consist of urban planning, network planning for mobile communication, tourism information systems, spatial analysis of air pollution and noise nuisance, microclimate investigations, and Geographical Information Systems (GISs). Previous, classical, 3D reconstruction algorithms solely utilized aerial photography. With the advent of LIDAR systems, current algorithms explore using captured LIDAR data as an additional feasible source of information for 3D reconstruction. Preprocessing techniques are proposed for the development of an autonomous 3D Reconstruction algorithm. The algorithm is designed for autonomously deriving three dimensional models of urban and residential buildings from raw LIDAR data. First, a greedy insertion triangulation algorithm, modified with a proposed noise filtering technique, triangulates the raw LIDAR data. The normal vectors of those triangles are then passed to an unsupervised clustering algorithm – Fuzzy Simplified Adaptive Resonance Theory (Fuzzy SART). Fuzzy SART returns a rough grouping of coplanar triangles. A proposed multiple regression algorithm then further refines the coplanar grouping by further removing outliers and deriving an improved planar segmentation of the raw LIDAR data. Finally, further refinement is achieved by calculating the intersection of the best fit roof planes and moving nearby points close to that intersection to exist at the intersection, resulting in straight roof ridges. The end result of the aforementioned techniques culminates in a well defined model approximating the considered building depicted by the LIDAR data.

Humans perceive the world in three dimensions, but many sensing capabilities only display two-dimensional information to users by way of images. In this work we develop two novel reconstruction techniques utilizing synthetic aperture radar (SAR) data in three dimensions given sparse amounts of available data. We additionally leverage a hybrid joint-sparsity and sparsity approach to remove a-priori influences on the environment and instead explore general imaging properties in our reconstructions. We evaluate the required sampling rates for our techniques and a thorough analysis of the accuracy of our methods. The results presented in this thesis suggest a solution to sparse three-dimensional object reconstruction that effectively uses a substantially less amount of phase history data (PHD) while still extracting critical features off an object of interest.

No description available.

No / Three-dimensional (3D) representations of com- plex geometric shapes, especially when they are recon- structed from magnetic resonance imaging (MRI) and com- puted tomography (CT) data, often result in large polygon meshes which require substantial storage for their handling, and normally have only one fixed level of detail (LOD). This can often be an obstacle for efficient data exchange and interactive work with such objects. We propose to re- place such large polygon meshes with a relatively small set of coefficients of the patchwise partial differential equation (PDE) function representation. With this model, the approx- imations of the original shapes can be rendered with any desired resolution at interactive rates. Our approach can di- rectly work with any common 3D reconstruction pipeline, which we demonstrate by applying it to a large reconstructed medical data set with irregular geometry.

Medical practitioners rely heavily on visualization of medical imaging to get a better understanding of the patient's anatomy. Most cancer treatment and surgery today are performed using medical imaging. Medical imaging is therefore of great importance to the medical industry. Medical imaging continues to depend heavily on a series of 2D scans, resulting in a series of 2D photographs being displayed using light boxes and/or computer monitors. Today, these 2D images are increasingly combined into 3D solid models using software. These 3D models can be used for improved visualization and understanding of the problem at hand, including fabricating physical 3D models using additive manufacturing technologies. Generating precise 3D solid models automatically from 2D scans is non-trivial. Geometric and/or topologic errors are common, and often costly manual editing is required to produce 3D solid models that sufficiently reflect the actual underlying human geometry. These errors arise from the ambiguity of converting from 2D data to 3D data, and also from inherent limitations of the .STL fileformat used in additive manufacturing. This thesis proposes a new, robust method for automatically generating 3D models from 2D scanned data (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)), where the resulting 3D solid models are specifically generated for use with additive manufacturing. This new method does not rely on complicated procedures such as contour evolution and geometric spline generation, but uses volume reconstruction instead. The advantage of this approach is that the original scan data values are kept intact longer, so that the resulting surface is more accurate. This new method is demonstrated using medical CT data of the human nasal airway system, resulting in physical 3D models fabricated via additive manufacturing. / Master of Science / Medical practitioners rely heavily on medical imaging to get a better understanding of the patient’s anatomy. Most cancer treatment and surgery today are performed using medical imaging. Medical imaging is therefore of great importance to the medical industry. Medical imaging continues to depend heavily on a series of 2D scans, resulting in a series of 2D photographs being displayed using light boxes and/or computer monitors. With additive manufacturing technologies (also known as 3D printing), it is now possible to fabricate real-size physical 3D models of the human anatomy. These physical models enable surgeons to practice ahead of time, using realistic true scale model, to increase the likelihood of a successful surgery. These physical models can potentially also be used to develop organ implants that are tailored specifically to each patient’s anatomy. Generating precise 3D solid models automatically from 2D scans is non-trivial. Automated processing often causes geometric and topological (logical) errors, while manual editing is frequently too labor intensisve and time consuming to be considered practical solution. This thesis proposes a new, robust method for automatically generating 3D models from 2D scanned data (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)), where the resulting 3D solid models are specifically generated for use with additive manufacturing. The advantage of this proposed method is that the resulting fabricated surfaces are more accurate.

This dissertation proposes an approach for 3D micro-scale shape reconstruction using photometric stereo (PS) with surface normal integration (SNI). Based on the proposed approach, a portable cost-effective stationary system is developed to capture 3D shapes in the order of micrometer scale. The PS with SNI technique is adopted to reconstruct 3D microtopology since this technique is highlighted for its capability to reproduce fine surface details at pixel resolution. Furthermore, since the primary hardware components are merely a camera and several typical LEDs, the system based on PS with SNI can be made portable at low cost. The principal contributions are three folds. First, a PS method based on dichromatic reflectance model (DRM) using color input images is proposed to generalize PS applicable to a wider range of surfaces with non-Lambertian reflectances. The proposed method not only estimates surface orientations from diffuse reflection but also exploits information from specularities owing to the proposed diffuse-specular separation algorithm. Using the proposed PS method, material-dependent features can be simultaneously extracted in addition to surface orientations, which offers much richer information in understanding the 3D scene and poses more potential functionalities, such as specular removal, intrinsic image decomposition, digital relighting, material-based segmentation, material transfer and material classification. The second contribution is the development of an SNI method dealing with perspective distortion. The proposed SNI is performed on the image plane instead of on the target surface as did by orthographic SNI owing to the newly derived representation of surface normals. The motivation behind the representation is from the observation that spatially uniform image points are simpler for integration than the non-uniform distribution of surface points under perspective projection. The new representation is then manipulated to the so-called log gradient space in analogy to the gradient space in orthographic SNI. With this analogy, the proposed method can inherit most past algorithms developed for orthographic SNI. By applying the proposed SNI, perspective distortion can be efficiently tackled with for smooth surfaces. In addition, the method is PS-independent, which can keep the image irradiance equation in a simple form during PS. The third contribution is the design and calibration of a 3D micro-scale shape reconstruction system using the derived PS and SNI methods. This system is originally designed for on-site measurement of pavement microtexture, while its applicability can be generalized to a wider range of surfaces. Optimal illumination was investigated in theory and through numerical simulations. Five different calibrations regarding various aspects of the system were either newly proposed or modified from existing methods. The performances of these calibrations were individually evaluated. Efficacy of the developed system was finally demonstrated through comprehensive comparative studies with existing systems. Its capability for on-site measurement was also confirmed. / Ph. D. / Shapes in our world are three-dimensional (3D). How to measure and digitize shapes in 3D into computer understandable virtual models using cameras is called 3D shape reconstruction in the field of computer vision. This dissertation concerns the problem of 3D shape reconstruction, while concentrates on recovering shapes at micro-meter scale, referred to as 3D micro-scale shape reconstruction. Quantifying 3D shapes at micro-scale is significant for both industry and academia. In industry, quantification of 3D shapes at micro-meter scale can be employed in precision parts manufacturing, industrial quality control and rapid prototyping, whilst in academia, even finer resolution may be required to study the microtopography of a surface, such as for the purpose of investigating the nature of friction between surfaces. In this dissertation, a systematic solution is given for 3D micro-scale shape reconstruction using techniques called photometric stereo (PS) and surface normal integration (SNI) sequentially. PS estimates surface normals for each pixel-corresponding surface patch using images captured under various illumination directions from a fixed viewpoint. These surface normals are then integrated to reconstruct the surface in 3D via SNI. Based on these general principles, a prototype system was developed. The hardware of the system is simple, mainly contains a color digital single-lens reflex (DSLR) camera with a macro lens, multiple LEDs, a control circuit and a cover. During operation, the LEDs are sequentially turned on and create different illuminations upon the surface of concern. The DSLR camera simultaneously captures images with one LED lit at a time. Having these images for the target surface under various illuminations, the 3D surface at micro-scale is reconstructed through post-processing by PS with SNI algorithms. Three principal contributions are presented in this dissertation. First, a PS algorithm using color images is demonstrated to improve the shape reconstruction accuracy and its applicability for a wider range of surfaces with different reflectance properties. The proposed PS algorithm can also estimate material-dependent properties of the surface, making potential applications, such as material classification and inference, feasible. The second contribution is to improve the SNI algorithm to deal with the camera’s perspective distortion. Experimental results suggested that the algorithm has been successful in dealing with the distortion for smooth surfaces. The design and calibration of the prototype system are presented as the third contribution. The system can achieve high data acquisition rate due to its area scanning nature, dense measurements at micro-scale due to the PS with SNI approach, and low-cost due to the simple hardware configurations. Efficacy of the system was demonstrated through comprehensive comparative studies with existing systems. Its capability for on-site measurement was also proven.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第25421号 / 情博第859号 / 新制||情||144(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 西野 恒, 教授 西田 眞也, 教授 河原 達也 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM