The ability to reconstruct 3D scenes of environments is of great interest in a number of fields such as autonomous driving, surveillance, and virtual reality. However, traditional methods often rely on multiple cameras or sensor-based depth measurements to accurately reconstruct 3D scenes. In this thesis we propose an alternative, deep learning-based approach to 3D scene reconstruction for objects of interest, using nothing but single RGB images. We evaluate our approach using the Deep Object Pose Estimation (DOPE) neural network for object detection and pose estimation, and the NVIDIA Deep learning Dataset Synthesizer for synthetic data generation. Using two unique objects, our results indicate that it is possible to reconstruct 3D scenes from single RGB images within a few centimeters of error margin.

京都大学 / 新制・課程博士 / 博士(医学) / 甲第24477号 / 医博第4919号 / 新制||医||1062(附属図書館) / 京都大学大学院医学研究科医学専攻 / (主査)教授 中本 裕士, 教授 波多野 悦朗, 教授 万代 昌紀 / 学位規則第4条第1項該当 / Doctor of Medical Science / Kyoto University / DFAM

3D modelling is a technology in massive demand now and can potentially become a key factor for enabling subsequent technological evolutions such as metaverses, digital twins, and virtual reality. Current 3D modellings include high-precision 3D human body modelling and rapid modelling through single or multiple monocular photos. However, some problems persist in both modellings. The modelling based on high-precision equipment has low practicability, few applicable scenarios, and high cost. Modelling through monocular photos, on the other hand, has low accuracy and is sensitive to noisy data. And both modellings generate static 3D models. Therefore, to realize the model's dynamic effect in various fields while retaining fast modelling, we propose a system that recovers a 3D model from a single photo to fuse skeleton animation extracted from videos, for a realization of the Digital Twin (DT). DT is defined as "digital replications of living as well as non-living entities that enable data to be seamlessly transmitted between the physical and virtual worlds". Rigging is setting up the skeleton-based animation to combine the 3D model and skeleton animation. Traditional rigging method is time-consuming and non-reusable, since rigging is often done manually or semi-automatically. In this thesis, we propose an automatic rigging method to achieve a loose coupling fusion of one-to-many or many-to-one 3D models and skeletal animations. Our rigging method is fast and efficient, and only needs a single photo as input.

As more data sources have become abundantly available, an increased interest in 3D reconstruction has emerged in the image processing academic community. Applications for 3D reconstruction of urban and residential buildings consist of urban planning, network planning for mobile communication, tourism information systems, spatial analysis of air pollution and noise nuisance, microclimate investigations, and Geographical Information Systems (GISs). Previous, classical, 3D reconstruction algorithms solely utilized aerial photography. With the advent of LIDAR systems, current algorithms explore using captured LIDAR data as an additional feasible source of information for 3D reconstruction. Preprocessing techniques are proposed for the development of an autonomous 3D Reconstruction algorithm. The algorithm is designed for autonomously deriving three dimensional models of urban and residential buildings from raw LIDAR data. First, a greedy insertion triangulation algorithm, modified with a proposed noise filtering technique, triangulates the raw LIDAR data. The normal vectors of those triangles are then passed to an unsupervised clustering algorithm – Fuzzy Simplified Adaptive Resonance Theory (Fuzzy SART). Fuzzy SART returns a rough grouping of coplanar triangles. A proposed multiple regression algorithm then further refines the coplanar grouping by further removing outliers and deriving an improved planar segmentation of the raw LIDAR data. Finally, further refinement is achieved by calculating the intersection of the best fit roof planes and moving nearby points close to that intersection to exist at the intersection, resulting in straight roof ridges. The end result of the aforementioned techniques culminates in a well defined model approximating the considered building depicted by the LIDAR data.

Humans perceive the world in three dimensions, but many sensing capabilities only display two-dimensional information to users by way of images. In this work we develop two novel reconstruction techniques utilizing synthetic aperture radar (SAR) data in three dimensions given sparse amounts of available data. We additionally leverage a hybrid joint-sparsity and sparsity approach to remove a-priori influences on the environment and instead explore general imaging properties in our reconstructions. We evaluate the required sampling rates for our techniques and a thorough analysis of the accuracy of our methods. The results presented in this thesis suggest a solution to sparse three-dimensional object reconstruction that effectively uses a substantially less amount of phase history data (PHD) while still extracting critical features off an object of interest.

No description available.

No / Three-dimensional (3D) representations of com- plex geometric shapes, especially when they are recon- structed from magnetic resonance imaging (MRI) and com- puted tomography (CT) data, often result in large polygon meshes which require substantial storage for their handling, and normally have only one fixed level of detail (LOD). This can often be an obstacle for efficient data exchange and interactive work with such objects. We propose to re- place such large polygon meshes with a relatively small set of coefficients of the patchwise partial differential equation (PDE) function representation. With this model, the approx- imations of the original shapes can be rendered with any desired resolution at interactive rates. Our approach can di- rectly work with any common 3D reconstruction pipeline, which we demonstrate by applying it to a large reconstructed medical data set with irregular geometry.

This dissertation proposes an approach for 3D micro-scale shape reconstruction using photometric stereo (PS) with surface normal integration (SNI). Based on the proposed approach, a portable cost-effective stationary system is developed to capture 3D shapes in the order of micrometer scale. The PS with SNI technique is adopted to reconstruct 3D microtopology since this technique is highlighted for its capability to reproduce fine surface details at pixel resolution. Furthermore, since the primary hardware components are merely a camera and several typical LEDs, the system based on PS with SNI can be made portable at low cost. The principal contributions are three folds. First, a PS method based on dichromatic reflectance model (DRM) using color input images is proposed to generalize PS applicable to a wider range of surfaces with non-Lambertian reflectances. The proposed method not only estimates surface orientations from diffuse reflection but also exploits information from specularities owing to the proposed diffuse-specular separation algorithm. Using the proposed PS method, material-dependent features can be simultaneously extracted in addition to surface orientations, which offers much richer information in understanding the 3D scene and poses more potential functionalities, such as specular removal, intrinsic image decomposition, digital relighting, material-based segmentation, material transfer and material classification. The second contribution is the development of an SNI method dealing with perspective distortion. The proposed SNI is performed on the image plane instead of on the target surface as did by orthographic SNI owing to the newly derived representation of surface normals. The motivation behind the representation is from the observation that spatially uniform image points are simpler for integration than the non-uniform distribution of surface points under perspective projection. The new representation is then manipulated to the so-called log gradient space in analogy to the gradient space in orthographic SNI. With this analogy, the proposed method can inherit most past algorithms developed for orthographic SNI. By applying the proposed SNI, perspective distortion can be efficiently tackled with for smooth surfaces. In addition, the method is PS-independent, which can keep the image irradiance equation in a simple form during PS. The third contribution is the design and calibration of a 3D micro-scale shape reconstruction system using the derived PS and SNI methods. This system is originally designed for on-site measurement of pavement microtexture, while its applicability can be generalized to a wider range of surfaces. Optimal illumination was investigated in theory and through numerical simulations. Five different calibrations regarding various aspects of the system were either newly proposed or modified from existing methods. The performances of these calibrations were individually evaluated. Efficacy of the developed system was finally demonstrated through comprehensive comparative studies with existing systems. Its capability for on-site measurement was also confirmed. / Ph. D.

Medical practitioners rely heavily on visualization of medical imaging to get a better understanding of the patient's anatomy. Most cancer treatment and surgery today are performed using medical imaging. Medical imaging is therefore of great importance to the medical industry. Medical imaging continues to depend heavily on a series of 2D scans, resulting in a series of 2D photographs being displayed using light boxes and/or computer monitors. Today, these 2D images are increasingly combined into 3D solid models using software. These 3D models can be used for improved visualization and understanding of the problem at hand, including fabricating physical 3D models using additive manufacturing technologies. Generating precise 3D solid models automatically from 2D scans is non-trivial. Geometric and/or topologic errors are common, and often costly manual editing is required to produce 3D solid models that sufficiently reflect the actual underlying human geometry. These errors arise from the ambiguity of converting from 2D data to 3D data, and also from inherent limitations of the .STL fileformat used in additive manufacturing. This thesis proposes a new, robust method for automatically generating 3D models from 2D scanned data (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)), where the resulting 3D solid models are specifically generated for use with additive manufacturing. This new method does not rely on complicated procedures such as contour evolution and geometric spline generation, but uses volume reconstruction instead. The advantage of this approach is that the original scan data values are kept intact longer, so that the resulting surface is more accurate. This new method is demonstrated using medical CT data of the human nasal airway system, resulting in physical 3D models fabricated via additive manufacturing. / Master of Science

The main purpose of this work is to investigate the behavior of the recently released by Microsoft company the Kinect sensor, which contains the properties that go beyond ordinary cameras. Normally, in order to create a 3D reconstruction of the scene two cameras are required. Whereas, the Kinect device, due to the properties of the Infrared projector and sensor allows to create the same type of the reconstruction using only one device. However, the depth images, which are generated by the Infrared laser projector and monochrome sensor in Kinect can contain undefined values. Therefore, in addition to other investigations this project contains an idea how to improve the quality of the depth images. However, the base aim of this work is to perform a reconstruction of the scene based on the color images using pair of Kinects which will be compared with the results generated by using depth information from one Kinect. In addition, the report contains the information how to check that all the performed calculations were done correctly. All  the algorithms which were used in the project as well as the achieved results will be described and discussed in the separate chapters in the current report.