In a roofing project, acquiring the underlying as-built 3D geometry and visualizing the roof structure is needed in different phases of the project life-cycle. Architectural drawings, building information model (BIM) files, or aerial photogrammetry are used to estimate the roofing area in the bidding process. However, as a roof structure is never built to the exact drawing dimensions, as-built dimensions of boundaries of every roof plane have to be obtained several times during the course of its build. There are a number of surveying methods that can be used for this purpose: tape measuring, total station surveying, aerial photogrammetry, and laser scanning. However, obtaining measurements using these methods could be costly in terms of equipment, labor, and/or worker exposure to safety hazards. Aiming to address this limitation and provide roofing practitioners with an alternative roof surveying and visualization method that is simple to use, automated, inexpensive, and safe, a close-range videogrammetric roof 3D reconstruction framework is presented in this research. When using this method, a roofing contractor will simply collect stereo video streams of a target roof. The captured data is processed to generate a 3D wire-diagram for every roof plane. In this process, distinctive visual features of the scene (e.g., 2D points and lines) are first automatically detected and matched between video frames. Matched features and the camera calibration information are used to compute an initial estimation of the 3D structure. Then, a hybrid bundle adjustment algorithm is used to refine the result and acquire the geometry that has the maximum likelihood. Afterwards, different roof planes are found and a measurable 3D wire-diagram is generated for each plane.

This thesis presents a survey of techniques to obtain the depth component from two­-dimensional (2D) images. Two common techniques - stereo and shape from shading are examined here. Their performance is compared with an emphasis on noting the fundamental limitations of each technique. An argument is presented which suggests an adjustment of the paradigm with which stereo and shape from shading have been treated in three-dimensional vision. The theoretical development of the stereo and lighting models is followed by experiments illustrating use of these models for a variety of objects in a scene. A comparison of the results provides a motivation for combining them in a particular way. This combination is developed, and its application is examined. Using the model that is consistent for both shape and lighting, significant improvement over either stereo or lighting models alone is shown. / Thesis / Master of Engineering (ME)

No description available.

Learning-based multi-view stereo (MVS) has shown promising results in the domain of general 3D reconstruction. However, no work before this thesis has applied learning-based MVS to urban 3D reconstruction from satellite images. In this thesis, learning-based MVS is used to infer depth maps from satellite images. Models are trained on both synthetic and real satellite images from Las Vegas with ground truth data from a high-resolution aerial-based 3D model. This thesis also evaluates different methods for reconstructing digital surface models (DSM) and compares them to existing satellite-based 3D models at Maxar Technologies. The DSMs are created by either post-processing point clouds obtained from predicted depth maps or by an end-to-end approach where the depth map for an orthographic satellite image is predicted.  This thesis concludes that learning-based MVS can be used to predict accurate depth maps. Models trained on synthetic data yielded relatively good results, but not nearly as good as for models trained on real satellite images. The trained models also generalize relatively well to cities not present in training. This thesis also concludes that the reconstructed DSMs achieve better quantitative results than the existing 3D model in Las Vegas and similar results for the test sets from other cities. Compared to ground truth, the best-performing method achieved an L1 and L2 error of 14 % and 29 % lower than Maxar's current 3D model, respectively. The method that uses a point cloud as an intermediate step achieves better quantitative results compared to the end-to-end system. Very promising qualitative results are achieved with the proposed methods, especially when utilizing an end-to-end approach.

Visual perception has been a significant subject matter of robotics research for decades but has accelerated in recent years as both technology and community are more prepared to take on new challenges with autonomous systems. In this thesis, a framework for 3D reconstruction using a stereo camera for the purpose of obstacle detection and mapping is presented. In this application, a UAV works collaboratively with a UGV to provide high level information of the environment by using a downward facing stereo camera. The approach uses frame to frame SURF feature matching to detect candidate points within the camera image. These feature points are projected into a sparse cloud of 3D points using stereophotogrammetry for ICP registration to estimate the rigid transformation between frames. The RTK-GPS constrained pose estimate from the UAV is fused with the feature matched estimate to align the reconstruction and eliminate drift. The reconstruction was tested on both simulated and real data. The results indicate that this approach improves frame to frame registration and produces a well aligned reconstruction for a single pass compared to using the raw UAV position estimate alone. However, multi-pass registration errors occur on the order of about 0.6 meters between parallel passes, and approximately 2 degrees of local rotation error when compared to a reconstruction produced with Agisoft Metashape. However, the proposed system performed at an average frame rate of about 1.3 Hz compared to Agisoft at 0.03 Hz. Overall, the system improved obstacle registration and can perform online within existing ROS frameworks. / Master of Science / Visual perception has been a significant subject matter of robotics research for decades but has accelerated in recent years as both technology and community are more prepared to take on new challenges with autonomous systems. In this thesis, a framework for 3D reconstruction using cameras for the purpose of obstacle detection and mapping is presented. In this application, a UAV works collaboratively with a UGV to provide high level information of the environment by using a downward facing stereo camera. The approach uses features extracted from camera images to detect candidate points to be aligned. These feature points are projected into a sparse cloud of 3D points using stereo triangulation techniques. The 3D points are aligned using an iterative solver to estimate the translation and rotation between frames. The RTK (Real Time Kinematic) GPS constrained position and orientation estimate from the UAV is combined with the feature matched estimate to align the reconstruction and eliminate accumulated errors. The reconstruction was tested on both simulated and real data. The results indicate that this approach improves frame to frame registration and produces a well aligned reconstruction for a single pass compared to using the raw UAV position estimate alone. However, multi-pass registration errors occur on the order of about 0.6 meters between parallel passes that overlap, and approximately 2 degrees of local rotation error when compared to a reconstruction produced with the commercial product, Agisoft. However, the proposed system performed at an average frame rate of about 1.3 Hz compared to Agisoft at 0.03 Hz. Overall, the system improved obstacle registration and can perform online within existing Robot Operating System frameworks.

L'angiographie coronarienne RX (ou coronarographie) reste la modalité de référence permettant de déterminer avec précision le degré et le nombre de sténoses coronariennes ainsi que le nombre de vaisseaux atteints. L'objectif de ce travail de thèse de doctorat concerne l'amélioration de la reconstruction 3D des coronaires afin d'améliorer le diagnostic, ainsi que la sécurité et la précision des interventions minimalement invasives. Dans un premier temps, une contribution majeure vise à améliorer l'étape de calibration du système d'imagerie rotationnelle R-X. Premièrement, nous avons donc proposé un nouvel algorithme de calibrage basé objet inspiré de la méthode de Xu et al, dédié au système robotisé Artis Zeego. Deuxièmement, nous transposons les géométries de projection dans le système de coordonnées du C-arm. Nous avons proposé des modèles de mouvement des géométries de projection en considérant objectivement et systématiquement tous les facteurs possibles. Ces modèles de mouvement permettent de simplifier la procédure de calibration en routine clinique. The résultats expérimentaux indiquent que les modèles de mouvement proposés ont une précision acceptable afin estimer les paramètres d'acquisition. Dans un second temps, une contribution majeure vise à proposer une nouvelle méthode de reconstruction des coronaires par compensation de mouvement. Les étapes de reconstruction sont: l'opérateur de projection, la segmentation des projections acquises, le recalage iconique, la reconstruction initiale et la reconstruction avec compensation de mouvement. Nous avons opté pour l'opérateur de projection Distance Driven simplifié proposé par Oukili. Nous avons adopté comme mesure de similarité l'information Mutuelle (MI) avec un terme de rigidité. Pour optimiser cette fonction de coût, nous avons utilisé la méthode Advanced Adaptive Stochastic Gradient Descent (ASGD). Nous avons adopté une reconstruction statistique itérative basée on MAP et l'a priori L0. Les résultats expérimentaux indiquent que la méthode proposée améliore visuellement la qualité de la reconstruction. Le contraste et les détails des reconstructions sont améliorés par compensation de mouvement. / The rotational angiography RX of the coronaries is a standard modality to determine the degree and the number of the coronaries stenosis. The objective of this dissertation aims at improving the 3D reconstruction of the coronary arteries, which can improve the diagnosis, the security and the precision of the minimal invasive interventions.For the first part, the major contribution is improving the calibration procedure of the rotational R-X imaging system. First, we propose a new calibration algorithm based on the classical helical phantom on the Artis-Zeego system. Second, we transfer the geometries to the C-arm coordinate system. Last, we propose the movement models of the projection geometries objectively and systematically at 3 representative work positions. The movement models simplify the clinical procedures. The experiment results indicate that the proposed movement models have an acceptable precision to estimate the acquisition parameters.For the second part work, the major contribution is proposing a new reconstruction method by motion compensation. The steps of the reconstruction method include: the forward projection, the segmentation of the acquired projection, registration, the initial and motion compensated reconstruction. We adopt the advanced Simplified Distance Driven projector to generate the forward projection. We use the mutual information (MI) and rigidity penalty (RP) to be the similarity measure. We adopt the advanced Adaptive Stochastic Gradient Descent (ASGD) to realize the optimization. The initial and the compensated reconstruction are based on the MAP iterative reconstruction. The experiment results indicate that the proposed method improves the quality of the 3D reconstruction. The contrast and the details of the coronary arteries are improved by the proposed motion compensation reconstruction method.

Image-based 3D reconstruction has been employed in industrial metrology for micro measurements and quality control purposes. However, generating a highly-detailed and reliable 3D reconstruction of non-collaborative surfaces (textureless, shiny, and transparent) is still an open issue. This thesis presents various methodologies to successfully generate a highly-detailed and reliable 3D reconstruction of non-collaborative objects using the proposed photometric stereo image acquisition system. The first proposed method employs geometric construction to integrate photogrammetry and photometric stereo in order to overcome each technique's limitations and to leverage each technique's strengths in order to reconstruct an accurate and high-resolution topography of non-collaborative surfaces. This method uses accurate photogrammetric 3D measurements to rectify the global shape deviation of photometric stereo meanwhile uses photometric stereo to recover the high detailed topography of the object. The second method combines the high spatial frequencies of photometric stereo depth map with the low frequencies of photogrammetric depth map in frequency domain to produce accurate low frequencies while retaining high frequencies. For the third approach, we utilize light directionality to improve texture quality by leveraging shade and shadow phenomena using the proposed image-capturing system that employs several light sources for highlighting roughness and microstructures on the surface. And finally, we present two methods that effectively orient images by leveraging the low-contrast textures highlighted on object surfaces (roughness and 3D microstructures) using proper lighting system. Various objects with different surface characteristics including textureless, reflective, and transparent are used to evaluate different proposed approaches. To assess the accuracy of each approach, a comprehensive comparison between reference data and generated 3D points is provided.

This thesis develops and describes methods for real-time tracking, segmentation and 3-dimensional (3D) model acquisition, in the context of developing games for stroke patients that are rehabilitating at home. Real-time tracking and reconstruction of a stroke patient's feet, hands and the control objects that they are touching can enable not only the graphical visualization of the virtual avatar in the rehabilitation games, but also permits measurement of the patient's performs. Depth or combined colour and depth imagery from a Kinect sensor is used as input data. The 3D signed distance function (SDF) is used as implicit shape representation, and a series of probabilistic graphical models are developed for the problem of model-based 3D tracking, simultaneous 3D tracking and reconstruction and 3D tracking of multiple objects with identical appearance. The work is based on the assumption that the observed imagery is generated jointly by the pose(s) and the shape(s). The depth of each pixel is randomly and independently sampled from the likelihood of the pose(s) and the shape(s). The pose(s) tracking and 3D shape reconstruction problems are then cast as the maximum likelihood (ML) or maximum a posterior (MAP) estimate of the pose(s) or 3D shape. This methodology first leads to a novel probabilistic model for tracking rigid 3D objects with only depth data. For a known 3D shape, optimization aims to find the optimal pose that back projects all object region pixels onto the zero level set of the 3D shape, thus effectively maximising the likelihood of the pose. The method is extended to consider colour information for more robust tracking in the presence of outliers and occlusions. Initialised with a coarse 3D model, the extended method is also able to simultaneously reconstruct and track an unknown 3D object in real time. Finally, the concept of `shape union' is introduced to solve the problem of tracking multiple 3D objects with identical appearance. This is formulated as the minimum value of all SDFs in camera coordinates, which (i) leads to a per-pixel soft membership weight for each object thus providing an elegant solution for the data association in multi-target tracking and (ii) it allows for probabilistic physical constraints that avoid collisions between objects to be naturally enforced. The thesis also explore the possibility of using implicit shape representation for online shape learning. We use the harmonics of 2D discrete cosine transform (DCT) to represent 2D shapes. High frequency harmonics are decoupled from low ones to represent the coarse information and the details of the 2D shape. A regression model is learnt online to model the relationship between the high and low frequency harmonics using Locally Weighted Projection Regression (LWPR). We have demonstrated that the learned regression model is able to detect occlusion and recover them to the complete shape.

Ryanodine receptors (RyR) are intracellular channels that are intricately involved in Ca2+ release. These channels large membrane proteins~2.26MDa in size. In this multi-goal project firstly we successfully studied the gating mechanics of the RyR1 in the presence of Mg2+. We used single particle reconstruction and image processing techniques to obtain the 3D structure of the RyR1 with Mg2+. The 3D structure in the presence of Mg2+ and an ATP analog is the closest representation of human physiological conditions. The open and closed state structures of RyR1 are known. However, the physiologically closed state has not been studied before. Understanding this structure will help in the understanding of protein interactions. Our second goal was the validation of this 3D structure under different levels of noise. Validation under different noise levels analyzed the problem of noise bias is present in the field of cryo-EM and single particle reconstruction in select cases.

Electron tomography allows visualization of objects in a form of reconstructed 3D virtual volumes with resolution power of electron microscopy. The thesis is focused primarily on biological applications of electron tomography applied on tilt series images acquired in transmission electron microscope at room temperature. Specifically, the interaction of tick-borne encephalitis virus with neural cells and 3D ultrastructure of the central electron-dense part of the flagellum 9 + 1 (Caryophyllaeides fennica) were investigated. Finally, electron tomography was combined and correlated with atomic force microscopy to allow repetitive examination of ultrathin sections on electron microscopy grids.