This master thesis is about providing the implementation of synthesis, detection of spiral symbols and estimating the pan/tilt angle and position by using camera calibration. The focus is however on the latter, the estimation of parameters of localization. Spiral symbols are used to be able to give an object an identity as well as to locate it. Due to the spiral symbol´s characteristic shape, we can use the generalized structure tensor (GST) algorithm which is particularly efficient to detect different members of the spiral family. Once we detect spirals, we know the position and identity parameters of the spirals within an apriori known geometric configuration (on a sheet of paper). In turn, this information can be used to estimate the 3D-position and orientation of the object on which spirals are attached using a camera calibration method.   This thesis provides an insight into how automatic detection of spirals attached on a sheet of paper, and from this, automatic deduction of position and pose parameters of the sheet, can be achieved by using a network camera. GST algorithm has an advantage of running the processes of detection of spirals efficiently w.r.t detection performance and computational resources because it uses a spiral image model well adapted to spiral spatial frequency characteristic. We report results on how detection is affected by zoom parameters of the network camera, as well as by the GST parameters; such as filter size. After all spirals centers are located and identified w.r.t. their twist/bending parameter, a flexible technique for camera calibration, proposed by Zhengyou Zhang implemented in Matlab within the present study, is performed. The performance of the position and pose estimation in 3D is reported. The main conclusion is, we have reasonable surface angle estimations for images which were taken by a WLAN network camera in different conditions such as different illumination and different distances.

Το πρόβλημα της εκτίμησης θέσης και του προσανατολισμού της κάμερας από τις γνωστές 3D συντεταγμένες n σημείων της σκηνής και των 2D προβολών τους στο επίπεδο της εικόνας, είναι γνωστό στην βιβλιογραφία ως "Perspective n Point(PnP)" πρόβλημα. Το πρόβλημα αυτό συναντάται σε πολλά σημαντικά επιστημονικά πεδία όπως αυτά της υπολογιστικής όρασης, της ρομποτικής, της αυτοματοποιημένης χαρτογραφίας, της επαυξημένης πραγματικότητας κ.α, και μπορεί να θεωρηθεί ως μια ειδική περίπτωση του προβλήματος βαθμονόμησης της κάμερας. Η ανάγκη για την ανάπτυξη ρωμαλέων και χαμηλής πολυπλοκότητας μεθόδων για την επίλυση του "PnP" προβλήματος σε πραγματικό χρόνο έχει αναδειχθεί από πολλούς ερευνητές τα τελευταία χρόνια. Στο πλαίσιο της προτεινόμενης διπλωματικής μελετήθηκαν σε βάθος οι πιο σημαντικές μέθοδοι που έχουν προταθεί στην διεθνή βιβλιογραφία μέχρι σήμερα. / The perspective camera pose estimation problem, given known 3D coordinates in the world coordinate system and their correspondent 2D image projections, is known as "Perspective n Point(PnP)" problem. It has many applications in Photogrammetry, Computer Vision, Robotics, Augmented Reality and can be considered as a special case of camera calibration problem. The need for development of robust and simultaneously low computational complexity real time solutions for the PnP problem is very strong as it has attracted much attention in the literature during the last few years. In this master thesis, most significant as well as state of the art techniques which provide solutions to camera pose estimation problem have been thoroughly studied.

Object pose estimation is an important problem in computer vision with applications in robotics, augmented reality and many other areas. An established strategy for object pose estimation consists of, firstly, finding correspondences between the image and the object’s reference frame, and, secondly, estimating the pose from outlier-free correspondences using Random Sample Consensus (RANSAC). The first step, namely finding correspondences, is difficult because object appearance varies depending on perspective, lighting and many other factors. Traditionally, correspondences have been established using handcrafted methods like sparse feature pipelines. In this thesis, we introduce a dense correspondence representation for objects, called object coordinates, which can be learned. By learning object coordinates, our pose estimation pipeline adapts to various aspects of the task at hand. It works well for diverse object types, from small objects to entire rooms, varying object attributes, like textured or texture-less objects, and different input modalities, like RGB-D or RGB images. The concept of object coordinates allows us to easily model and exploit uncertainty as part of the pipeline such that even repeating structures or areas with little texture can contribute to a good solution. Although we can train object coordinate predictors independent of the full pipeline and achieve good results, training the pipeline in an end-to-end fashion is desirable. It enables the object coordinate predictor to adapt its output to the specificities of following steps in the pose estimation pipeline. Unfortunately, the RANSAC component of the pipeline is non-differentiable which prohibits end-to-end training. Adopting techniques from reinforcement learning, we introduce Differentiable Sample Consensus (DSAC), a formulation of RANSAC which allows us to train the pose estimation pipeline in an end-to-end fashion by minimizing the expectation of the final pose error.

Flying animals such as bats, birds, and moths are actively studied by researchers wanting to better understand these animals’ behavior and flight characteristics. Towards this goal, multi-view videos of flying animals have been recorded both in lab- oratory conditions and natural habitats. The analysis of these videos has shifted over time from manual inspection by scientists to more automated and quantitative approaches based on computer vision algorithms. This thesis describes a study on the largely unexplored problem of 3D pose estimation of flying animals in multi-view video data. This problem has received little attention in the computer vision community where few flying animal datasets exist. Additionally, published solutions from researchers in the natural sciences have not taken full advantage of advancements in computer vision research. This thesis addresses this gap by proposing three different approaches for 3D pose estimation of flying animals in multi-view video datasets, which evolve from successful pose estimation paradigms used in computer vision. The first approach models the appearance of a flying animal with a synthetic 3D graphics model and then uses a Markov Random Field to model 3D pose estimation over time as a single optimization problem. The second approach builds on the success of Pictorial Structures models and further improves them for the case where only a sparse set of landmarks are annotated in training data. The proposed approach first discovers parts from regions of the training images that are not annotated. The discovered parts are then used to generate more accurate appearance likelihood terms which in turn produce more accurate landmark localizations. The third approach takes advantage of the success of deep learning models and adapts existing deep architectures to perform landmark localization. Both the second and third approaches perform 3D pose estimation by first obtaining accurate localization of key landmarks in individual views, and then using calibrated cameras and camera geometry to reconstruct the 3D position of key landmarks. This thesis shows that the proposed algorithms generate first-of-a-kind and leading results on real world datasets of bats and moths, respectively. Furthermore, a variety of resources are made freely available to the public to further strengthen the connection between research communities.

This thesis proposes a solution to the pose estimation problem for mobile devices in an outdoors environment. The proposed solution is intended for usage within an augmented reality application to visualize large objects such as buildings. As such, the system needs to provide both accurate and stable pose estimations with real-time requirements. The proposed solution combines inertial navigation for orientation estimation with a vision-based support component to reduce noise from the inertial orientation estimation. A GNSS-based component provides the system with an absolute reference of position. The orientation and position estimation were tested in two separate experiments. The orientation estimate was tested with the camera in a static position and orientation and was able to attain an estimate that is accurate and stable down to a few fractions of a degree. The position estimation was able to achieve centimeter-level stability during optimal conditions. Once the position had converged to a location, it was stable down to a couple of centimeters, which is sufficient for outdoors augmented reality applications.

A mobile manipulator is a robotic arm mounted on a robotic mobile platform. In such a system, the degrees of freedom of the mobile platform are combined with that of the manipulator. As a result, the workspace of the manipulator is substantially extended. A mobile manipulator has two trajectories: the end-effector trajectory and the mobile platform trajectory. Typically, the mobile platform trajectory is not defined and is determined through inverse kinematics. But in some applications it is important to follow a specified mobile platform trajectory. The main focus of this work is to determine the inverse kinematics of a mobile manipulator to follow the specified end-effector and mobile platform trajectories, especially when both trajectories cannot be exactly followed simultaneously due to physical limitations. Two new control algorithms are developed to solve this problem. In the first control algorithm, three joint-dependent control variables (spherical coordinates D, α and β) are introduced to define the mobile platform trajectory in relation to the end-effector trajectory and vice versa. This allows direct control of the mobile platform motion relative to the end-effector. Singularity-robust and task-priority inverse kinematics with gradient projection method is used to find best possible least-square solutions for the dual-trajectory tracking while maximizing the whole system manipulability. MATLAB Simulated Planar Mobile Manipulation is used to test and optimize the proposed control system. The results demonstrate the effectiveness of the control system in following the two trajectories as much as possible while optimizing the whole system manipulability measure. The second new inverse kinematics algorithm is introduced when the mobile platform motion is restricted to stay on a specified virtual or physical track. The control scheme allows xii the mobile manipulator to follow the desired end-effector trajectory while keeping the mobile platform on a specified track. The mobile platform is moved along a track to position the arm at a pose that facilitates the end-effector task. The translation of the redundant mobile manipulator over the mobile platform track is determined by combining the mobility of the platform and the manipulation of the redundant arm in a single control system. The mobile platform is allowed to move forward and backward with different velocities along its track to enable the end-effector in following its trajectory. MATLAB simulated 5 DoF redundant planar mobile manipulator is used to implement and test the proposed control algorithm. The results demonstrate the effectiveness of the control system in adjusting the mobile platform translations along its track to allow the arm to follow its own trajectory with high manipulability. Both control algorithms are implemented on MATLAB simulated wheelchair mounted robotic arm system (WMRA-II). These control algorithms are also implemented on real the WMRA-II hardware. In order to facilitate mobile manipulation, a control motion scheme is proposed to detect and correct the mobile platform pose estimation error using computer vision algorithm. The Iterative Closest Point (ICP) algorithm is used to register two consecutive Microsoft Kinect camera views. Two local transformation matrices i. e., Encoder and ICP transformation matrices, are fused using Extended Kalman Filter (EKF) to filter the encoder pose estimation error. VICON motion analysis system is used to capture the ground truth of the mobile platform. Real time implementation results show significant improvement in platform pose estimation. A real time application involving obstacle avoidance is used to test the proposed updated motion control system.

The possibility to use real-time computer vision in video sequences gives many opportunities for a system to interact with the environment. Possible ways for interaction are e.g. augmented reality like in the MATRIS project where the purpose is to add new objects into the video sequence, or surveillance where the purpose is to find abnormal events. The increase of the speed of computers the last years has simplified this process and it is now possible to use at least some of the more advanced computer vision algorithms that are available. The computational speed of computers is however still a problem, for an efficient real-time system efficient code and methods are necessary. This thesis deals with both problems, one part is about efficient implementations using single instruction multiple data (SIMD) instructions and one part is about robust tracking. An efficient real-time system requires efficient implementations of the used computer vision methods. Efficient implementations requires knowledge about the CPU and the possibilities given. In this thesis, one method called SIMD is explained. SIMD is useful when the same operation is applied to multiple data which usually is the case in computer vision, the same operation is executed on each pixel. Following the position of a feature or object in a video sequence is called tracking. Tracking can be used for a number of applications. The application in this thesis is to use tracking for pose estimation. One way to do tracking is to cut out a small region around the feature, creating a patch and find the position on this patch in the other frames. To find the position, a measure of the difference between the patch and the image in a given position is used. This thesis thoroughly investigates the sum of absolute difference (SAD) error measure. The investigation involves different ways to improve the robustness and to decrease the average error. One method to estimate the average error, the covariance of the position error is proposed. An estimate of the average error is needed when different measurements are combined. Finally, a system for camera pose estimation is presented. The computer vision part of this system is based on the result in this thesis. This presentation contains also a discussion about the result of this system. / Report code: LIU-TEK-LIC-2007:5. The report code in the thesis is incorrect.

This thesis is concerned with topics related to dense mapping of large scale three-dimensional spaces. In particular, the motivating scenario of this work is one in which a mobile robot with limited computational resources explores an unknown environment using a depth-camera. To this end, low-level topics such as sensor noise, map representation, interpolation, bit-rates, compression are investigated, and their impacts on more complex tasks, such as feature detection and description, camera-tracking, and mapping are evaluated thoroughly. A central idea of this thesis is the use of truncated signed distance fields (TSDF) as a map representation and a comprehensive yet accessible treatise on this subject is the first major contribution of this dissertation. The TSDF is a voxel-based representation of 3D space that enables dense mapping with high surface quality and robustness to sensor noise, making it a good candidate for use in grasping, manipulation and collision avoidance scenarios. The second main contribution of this thesis deals with the way in which information can be efficiently encoded in TSDF maps. The redundant way in which voxels represent continuous surfaces and empty space is one of the main impediments to applying TSDF representations to large-scale mapping. This thesis proposes two algorithms for enabling large-scale 3D tracking and mapping: a fast on-the-fly compression method based on unsupervised learning, and a parallel algorithm for lifting a sparse scene-graph representation from the dense 3D map. The third major contribution of this work consists of thorough evaluations of the impacts of low-level choices on higher-level tasks. Examples of these are the relationships between gradient estimation methods and feature detector repeatability, voxel bit-rate, interpolation strategy and compression ratio on camera tracking performance. Each evaluation thus leads to a better understanding of the trade-offs involved, which translate to direct recommendations for future applications, depending on their particular resource constraints.

This thesis proposes novel algorithms for using Gaussian processes for Discriminative pose estimation. We overcome the traditional limitations of Gaussian processes, their cubic training complexity and their uni-modal predictive distribution by assembling them in a mixture of experts formulation. Our First contribution shows that by creating a large number of Fixed size Gaussian process experts, we can build a model that is able to scale to large data sets and accurately learn the multi-modal and non- linear mapping between image features and the subject’s pose. We demonstrate that this model gives state of the art performance compared to other discriminative pose estimation techniques.We then extend the model to automatically learn the size and location of each expert. Gaussian processes are able to accurately model non-linear functional regression problems where the output is given as a function of the input. However, when an individual Gaussian process is trained on data which contains multi-modalities, or varying levels of ambiguity, the Gaussian process is unable to accurately model the data. We propose a novel algorithm for learning the size and location of each expert in our mixture of Gaussian processes model to ensure that the training data of each expert matches the assumptions of a Gaussian process. We show that this model is able to out perform our previous mixture of Gaussian processes model.Our final contribution is a dynamics framework for inferring a smooth sequence of pose estimates from a sequence of independent predictive distributions. Discriminative pose estimation infers the pose of each frame independently, leading to jittery tracking results. Our novel algorithm uses a model of human dynamics to infer a smooth path through a sequence of Gaussian mixture models as given by our mixture of Gaussian processes model. We show that our algorithm is able to smooth and correct some mis- takes made by the appearance model alone, and outperform a baseline linear dynamical system.

No description available.