This thesis proposes a solution to the pose estimation problem for mobile devices in an outdoors environment. The proposed solution is intended for usage within an augmented reality application to visualize large objects such as buildings. As such, the system needs to provide both accurate and stable pose estimations with real-time requirements. The proposed solution combines inertial navigation for orientation estimation with a vision-based support component to reduce noise from the inertial orientation estimation. A GNSS-based component provides the system with an absolute reference of position. The orientation and position estimation were tested in two separate experiments. The orientation estimate was tested with the camera in a static position and orientation and was able to attain an estimate that is accurate and stable down to a few fractions of a degree. The position estimation was able to achieve centimeter-level stability during optimal conditions. Once the position had converged to a location, it was stable down to a couple of centimeters, which is sufficient for outdoors augmented reality applications.

Esta dissertação apresenta um novo método para cálculo da pose da cabeça em imagens monoculares. Este cálculo é estimado no sistema de coordenadas da câmera, comparando as posições das características faciais específicas com as de múltiplas instâncias do modelo da face em 3D. Dada uma imagem de uma face humana, o método localiza inicialmente as características faciais, como nariz, olhos e boca. Estas últimas são detectadas e localizadas através de um modelo ativo de forma para faces. O algoritmo foi treinado sobre um conjunto de dados com diferentes poses de cabeça. Para cada face, obtemos um conjunto de pontos característicos no espaço de imagem 2D. Esses pontos são usados como referências na comparação com os respectivos pontos principais das múltiplas instâncias do nosso modelo de face em 3D projetado no espaço da imagem. Para obter a profundidade de cada ponto, usamos as restrições impostas pelo modelo 3D da face por exemplo, os olhos tem uma determinada profundidade em relação ao nariz. A pose da cabeça é estimada, minimizando o erro de comparação entre os pontos localizados numa instância do modelo 3D da face e os localizados na imagem. Nossos resultados preliminares são encorajadores e indicam que a nossa abordagem produz resultados mais precisos que os métodos disponíveis na literatura. / This dissertation presents a new method to accurately compute the head pose in mono cular images. The head pose is estimated in the camera coordinate system, by comparing the positions of specific facial features with the positions of these facial features in multiple instances of a prior 3D face model. Given an image containing a face, our method initially locates some facial features, such as nose, eyes, and mouth; these features are detected and located using an Adaptive Shape Model for faces , this algorithm was trained using on a data set with a variety of head poses. For each face, we obtain a collection of feature locations (i.e. points) in the 2D image space. These 2D feature locations are then used as references in the comparison with the respective feature locations of multiple instances of our 3D face model, projected on the same 2D image space. To obtain the depth of every feature point, we use the 3D spatial constraints imposed by our face model (i.e. eyes are at a certain depth with respect to the nose, and so on). The head pose is estimated by minimizing the comparison error between the 3D feature locations of the face in the image and a given instance of the face model (i.e. a geometrical transformation of the face model in the 3D camera space). Our preliminary experimental results are encouraging, and indicate that our approach can provide more accurate results than comparable methods available in the literature.

abstract: This study investigated the ability to relate a test taker’s non-verbal cues during online assessments to probable cheating incidents. Specifically, this study focused on the role of time delay, head pose and affective state for detection of cheating incidences in a lab-based online testing session. The analysis of a test taker’s non-verbal cues indicated that time delay, the variation of a student’s head pose relative to the computer screen and confusion had significantly statistical relation to cheating behaviors. Additionally, time delay, head pose relative to the computer screen, confusion, and the interaction term of confusion and time delay were predictors in a support vector machine of cheating prediction with an average accuracy of 70.7%. The current algorithm could automatically flag suspicious student behavior for proctors in large scale online courses during remotely administered exams. / Dissertation/Thesis / Doctoral Dissertation Computer Science 2015

Esta dissertação apresenta um novo método para cálculo da pose da cabeça em imagens monoculares. Este cálculo é estimado no sistema de coordenadas da câmera, comparando as posições das características faciais específicas com as de múltiplas instâncias do modelo da face em 3D. Dada uma imagem de uma face humana, o método localiza inicialmente as características faciais, como nariz, olhos e boca. Estas últimas são detectadas e localizadas através de um modelo ativo de forma para faces. O algoritmo foi treinado sobre um conjunto de dados com diferentes poses de cabeça. Para cada face, obtemos um conjunto de pontos característicos no espaço de imagem 2D. Esses pontos são usados como referências na comparação com os respectivos pontos principais das múltiplas instâncias do nosso modelo de face em 3D projetado no espaço da imagem. Para obter a profundidade de cada ponto, usamos as restrições impostas pelo modelo 3D da face por exemplo, os olhos tem uma determinada profundidade em relação ao nariz. A pose da cabeça é estimada, minimizando o erro de comparação entre os pontos localizados numa instância do modelo 3D da face e os localizados na imagem. Nossos resultados preliminares são encorajadores e indicam que a nossa abordagem produz resultados mais precisos que os métodos disponíveis na literatura. / This dissertation presents a new method to accurately compute the head pose in mono cular images. The head pose is estimated in the camera coordinate system, by comparing the positions of specific facial features with the positions of these facial features in multiple instances of a prior 3D face model. Given an image containing a face, our method initially locates some facial features, such as nose, eyes, and mouth; these features are detected and located using an Adaptive Shape Model for faces , this algorithm was trained using on a data set with a variety of head poses. For each face, we obtain a collection of feature locations (i.e. points) in the 2D image space. These 2D feature locations are then used as references in the comparison with the respective feature locations of multiple instances of our 3D face model, projected on the same 2D image space. To obtain the depth of every feature point, we use the 3D spatial constraints imposed by our face model (i.e. eyes are at a certain depth with respect to the nose, and so on). The head pose is estimated by minimizing the comparison error between the 3D feature locations of the face in the image and a given instance of the face model (i.e. a geometrical transformation of the face model in the 3D camera space). Our preliminary experimental results are encouraging, and indicate that our approach can provide more accurate results than comparable methods available in the literature.

Esta dissertação apresenta um novo método para cálculo da pose da cabeça em imagens monoculares. Este cálculo é estimado no sistema de coordenadas da câmera, comparando as posições das características faciais específicas com as de múltiplas instâncias do modelo da face em 3D. Dada uma imagem de uma face humana, o método localiza inicialmente as características faciais, como nariz, olhos e boca. Estas últimas são detectadas e localizadas através de um modelo ativo de forma para faces. O algoritmo foi treinado sobre um conjunto de dados com diferentes poses de cabeça. Para cada face, obtemos um conjunto de pontos característicos no espaço de imagem 2D. Esses pontos são usados como referências na comparação com os respectivos pontos principais das múltiplas instâncias do nosso modelo de face em 3D projetado no espaço da imagem. Para obter a profundidade de cada ponto, usamos as restrições impostas pelo modelo 3D da face por exemplo, os olhos tem uma determinada profundidade em relação ao nariz. A pose da cabeça é estimada, minimizando o erro de comparação entre os pontos localizados numa instância do modelo 3D da face e os localizados na imagem. Nossos resultados preliminares são encorajadores e indicam que a nossa abordagem produz resultados mais precisos que os métodos disponíveis na literatura. / This dissertation presents a new method to accurately compute the head pose in mono cular images. The head pose is estimated in the camera coordinate system, by comparing the positions of specific facial features with the positions of these facial features in multiple instances of a prior 3D face model. Given an image containing a face, our method initially locates some facial features, such as nose, eyes, and mouth; these features are detected and located using an Adaptive Shape Model for faces , this algorithm was trained using on a data set with a variety of head poses. For each face, we obtain a collection of feature locations (i.e. points) in the 2D image space. These 2D feature locations are then used as references in the comparison with the respective feature locations of multiple instances of our 3D face model, projected on the same 2D image space. To obtain the depth of every feature point, we use the 3D spatial constraints imposed by our face model (i.e. eyes are at a certain depth with respect to the nose, and so on). The head pose is estimated by minimizing the comparison error between the 3D feature locations of the face in the image and a given instance of the face model (i.e. a geometrical transformation of the face model in the 3D camera space). Our preliminary experimental results are encouraging, and indicate that our approach can provide more accurate results than comparable methods available in the literature.

A mobile manipulator is a robotic arm mounted on a robotic mobile platform. In such a system, the degrees of freedom of the mobile platform are combined with that of the manipulator. As a result, the workspace of the manipulator is substantially extended. A mobile manipulator has two trajectories: the end-effector trajectory and the mobile platform trajectory. Typically, the mobile platform trajectory is not defined and is determined through inverse kinematics. But in some applications it is important to follow a specified mobile platform trajectory. The main focus of this work is to determine the inverse kinematics of a mobile manipulator to follow the specified end-effector and mobile platform trajectories, especially when both trajectories cannot be exactly followed simultaneously due to physical limitations. Two new control algorithms are developed to solve this problem. In the first control algorithm, three joint-dependent control variables (spherical coordinates D, α and β) are introduced to define the mobile platform trajectory in relation to the end-effector trajectory and vice versa. This allows direct control of the mobile platform motion relative to the end-effector. Singularity-robust and task-priority inverse kinematics with gradient projection method is used to find best possible least-square solutions for the dual-trajectory tracking while maximizing the whole system manipulability. MATLAB Simulated Planar Mobile Manipulation is used to test and optimize the proposed control system. The results demonstrate the effectiveness of the control system in following the two trajectories as much as possible while optimizing the whole system manipulability measure. The second new inverse kinematics algorithm is introduced when the mobile platform motion is restricted to stay on a specified virtual or physical track. The control scheme allows xii the mobile manipulator to follow the desired end-effector trajectory while keeping the mobile platform on a specified track. The mobile platform is moved along a track to position the arm at a pose that facilitates the end-effector task. The translation of the redundant mobile manipulator over the mobile platform track is determined by combining the mobility of the platform and the manipulation of the redundant arm in a single control system. The mobile platform is allowed to move forward and backward with different velocities along its track to enable the end-effector in following its trajectory. MATLAB simulated 5 DoF redundant planar mobile manipulator is used to implement and test the proposed control algorithm. The results demonstrate the effectiveness of the control system in adjusting the mobile platform translations along its track to allow the arm to follow its own trajectory with high manipulability. Both control algorithms are implemented on MATLAB simulated wheelchair mounted robotic arm system (WMRA-II). These control algorithms are also implemented on real the WMRA-II hardware. In order to facilitate mobile manipulation, a control motion scheme is proposed to detect and correct the mobile platform pose estimation error using computer vision algorithm. The Iterative Closest Point (ICP) algorithm is used to register two consecutive Microsoft Kinect camera views. Two local transformation matrices i. e., Encoder and ICP transformation matrices, are fused using Extended Kalman Filter (EKF) to filter the encoder pose estimation error. VICON motion analysis system is used to capture the ground truth of the mobile platform. Real time implementation results show significant improvement in platform pose estimation. A real time application involving obstacle avoidance is used to test the proposed updated motion control system.

The goal of this work is to examine the possibility of surface reconstruction from the images produced by the Tobii H5 eye tracker. It starts of by examining classic photometric stereo and its limitations under the illuminator configuration of the eye tracker. It then proceeds to investigate two alternative solutions: photometric stereo under the assumption that the albedo is known and a method that uses the images from the eye tracker as a guide to mold a reference model into the users face. In the second method the pose of the reference model is estimated by minimizing a photometric error under the assumption that the face is Lambertian, using particle swarm optimization. The position of the generic 3D model is then used in an attempt to mold its shape into the face of the user using shape-from-shading.

In this thesis it is examined whether the pose of an object can be determined by a system trained with a synthetic 3D model of said object. A number of variations of methods using P-channel representation are examined. Reference images are rendered from the 3D model, features, such as gradient orientation and color information are extracted and encoded into P-channels. The P-channel representation is then used to estimate an overlapping channel representation, using B1-spline functions, to estimate a density function for the feature set. Experiments were conducted with this representation as well as the raw P-channel representation in conjunction with a number of distance measures and estimation methods. It is shown that, with correct preprocessing and choice of parameters, the pose can be detected with some accuracy and, if not in real-time, fast enough to be useful in a tracker initialization scenario. It is also concluded that the success rate of the estimation depends heavily on the nature of the object.

The possibility to use real-time computer vision in video sequences gives many opportunities for a system to interact with the environment. Possible ways for interaction are e.g. augmented reality like in the MATRIS project where the purpose is to add new objects into the video sequence, or surveillance where the purpose is to find abnormal events. The increase of the speed of computers the last years has simplified this process and it is now possible to use at least some of the more advanced computer vision algorithms that are available. The computational speed of computers is however still a problem, for an efficient real-time system efficient code and methods are necessary. This thesis deals with both problems, one part is about efficient implementations using single instruction multiple data (SIMD) instructions and one part is about robust tracking. An efficient real-time system requires efficient implementations of the used computer vision methods. Efficient implementations requires knowledge about the CPU and the possibilities given. In this thesis, one method called SIMD is explained. SIMD is useful when the same operation is applied to multiple data which usually is the case in computer vision, the same operation is executed on each pixel. Following the position of a feature or object in a video sequence is called tracking. Tracking can be used for a number of applications. The application in this thesis is to use tracking for pose estimation. One way to do tracking is to cut out a small region around the feature, creating a patch and find the position on this patch in the other frames. To find the position, a measure of the difference between the patch and the image in a given position is used. This thesis thoroughly investigates the sum of absolute difference (SAD) error measure. The investigation involves different ways to improve the robustness and to decrease the average error. One method to estimate the average error, the covariance of the position error is proposed. An estimate of the average error is needed when different measurements are combined. Finally, a system for camera pose estimation is presented. The computer vision part of this system is based on the result in this thesis. This presentation contains also a discussion about the result of this system. / Report code: LIU-TEK-LIC-2007:5. The report code in the thesis is incorrect.

This thesis is concerned with topics related to dense mapping of large scale three-dimensional spaces. In particular, the motivating scenario of this work is one in which a mobile robot with limited computational resources explores an unknown environment using a depth-camera. To this end, low-level topics such as sensor noise, map representation, interpolation, bit-rates, compression are investigated, and their impacts on more complex tasks, such as feature detection and description, camera-tracking, and mapping are evaluated thoroughly. A central idea of this thesis is the use of truncated signed distance fields (TSDF) as a map representation and a comprehensive yet accessible treatise on this subject is the first major contribution of this dissertation. The TSDF is a voxel-based representation of 3D space that enables dense mapping with high surface quality and robustness to sensor noise, making it a good candidate for use in grasping, manipulation and collision avoidance scenarios. The second main contribution of this thesis deals with the way in which information can be efficiently encoded in TSDF maps. The redundant way in which voxels represent continuous surfaces and empty space is one of the main impediments to applying TSDF representations to large-scale mapping. This thesis proposes two algorithms for enabling large-scale 3D tracking and mapping: a fast on-the-fly compression method based on unsupervised learning, and a parallel algorithm for lifting a sparse scene-graph representation from the dense 3D map. The third major contribution of this work consists of thorough evaluations of the impacts of low-level choices on higher-level tasks. Examples of these are the relationships between gradient estimation methods and feature detector repeatability, voxel bit-rate, interpolation strategy and compression ratio on camera tracking performance. Each evaluation thus leads to a better understanding of the trade-offs involved, which translate to direct recommendations for future applications, depending on their particular resource constraints.