Global position system (GPS) is widely used in land vehicles but suffers deterioration in its accuracy in urban canyons; mostly due to satellite signal blockage and signal multipath. To obtain accurate, reliable, and continuous positioning solutions, GPS is usually augmented with inertial sensors, including accelerometers and gyroscopes to monitor both translational and rotational motions of a moving vehicle. Due to space and cost requirements, micro-electro-mechanical-system (MEMS) inertial sensors, which are typically inexpensive are presently utilized in land vehicles for various reasons and can be used for integration with GPS for navigation purposes. Kalman filtering (KF) usually used to performs this integration. However, the complex error characteristics of these MEMS based sensors lead to divergence of the positioning solution. Furthermore, the residual GPS pseudorange correlated errors are always ignored, thus reducing the GPS overall positioning accuracy. This thesis targets enhancing the performance of integrated MEMS based INS/GPS navigation systems through exploring new non-linear modelling approaches that can deal with the non-linear and correlated parts of INS and GPS errors. The research approach in this thesis relies on reduced inertial sensor systems (RISS) incorporating single axis gyroscope, vehicle odometer, and accelerometers is considered for the integration with GPS in one of two schemes; either loosely-coupled where GPS position and velocity are used for the integration or tightly-coupled where GPS pseudorange and pseudorange rates are utilized. A new method based on parallel cascade identification (PCI) is developed in this research to enhance the performance of KF by modelling azimuth errors for the RISS/GPS loosely-coupled integration scheme. In addition, PCI is also utilized for the modelling of residual GPS pseudorange correlated errors. This thesis develops a method to augment a PCI – based model of GPS pseudorange correlated errors to a tightly-coupled KF. In order to take full advantage of the PCI based models, this thesis explores the Particle filter (PF) as a non-linear integration scheme that is capable of accommodating the arbitrary sensor characteristics, motion dynamics, and noise distributions. The performance of the proposed methods is examined through several road test experiments in land vehicles involving different types of inertial sensors and GPS receivers. / Thesis (Ph.D, Electrical & Computer Engineering) -- Queen's University, 2012-07-31 16:09:16.559

Indiana University-Purdue University Indianapolis (IUPUI) / Indoor tracking is a challenging problem. The level of accepted error is on a much smaller scale than that of its outdoor counterpart. While the global positioning system has become omnipresent, and a widely accepted outdoor tracking system it has limitations in indoor environments due to loss or degradation of signal. Many attempts have been made to address this challenge, but currently none have proven to be the de-facto standard. In this thesis, we introduce the concept of opportunistic tracking in which tracking takes place with whatever sensing infrastructure is present – static or mobile, within a given indoor environment. In this approach many of the challenges (e.g., high cost, infeasible infrastructure deployment, etc.) that prohibit usage of existing systems in typical application domains (e.g., asset tracking, emergency rescue) are eliminated. Challenges do still exist when it comes to provide an accurate positional estimate of an entities location in an indoor environment, namely: sensor classification, sensor selection, and multi-sensor data fusion. We propose an enhanced tracking framework that through the infusion of QoS-based selection criteria of trust and reliability we can improve the overall accuracy of the tracking estimate. This improvement is predicated on the introduction of learning techniques to classify sensors that are dynamically discovered as part of this opportunistic tracking approach. This classification allows for sensors to be properly identified and evaluated based upon their specific behavioral characteristics through performance evaluation. This in-depth evaluation of sensors provides the basis for improving the sensor selection process. A side effect of obtaining this improved accuracy is the cost, found in the form of system runtime. This thesis provides a solution for this tradeoff between accuracy and cost through an optimization function that analyzes this tradeoff in an effort to find the optimal subset of sensors to fulfill the goal of tracking an object as it moves indoors. We demonstrate that through this improved sensor classification, selection, data fusion, and tradeoff optimization we can provide an improvement, in terms of accuracy, over other existing indoor tracking systems.

A significant progress in the evolution of the computer systems and their interconnection over the past 70 years has allowed replacing the frequently used centralized architectures with the highly distributed ones, formed by independent entities fulfilling specific functionalities as one user-intransparent unit. This has resulted in an intense scientic interest in distributed algorithms and their frequent implementation into real systems. Especially, distributed algorithms for multi-sensor data fusion, ensuring an enhanced QoS of executed applications, find a wide usage. This doctoral thesis addresses an optimization and an analysis of the distributed systems, namely the distributed consensus-based algorithms for an aggregate function estimation (primarily, my attention is focused on a mean estimation). The first section is concerned with a theoretical background of the distributed systems, their evolution, their architectures, and a comparison with the centralized systems (i.e. their advantages/disadvantages). The second chapter deals with multi-sensor data fusion, its application, the classification of the distributed estimation techniques, their mathematical modeling, and frequently quoted algorithms for distributed averaging (e.g. protocol Push-Sum, Metropolis-Hastings weights, Best Constant weights etc.). The practical part is focused on mechanisms for an optimization of the distributed systems, the proposal of novel algorithms and complements for the distributed systems, their analysis, and comparative studies in terms of such as the convergence rate, the estimation precision, the robustness, the applicability to real systems etc.

Les travaux présentés dans ce manuscrit concernent le domaine de la localisation et la reconstruction 3D par vision artificielle. Dans ce contexte, la trajectoire d’une caméra et la structure3D de la scène filmée sont initialement estimées par des algorithmes linéaires puis optimisées par un algorithme non-linéaire, l’ajustement de faisceaux. Cette thèse présente tout d’abord une technique de recherche de l’amplitude de déplacement (recherche linéaire), ou line search pour les algorithmes de minimisation itérative. La technique proposée est non itérative et peut être rapidement implantée dans un ajustement de faisceaux traditionnel. Cette technique appelée recherche linéaire algébrique globale (G-ALS), ainsi que sa variante à deux dimensions (Two way-ALS), accélèrent la convergence de l’algorithme d’ajustement de faisceaux. L’approximation de l’erreur de reprojection par une distance algébrique rend possible le calcul analytique d’une amplitude de déplacement efficace (ou de deux pour la variante Two way-ALS), par la résolution d’un polynôme de degré 3 (G-ALS) ou 5 (Two way-ALS). Nos expérimentations sur des données simulées et réelles montrent que cette amplitude, optimale en distance algébrique, est performante en distance euclidienne, et permet de réduire le temps de convergence des minimisations. Une difficulté des algorithmes de localisation en temps réel par la vision (SLAM monoculaire) est que la trajectoire estimée est souvent affectée par des dérives : dérives d’orientation, de position et d’échelle. Puisque ces algorithmes sont incrémentaux, les erreurs et approximations sont cumulées tout au long de la trajectoire, et une dérive se forme sur la localisation globale. De plus, un système de localisation par vision peut toujours être ébloui ou utilisé dans des conditions qui ne permettent plus temporairement de calculer la localisation du système. Pour résoudre ces problèmes, nous proposons d’utiliser un capteur supplémentaire mesurant les déplacements de la caméra. Le type de capteur utilisé varie suivant l’application ciblée (un odomètre pour la localisation d’un véhicule, une centrale inertielle légère ou un système de navigation à guidage inertiel pour localiser une personne). Notre approche consiste à intégrer ces informations complémentaires directement dans l’ajustement de faisceaux, en ajoutant un terme de contrainte pondéré dans la fonction de coût. Nous évaluons trois méthodes permettant de sélectionner dynamiquement le coefficient de pondération et montrons que ces méthodes peuvent être employées dans un SLAM multi-capteur temps réel, avec différents types de contrainte, sur l’orientation ou sur la norme du déplacement de la caméra. La méthode est applicable pour tout autre terme de moindres carrés. Les expérimentations menées sur des séquences vidéo réelles montrent que cette technique d’ajustement de faisceaux contraint réduit les dérives observées avec les algorithmes de vision classiques. Ils améliorent ainsi la précision de la localisation globale du système. / The works presented in this manuscript are in the field of computer vision, and tackle the problem of real-time vision based localization and 3D reconstruction. In this context, the trajectory of a camera and the 3D structure of the filmed scene are initially estimated by linear algorithms and then optimized by a nonlinear algorithm, bundle adjustment. The thesis first presents a new technique of line search, dedicated to the nonlinear minimization algorithms used in Structure-from-Motion. The proposed technique is not iterative and can be quickly installed in traditional bundle adjustment frameworks. This technique, called Global Algebraic Line Search (G-ALS), and its two-dimensional variant (Two way-ALS), accelerate the convergence of the bundle adjustment algorithm. The approximation of the reprojection error by an algebraic distance enables the analytical calculation of an effective displacement amplitude (or two amplitudes for the Two way-ALS variant) by solving a degree 3 (G-ALS) or 5 (Two way-ALS) polynomial. Our experiments, conducted on simulated and real data, show that this amplitude, which is optimal for the algebraic distance, is also efficient for the Euclidean distance and reduces the convergence time of minimizations. One difficulty of real-time tracking algorithms (monocular SLAM) is that the estimated trajectory is often affected by drifts : on the absolute orientation, position and scale. Since these algorithms are incremental, errors and approximations are accumulated throughout the trajectory and cause global drifts. In addition, a tracking vision system can always be dazzled or used under conditions which prevented temporarily to calculate the location of the system. To solve these problems, we propose to use an additional sensor measuring the displacement of the camera. The type of sensor used will vary depending on the targeted application (an odometer for a vehicle, a lightweight inertial navigation system for a person). We propose to integrate this additional information directly into an extended bundle adjustment, by adding a constraint term in the weighted cost function. We evaluate three methods (based on machine learning or regularization) that dynamically select the weight associated to the constraint and show that these methods can be used in a real time multi-sensor SLAM, and validate them with different types of constraint on the orientation or on the scale. Experiments conducted on real video sequences show that this technique of constrained bundle adjustment reduces the drifts observed with the classical vision algorithms and improves the global accuracy of the positioning system.

Fusion is a fundamental human process that occurs in some form at all levels of sense organs such as visual and sound information received from eyes and ears respectively, to the highest levels of decision making such as our brain fuses visual and sound information to make decisions. Multi-sensor data fusion is concerned with gaining information from multiple sensors by fusing across raw data, features or decisions. The traditional frameworks for multi-sensor data fusion only concern fusion at specific points in time. However, many real world situations change over time. When the multi-sensor system is used for situation awareness, it is useful not only to know the state or event of the situation at a point in time, but also more importantly, to understand the causalities of those states or events changing over time. / Hence, we proposed a multi-agent framework for temporal fusion, which emphasises the time dimension of the fusion process, that is, fusion of the multi-sensor data or events derived over a period of time. The proposed multi-agent framework has three major layers: hardware, agents, and users. There are three different fusion architectures: centralized, hierarchical, and distributed, for organising the group of agents. The temporal fusion process of the proposed framework is elaborated by using the information graph. Finally, the core of the proposed temporal fusion framework – Dynamic Time Warping (DTW) temporal fusion agent is described in detail. / Fusing multisensory data over a period of time is a challenging task, since the data to be fused consists of complex sequences that are multi–dimensional, multimodal, interacting, and time–varying in nature. Additionally, performing temporal fusion efficiently in real–time is another challenge due to the large amount of data to be fused. To address these issues, we proposed the DTW temporal fusion agent that includes four major modules: data pre-processing, DTW recogniser, class templates, and decision making. The DTW recogniser is extended in various ways to deal with the variability of multimodal sequences acquired from multiple heterogeneous sensors, the problems of unknown start and end points, multimodal sequences of the same class that hence has different lengths locally and/or globally, and the challenges of online temporal fusion. / We evaluate the performance of the proposed DTW temporal fusion agent on two real world datasets: 1) accelerometer data acquired from performing two hand gestures, and 2) a benchmark dataset acquired from carrying a mobile device and performing pre-defined user scenarios. Performance results of the DTW based system are compared with those of a Hidden Markov Model (HMM) based system. The experimental results from both datasets demonstrate that the proposed DTW temporal fusion agent outperforms HMM based systems, and has the capability to perform online temporal fusion efficiently and accurately in real–time.

abstract: Modern intelligent transportation systems (ITS) make driving more efficient, easier, and safer. Knowledge of real-time traffic conditions is a critical input for operating ITS. Real-time freeway traffic state estimation approaches have been used to quantify traffic conditions given limited amount of data collected by traffic sensors. Currently, almost all real-time estimation methods have been developed for estimating laterally aggregated traffic conditions in a roadway segment using link-based models which assume homogeneous conditions across multiple lanes. However, with new advances and applications of ITS, knowledge of lane-based traffic conditions is becoming important, where the traffic condition differences among lanes are recognized. In addition, most of the current real-time freeway traffic estimators consider only data from loop detectors. This dissertation develops a bi-level data fusion approach using heterogeneous multi-sensor measurements to estimate real-time lane-based freeway traffic conditions, which integrates a link-level model-based estimator and a lane-level data-driven estimator. Macroscopic traffic flow models describe the evolution of aggregated traffic characteristics over time and space, which are required by model-based traffic estimation approaches. Since current first-order Lagrangian macroscopic traffic flow model has some unrealistic implicit assumptions (e.g., infinite acceleration), a second-order Lagrangian macroscopic traffic flow model has been developed by incorporating drivers’ anticipation and reaction delay. A multi-sensor extended Kalman filter (MEKF) algorithm has been developed to combine heterogeneous measurements from multiple sources. A MEKF-based traffic estimator, explicitly using the developed second-order traffic flow model and measurements from loop detectors as well as GPS trajectories for given fractions of vehicles, has been proposed which gives real-time link-level traffic estimates in the bi-level estimation system. The lane-level estimation in the bi-level data fusion system uses the link-level estimates as priors and adopts a data-driven approach to obtain lane-based estimates, where now heterogeneous multi-sensor measurements are combined using parallel spatial-temporal filters. Experimental analysis shows that the second-order model can more realistically reproduce real world traffic flow patterns (e.g., stop-and-go waves). The MEKF-based link-level estimator exhibits more accurate results than the estimator that uses only a single data source. Evaluation of the lane-level estimator demonstrates that the proposed new bi-level multi-sensor data fusion system can provide very good estimates of real-time lane-based traffic conditions. / Dissertation/Thesis / Doctoral Dissertation Industrial Engineering 2015

Cette thèse aborde le problèeme du suivi et de la classification de plusieurs objets simultanément.Il est montré dans la thèese que les fonctions de croyance permettent d'améliorer les résultatsfournis par des méthodes classiques à base d'approches Bayésiennes. En particulier, une précédenteapproche développée dans le cas d'un seul objet est étendue au cas de plusieurs objets. Il est montréque dans toutes les approches multi-objets, la phase d'association entre observations et objetsconnus est fondamentale. Cette thèse propose également de nouvelles méthodes d'associationcrédales qui apparaissent plus robustes que celles trouvées dans la littérature. Enfin, est abordée laquestion de la classification multi-capteurs qui nécessite une seconde phase d'association. Dans cedernier cas, deux architectures de fusion des données capteurs sont proposées, une dite centraliséeet une autre dite distribuée. De nombreuses comparaisons illustrent l'intérêt de ces travaux, queles classes des objets soient constantes ou variantes dans le temps. / This thesis deals with multi-objet tracking and classification problem. It was shown that belieffunctions allow the results of classical Bayesian methods to be improved. In particular, a recentapproach dedicated to a single object classification which is extended to multi-object framework. Itwas shown that detected observations to known objects assignment is a fundamental issue in multiobjecttracking and classification solutions. New assignment solutions based on belief functionsare proposed in this thesis, they are shown to be more robust than the other credal solutions fromrecent literature. Finally, the issue of multi-sensor classification that requires a second phase ofassignment is addressed. In the latter case, two different multi-sensor architectures are proposed, aso-called centralized one and another said distributed. Many comparisons illustrate the importanceof this work, in both situations of constant and changing objects classes.

L’objectif de cette thèse est de développer des méthodes permettant la cartographie d’un environnement tridimensionnel de grande dimension en combinant radar panoramique MMW et caméras optiques. Contrairement aux méthodes existantes de fusion de données multi-capteurs, telles que le SLAM, nous souhaitons réaliser un capteur de type RGB-D fournissant directement des mesures de profondeur enrichies par l’apparence (couleur, texture...). Après avoir modélisé géométriquement le système radar/caméra, nous proposons une méthode de calibrage originale utilisant des correspondances de points. Pour obtenir ces correspondances, des cibles permettant une mesure ponctuelle aussi bien par le radar que la caméra ont été conçues. L’approche proposée a été élaborée pour pouvoir être mise en oeuvre dans un environnement libre et par un opérateur non expert. Deuxièmement, une méthode de reconstruction de points tridimensionnels sur la base de correspondances de points radar et image a été développée. Nous montrons par une analyse théorique des incertitudes combinées des deux capteurs et par des résultats expérimentaux, que la méthode proposée est plus précise que la triangulation stéréoscopique classique pour des points éloignés comme on en trouve dans le cas de cartographie d’environnements extérieurs. Enfin, nous proposons une stratégie efficace de mise en correspondance automatique des données caméra et radar. Cette stratégie utilise deux caméras calibrées. Prenant en compte l’hétérogénéité des données radar et caméras, l’algorithme développé commence par segmenter les données radar en régions polygonales. Grâce au calibrage, l’enveloppe de chaque région est projetée dans deux images afin de définir des régions d’intérêt plus restreintes. Ces régions sont alors segmentées à leur tour en régions polygonales générant ainsi une liste restreinte d’appariement candidats. Un critère basé sur l’inter corrélation et la contrainte épipolaire est appliqué pour valider ou rejeter des paires de régions. Tant que ce critère n’est pas vérifié, les régions sont, elles même, subdivisées par segmentation. Ce processus, favorise l’appariement de régions de grande dimension en premier. L’objectif de cette approche est d’obtenir une cartographie sous forme de patchs localement denses. Les méthodes proposées, ont été testées aussi bien sur des données de synthèse que sur des données expérimentales réelles. Les résultats sont encourageants et montrent, à notre sens, la faisabilité de l’utilisation de ces deux capteurs pour la cartographie d’environnements extérieurs de grande échelle. / The main goal of this PhD work is to develop 3D mapping methods of large scale environment by combining panoramic radar and cameras. Unlike existing sensor fusion methods, such as SLAM (simultaneous localization and mapping), we want to build a RGB-D sensor which directly provides depth measurement enhanced with texture and color information. After modeling the geometry of the radar/camera system, we propose a novel calibration method using points correspondences. To obtain these points correspondences, we designed special targets allowing accurate point detection by both the radar and the camera. The proposed approach has been developed to be implemented by non-expert operators and in unconstrained environment. Secondly, a 3D reconstruction method is elaborated based on radar data and image point correspondences. A theoretical analysis is done to study the influence of the uncertainty zone of each sensor on the reconstruction method. This theoretical study, together with the experimental results, show that the proposed method outperforms the conventional stereoscopic triangulation for large scale outdoor scenes. Finally, we propose an efficient strategy for automatic data matching. This strategy uses two calibrated cameras. Taking into account the heterogeneity of cameras and radar data, the developed algorithm starts by segmenting the radar data into polygonal regions. The calibration process allows the restriction of the search by defining a region of interest in the pair of images. A similarity criterion based on both cross correlation and epipolar constraint is applied in order to validate or reject region pairs. While the similarity test is not met, the image regions are re-segmented iteratively into polygonal regions, generating thereby a shortlist of candidate matches. This process promotes the matching of large regions first which allows obtaining maps with locally dense patches. The proposed methods were tested on both synthetic and real experimental data. The results are encouraging and prove the feasibility of radar and vision sensor fusion for the 3D mapping of large scale urban environment.

Mit der vorliegenden Dissertation soll ein Beitrag zur Steigerung der Genauigkeit und Zuverlässigkeit einer sensorgestützten Objekterkennung im Fahrzeugumfeld geleistet werden. Aufbauend auf einem Erfassungssystem, bestehend aus einer Stereokamera und einem Mehrzeilen-Laserscanner, werden teils neu entwickelte Verfahren für die gesamte Verarbeitungskette vorgestellt. Zusätzlich wird ein neuartiges Framework zur Fusion heterogener Sensordaten eingeführt, welches über eine Zusammenführung der Fusionsergebnisse aus den unterschiedlichen Verarbeitungsebenen in der Lage ist, die Objektbestimmung zu verbessern. Nach einer Beschreibung des verwendeten Sensoraufbaus werden die entwickelten Verfahren zur Kalibrierung des Sensorpaares vorgestellt. Bei der Segmentierung der räumlichen Punktdaten werden bestehende Verfahren durch die Einbeziehung von Messgenauigkeit und Messspezifik des Sensors erweitert. In der anschließenden Objektverfolgung wird neben einem neuartigen berechnungsoptimierten Ansatz zur Objektassoziierung ein Modell zur adaptiven Referenzpunktbestimmung und –Verfolgung beschrieben. Durch das vorgestellte Fusions-Framework ist es möglich, die Sensordaten wahlweise auf drei unterschiedlichen Verarbeitungsebenen (Punkt-, Objekt- und Track-Ebene) zu vereinen. Hierzu wird ein sensorunabhängiger Ansatz zur Fusion der Punktdaten dargelegt, der im Vergleich zu den anderen Fusionsebenen und den Einzelsensoren die genaueste Objektbeschreibung liefert. Für die oberen Fusionsebenen wurden unter Ausnutzung der konkurrierenden Sensorinformationen neuartige Verfahren zur Bestimmung und Reduzierung der Detektions- und Verarbeitungsfehler entwickelt. Abschließend wird beschrieben, wie die fehlerreduzierenden Verfahren der oberen Fusionsebenen mit der optimalen Objektbeschreibung der unteren Fusionsebene für eine optimale Objektbestimmung zusammengeführt werden können. Die Effektivität der entwickelten Verfahren wurde durch Simulation oder in realen Messszenarien überprüft. / With the present thesis a contribution to the increase of the accuracy and reliability of a sensor-supported recognition and tracking of objects in a vehicle’s surroundings should be made. Based on a detection system, consisting of a stereo camera and a laser scanner, novel developed procedures are introduced for the whole processing chain of the sensor data. In addition, a new framework is introduced for the fusion of heterogeneous sensor data. By combining the data fusion results from the different processing levels the object detection can be improved. After a short description of the used sensor setup the developed procedures for the calibration and mutual orientation are introduced. With the segmentation of the spatial point data existing procedures are extended by the inclusion of measuring accuracy and specificity of the sensor. In the subsequent object tracking a new computation-optimized approach for the association of the related object hypotheses is presented. In addition, a model for a dynamic determination and tracking of an object reference point is described which exceeds the classical tracking of the object center in the track accuracy. By the introduced fusion framework it is possible to merge the sensor data at three different processing levels (point, object and track level). A sensor independent approach for the low fusion of point data is demonstrated which delivers the most precise object description in comparison to the other fusion levels and the single sensors. For the higher fusion levels new procedures were developed to discover and clean up the detection and processing mistakes benefiting from the competing sensor information. Finally it is described how the fusion results of the upper and lower levels can be brought together for an ideal object description. The effectiveness of the newly developed methods was checked either by simulation or in real measurement scenarios.

Interventional magnetic resonance imaging (MRI) aims at performing minimally invasive percutaneous interventions, such as tumor ablations and biopsies, under MRI guidance. During such interventions, the acquired MR image planes are typically aligned to the surgical instrument (needle) axis and to surrounding anatomical structures of interest in order to efficiently monitor the advancement in real-time of the instrument inside the patient's body. Object tracking inside the MRI is expected to facilitate and accelerate MR-guided interventions by allowing to automatically align the image planes to the surgical instrument. In this PhD thesis, an image-based workflow is proposed and refined for automatic image plane alignment. An automatic tracking workflow was developed, performing detection and tracking of a passive marker directly in clinical real-time images. This tracking workflow is designed for fully automated image plane alignment, with minimization of tracking-dedicated time. Its main drawback is its inherent dependence on the slow clinical MRI update rate. First, the addition of motion estimation and prediction with a Kalman filter was investigated and improved the workflow tracking performance. Second, a complementary optical sensor was used for multi-sensor tracking in order to decouple the tracking update rate from the MR image acquisition rate. Performance of the workflow was evaluated with both computer simulations and experiments using an MR compatible testbed. Results show a high robustness of the multi-sensor tracking approach for dynamic image plane alignment, due to the combination of the individual strengths of each sensor.