Feature extraction is an essential part of the Visual Odometry problem. In recent years, with the rise of Neural Networks, the problem has shifted from a more classical to a deep learning approach. This thesis presents a fine-tuned feature extraction network trained on pose estimation as a proxy task. The architecture aims at integrating inertial information coming from IMU sensor data in the deep local feature extraction paradigm. Specifically, visual features and inertial features are extracted using Neural Networks. These features are then fused together and further processed to regress the pose of a moving agent. The visual feature extraction network is effectively fine-tuned and is used stand-alone for inference. The approach is validated via a qualitative analysis on the keypoints extracted and also in a more quantitative way. Quantitatively, the feature extraction network is used to perform Visual Odometry on the Kitti dataset where the ATE for various sequences is reported. As a comparison, the proposed method, the proposed without IMU and the original pre-trained feature extraction network are used to extract features for the Visual Odometry task. Their ATE results and relative trajectories show that in sequences with great change in orientation the proposed system outperforms the original one, while on mostly straight sequences the original system performs slightly better. / Feature extraktion är en viktig del av visuell odometri (VO). Under de senaste åren har framväxten av neurala nätverk gjort att tillvägagångsättet skiftat från klassiska metoder till Deep Learning metoder. Denna rapport presenterar ett kalibrerat feature extraheringsnätverk som är tränat med posesuppskattning som en proxyuppgift. Arkitekturen syftar till att integrera tröghetsinformation som kommer från sensordata i feature extraheringsnätverket. Specifikt extraheras visuella features och tröghetsfeatures med hjälp av neurala nätverk. Dessa features slås ihop och bearbetas ytterligare för att estimera position och riktning av en rörlig kamera. Metoden har undersökts genom en kvalitativ analys av featurepunkternas läge men även på ett mer kvantitativt sätt där VO-estimering på olika bildsekvenser från KITTI-datasetet har jämförts. Resultaten visar att i sekvenser med stora riktningsförändringar överträffar det föreslagna systemet det ursprungliga, medan originalsystemet presterar något bättre på sekvenser som är mestadels raka.

Visual Odometry (VO) is the process of estimating the relative motion of a vehicle by using solely image data gathered from the camera. In underwater environments, VO becomes extremely challenging but valuable since ordinary sensors for on-road localization are usually unpractical in these hostile environments. For years, VO methods have been purely based on Computer Vision (CV) principles. However, the recent advances in Deep Learning (DL) have ushered in a new era for VO approaches. These novel methods have achieved impressive performance with state-of-the-art results on urban datasets. Nevertheless, little effort has been made to push learning-based research towards natural environments, such as underwater. Consequently, this work aims to bridge the research gap by evaluating the effectiveness of the learning-based approach in the navigation of Autonomous Underwater Vehicles (AUVs). We compare two learning-based methods with a traditional feature-based method on the Underwater Caves dataset, a very challenging dataset collected in the unstructured environment of an underwater cave complex. Extensive experiments are thus conducted training the models on this dataset. Moreover, we investigate different aspects and propose several improvements, such as sub-sampling the video clips to emphasize the camera motion between consecutive frames, or training exclusively on images with relevant content discarding those with dark borders and representing solely sandy bottoms. Finally, during the training, we also leverage underwater images from other datasets, hence acquired from different cameras. However, the best improvement is obtained by penalizing rotations around the x-axis of the camera coordinate system. The three methods are evaluated on test sequences that cover different lighting conditions. In the most favorable environments, although learning-based methods are not up to par with the feature-based method, the results show great potential. Furthermore, in extreme lighting conditions, where the feature-based baseline sharply fails to bootstrap, one of the two learning-based methods produces instead qualitatively good trajectory results, revealing the power of the learning-based approach in this peculiar context. / Visuell Odometri (VO) används för att uppskatta den relativa rörelsen för ett fordon med hjälp av enbart bilddata från en eller flera kameror. I undervattensmiljöer blir VO extremt utmanande men värdefullt eftersom vanliga sensorer för lokalisering vanligtvis är opraktiska i dessa svåra miljöer. I åratal har VO-metoder enbart baserats på klassisk datorseende. De senaste framstegen inom djupinlärning har dock inlett en ny era för VO-metoder. Dessa nya metoder har uppnått imponerande prestanda på dataset urbana miljöer. Trots detta har ganska lite gjorts för att driva den inlärningsbaserad forskningen mot naturliga miljöer, till exempel under vattnet. Följaktligen syftar detta arbete till att överbrygga forskningsgapet genom att utvärdera effektiviteten hos det inlärningsbaserade tillvägagångssättet vid navigering av autonoma undervattensfordon (AUV). Vi jämför två inlärningsbaserade metoder med en traditionell nyckelpunktsbaserad metod som referens. Vi gör jämförelsen på Underwater Caves-datasetet, ett mycket utmanande dataset som samlats in i den ostrukturerade miljön i ett undervattensgrottkomplex. Omfattande experiment utförs för att träna modellerna på detta dataset. Vi undersöker också olika aspekter och föreslår flera förbättringar, till exempel, att delsampla videoklippen för att betona kamerarörelsen mellan på varandra följande bildrutor, eller att träna på en delmängd av datasetet bestående uteslutande på bilder med relevant innehåll för att förbättra skattningen av rörelsen. Under träningen utnyttjar vi också undervattensbilder från andra datamängder, och därmed från olika kameror. Den bästa förbättringen uppnås dock genom att straffa skattningar av stora rotationer runt kamerakoordinatsystemets x-axel. De tre metoderna utvärderas på testsekvenser som täcker olika ljusförhållanden. I de mest gynnsamma miljöerna visar resultaten stor potential, även om de inlärningsbaserade metoder inte är i nivå med den traditionella referensmetoden. Vid extrema ljusförhållanden, där referensmetoden misslyckas att ens initialisera, ger en av de två inlärningsbaserade metoderna istället kvalitativt bra resultat, vilket demonstrerar kraften i det inlärningsbaserade tillvägagångssättet i detta specifika sammanhang.

No description available.

Intelligent vehicles require perception systems with high performances. Usually, perception system consists of multiple sensors, such as cameras, 2D/3D lidars or radars. The works presented in this Ph.D thesis concern several topics on cameras and lidar based perception for understanding dynamic scenes in urban environments. The works are composed of four parts.In the first part, a stereo vision based visual odometry is proposed by comparing several different approaches of image feature detection and feature points association. After a comprehensive comparison, a suitable feature detector and a feature points association approach is selected to achieve better performance of stereo visual odometry. In the second part, independent moving objects are detected and segmented by the results of visual odometry and U-disparity image. Then, spatial features are extracted by a kernel-PCA method and classifiers are trained based on these spatial features to recognize different types of common moving objects e.g. pedestrians, vehicles and cyclists. In the third part, an extrinsic calibration method between a 2D lidar and a stereoscopic system is proposed. This method solves the problem of extrinsic calibration by placing a common calibration chessboard in front of the stereoscopic system and 2D lidar, and by considering the geometric relationship between the cameras of the stereoscopic system. This calibration method integrates also sensor noise models and Mahalanobis distance optimization for more robustness. At last, dynamic occupancy grid mapping is proposed by 3D reconstruction of the environment, obtained from stereovision and Lidar data separately and then conjointly. An improved occupancy grid map is obtained by estimating the pitch angle between ground plane and the stereoscopic system. The moving object detection and recognition results (from the first and second parts) are incorporated into the occupancy grid map to augment the semantic meanings. All the proposed and developed methods are tested and evaluated with simulation and real data acquired by the experimental platform "intelligent vehicle SetCar" of IRTES-SET laboratory.

Dans ce travail, nous proposons une représentation efficace de l’environnement adaptée à la problématique de la navigation autonome. Cette représentation topométrique est constituée d’un graphe de sphères de vision augmentées d’informations de profondeur. Localement la sphère de vision augmentée constitue une représentation égocentrée complète de l’environnement proche. Le graphe de sphères permet de couvrir un environnement de grande taille et d’en assurer la représentation. Les "poses" à 6 degrés de liberté calculées entre sphères sont facilement exploitables par des tâches de navigation en temps réel. Dans cette thèse, les problématiques suivantes ont été considérées : Comment intégrer des informations géométriques et photométriques dans une approche d’odométrie visuelle robuste ; comment déterminer le nombre et le placement des sphères augmentées pour représenter un environnement de façon complète ; comment modéliser les incertitudes pour fusionner les observations dans le but d’augmenter la précision de la représentation ; comment utiliser des cartes de saillances pour augmenter la précision et la stabilité du processus d’odométrie visuelle. / Our aim is concentrated around building ego-centric topometric maps represented as a graph of keyframe nodes which can be efficiently used by autonomous agents. The keyframe nodes which combines a spherical image and a depth map (augmented visual sphere) synthesises information collected in a local area of space by an embedded acquisition system. The representation of the global environment consists of a collection of augmented visual spheres that provide the necessary coverage of an operational area. A "pose" graph that links these spheres together in six degrees of freedom, also defines the domain potentially exploitable for navigation tasks in real time. As part of this research, an approach to map-based representation has been proposed by considering the following issues : how to robustly apply visual odometry by making the most of both photometric and ; geometric information available from our augmented spherical database ; how to determine the quantity and optimal placement of these augmented spheres to cover an environment completely ; how tomodel sensor uncertainties and update the dense infomation of the augmented spheres ; how to compactly represent the information contained in the augmented sphere to ensure robustness, accuracy and stability along an explored trajectory by making use of saliency maps.

Les robots et véhicules autonomes représentent le futur des modes de déplacements et de production. Les enjeux de l’avenir reposent sur la robustesse de leurs perceptions et flexibilité face aux environnements changeant et situations inattendues. Les capteurs stéréoscopiques sont des capteurs passifs qui permettent d'obtenir à la fois image et information 3D de la scène à la manière de la vision humaine. Dans ces travaux nous avons développé un système de localisation, par odométrie visuelle permettant de déterminer la position dans l'espace du capteur de façon efficace et performante en tirant partie de la carte de profondeur dense mais également associé à un système de SLAM, rendant la localisation robuste aux perturbations et aux décalages potentiels. Nous avons également développé plusieurs solutions de cartographie et interprétation d’obstacles, à la fois pour le véhicule aérien et terrestre. Ces travaux sont en partie intégrés dans des produits commerciaux. / Autonomous vehicles and robots represent the future of transportation and production industries. The challenge ahead will come from the robustness of perception and flexibility from unexpected situations and changing environments. Stereoscopic cameras are passive sensors that provide color images and depth information of the scene by correlating 2 images like the human vision. In this work, we developed a localization system, by visual odometry that can determine efficiently the position in space of the sensor by exploiting the dense depth map. It is also combined with a SLAM system that enables robust localization against disturbances and potentials drifts. Additionally, we developed a few mapping and obstacles detections solutions, both for aerial and terrestrial vehicles. These algorithms are now partly integrated into commercial products.

[pt] Alvo de intenso estudo da visão computacional, a reconstrução densa 3D teve um importante marco com os primeiros sistemas em tempo real a alcançarem precisão milimétrica com uso de câmeras RGBD e GPUs. Entretanto estes métodos não são aplicáveis a dispositivos de menor poder computacional. Tendo a limitação de recursos computacionais como requisito, o objetivo deste trabalho é apresentar um método de odometria visual utilizando câmeras comuns e sem a necessidade de GPU, baseado em técnicas de Structure from Motion (SFM) com features esparsos, utilizando as informações de uma reconstrução densa. A Odometria visual é o processo de estimar a orientação e posição de um agente (um robô, por exemplo), a partir das imagens. Esta dissertação fornece uma comparação entre a precisão da odometria calculada pelo método proposto e pela reconstrução densa utilizando o Kinect Fusion. O resultado desta pesquisa é diretamente aplicável na área de realidade aumentada, tanto pelas informações da odometria que podem ser usadas para definir a posição de uma câmera, como pela reconstrução densa, que pode tratar aspectos como oclusão dos objetos virtuais com reais. / [en] Aim of intense research in the field computational vision, dense 3D reconstruction achieves an important landmark with first methods running in real time with millimetric precision, using RGBD cameras and GPUs. However these methods are not suitable for low computational resources. Having low computational resources as requirement, the goal of this work is to show a method of visual odometry using regular cameras, without using a GPU. The proposed method is based on technics of sparse Structure From Motion (SFM), using data provided by dense 3D reconstruction. Visual odometry is the process of estimating the position and orientation of an agent (a robot, for instance), based on images. This dissertation compares the proposed method with the odometry calculated by Kinect Fusion. Results of this research are applicable in augmented reality. Odometry provided by this work can be used to model a camera and the data from dense 3D reconstruction, can be used to handle occlusion between virtual and real objects.

This master thesis, conducted at SAAB Dynamics Linköping, studies Visual Odometry (VO), or camera pose estimation, using a monocular event camera. Event cameras are not yet widely used in the industry, and there is significant interest in understanding the methodological differences between performing Visual Odometry (VO) with event cameras compared to traditional frame cameras. Event cameras have the potential to capture information between frames, which may include data that is lost or not captured by frame cameras. This thesis compares two different feature detectors and evaluates their performance against a frame-based method. Visual Odometry was conducted both with and without known 3D points. However, attempts to perform VO without known 3D points did not yield a robust pipeline within the limited time frame of this thesis, and thus was not further developed to function with a purely event-based method. On the other hand, VO performance with known 3D points achieved continues 6-DoF pose estimation. The results demonstrate that event cameras have the potential to detect and track features in challenging scenes, such as those with dark or bright lighting conditions, for example, objects passing by the sun. This thesis suggests that implementing a robust 6-DoF pose estimation would be feasible with a more reliable 3D-2D point pair matching technique or a more sophisticated VO pipeline.

This bachelors thesis presents a vision based control system for the quadrotor aerial vehicle,Crazy ie 2.0, developed by Bitcraze AB. The main goal of this thesis is to design andimplement an o-board control system based on visual input, in order to control the positionand orientation of the vehicle with respect to a single ducial marker. By integrating a cameraand wireless video transmitter onto the MAV platform, we are able to achieve autonomousnavigation and landing in relatively close proximity to the dedicated target location.The control system was developed in the programming language Python and all processing ofthe vision-data take place on an o-board computer. This thesis describes the methods usedfor developing and implementing the control system and a number of experiments have beencarried out in order to determine the performance of the overall vision control system. Withthe proposed method of using ducial markers for calculating the control demands for thequadrotor, we are able to achieve autonomous targeted landing within a radius of 10centimetres away from the target location. / I detta examensarbete presenteras ett visionsbaserat kontrollsystem for dronaren Crazy ie 2.0som har utvecklats av Bitcraze AB. Malet med detta arbete ar att utforma och implementeraett externt kontrollsystem baserat pa data som inhamtas av en kamera for att reglera fordonetsposition och riktning med avseende pa en markor placerad i synfaltet av kameran. Genom attintegrera kameran tillsammans med en tradlos videosandare pa plattformen, visar vi i dennaavhandling att det ar mojligt att astadkomma autonom navigering och landning i narheten avmarkoren.Kontrollsystemet utvecklades i programmeringsspraket Python och all processering avvisions-datan sker pa en extern dator. Metoderna som anvands for att utvecklakontrollsystemet och som beskrivs i denna rapport har testats under ett ertal experiment somvisar pa hur val systemet kan detektera markoren och hur val de olika ingaendekomponenterna samspelar for att kunna utfora den autonoma styrningen. Genom den metodsom presenteras i den har rapporten for att berakna styrsignalerna till dronaren med hjalp avvisuell data, visar vi att det ar mojligt att astadkomma autonom styrning och landning motmalet inom en radie av 10 centimeter.

Most micro air vehicles rely heavily on reliable GPS measurements for proper estimation and control, and therefore struggle in GPS-degraded environments. When GPS is not available, the global position and heading of the vehicle is unobservable. This dissertation establishes the theoretical and practical advantages of a relative navigation framework for MAV navigation in GPS-degraded environments. This dissertation explores how the consistency, accuracy, and stability of current navigation approaches degrade during prolonged GPS dropout and in the presence of heading uncertainty. Relative navigation (RN) is presented as an alternative approach that maintains observability by working with respect to a local coordinate frame. RN is compared with several current estimation approaches in a simulation environment and in hardware experiments. While still subject to global drift, RN is shown to produce consistent state estimates and stable control. Estimating relative states requires unique modifications to current estimation approaches. This dissertation further provides a tutorial exposition of the relative multiplicative extended Kalman filter, presenting how to properly ensure observable state estimation while maintaining consistency. The filter is derived using both inertial and body-fixed state definitions and dynamics. Finally, this dissertation presents a series of prolonged flight tests, demonstrating the effectiveness of the relative navigation approach for autonomous GPS-degraded MAV navigation in varied, unknown environments. The system is shown to utilize a variety of vision sensors, work indoors and outdoors, run in real-time with onboard processing, and not require special tuning for particular sensors or environments. Despite leveraging off-the-shelf sensors and algorithms, the flight tests demonstrate stable front-end performance with low drift. The flight tests also demonstrate the onboard generation of a globally consistent, metric, and localized map by identifying and incorporating loop-closure constraints and intermittent GPS measurements. With this map, mission objectives are shown to be autonomously completed.