Les travaux réalisés au cours de cette thèse s’inscrivent dans la problématique de localisation en temps réel d’une caméra par vision monoculaire. Dans la littérature, il existe différentes méthodes qui peuvent être classées en trois catégories. La première catégorie de méthodes considère une caméra évoluant dans un environnement complètement inconnu (SLAM). Cette méthode réalise une reconstruction enligne de primitives observées dans des images d’une séquence vidéo et utilise cette reconstruction pour localiser la caméra. Les deux autres permettent une localisation par rapport à un objet 3D de la scène en s’appuyant sur la connaissance, a priori, d’un modèle de cet objet (suivi basé modèle). L’une utilise uniquement l’information du modèle 3D de l’objet pour localiser la caméra, l’autre peut être considérée comme l’intermédiaire entre le SLAM et le suivi basé modèle. Cette dernière méthode consiste à localiser une caméra par rapport à un objet en utilisant, d’une part, le modèle de ce dernier et d’autre part, une reconstruction en ligne des primitives de l’objet d’intérêt. Cette reconstruction peut être assimilée à une mise à jour du modèle initial (suivi basé modèle avec mise à jour). Chacune de ces méthodes possède des avantages et des inconvénients. Dans le cadre de ces travaux de thèse, nous proposons une solution unifiant l’ensemble de ces méthodes de localisation dans un unique cadre désigné sous le terme de SLAM contraint. Cette solution, qui unifie ces différentes méthodes, permet de tirer profit de leurs avantages tout en limitant leurs inconvénients respectifs. En particulier, nous considérons que la caméra évolue dans un environnement partiellement connu, c’est-à-dire pour lequel un modèle (géométrique ou photométrique) 3D d’un objet statique de la scène est disponible. L’objectif est alors d’estimer de manière précise la pose de la caméra par rapport à cet objet 3D. L’information absolue issue du modèle 3D de l’objet d’intérêt est utilisée pour améliorer la localisation de type SLAM en incluant cette information additionnelle directement dans le processus d’ajustement de faisceaux. Afin de pouvoir gérer un large panel d’objets 3D et de scènes, plusieurs types de contraintes sont proposées dans ce mémoire. Ces différentes contraintes sont regroupées en deux approches. La première permet d’unifier les méthodes SLAM et de suivi basé modèle, en contraignant le déplacement de la caméra via la projection de primitives existantes extraites du modèle 3D dans les images. La seconde unifie les méthodes SLAM et de suivi basé modèle avec mise à jour en contraignant les primitives reconstruites par le SLAM à appartenir à la surface du modèle (unification SLAM et mise à jour du modèle). Les avantages de ces différents ajustements de faisceaux contraints, en terme de précision, de stabilité de recalage et de robustesse aux occultations, sont démontrés sur un grand nombre de données de synthèse et de données réelles. Des applications temps réel de réalité augmentée sont également présentées sur différents types d’objets 3D. Ces travaux ont fait l’objet de 4 publications internationales, de 2 publications nationales et d’un dépôt de brevet. / This thesis tackles the problem of real time location of a monocular camera. In the literature, there are different methods which can be classified into three categories. The first category considers a camera moving in a completely unknown environment (SLAM). This method performs an online reconstruction of the observed primitives in the images and uses this reconstruction to estimate the location of the camera. The two other categories of methods estimate the location of the camera with respect to a 3D object in the scene. The estimation is based on an a priori knowledge of a model of the object (Model-based). One of these two methods uses only the information of the 3D model of the object to locate the camera. The other method may be considered as an intermediary between the SLAM and Model-based approaches. It consists in locating the camera with respect to the object of interest by using, on one hand the 3D model of this object, and on the other hand an online reconstruction of the primitives of the latter. This last online reconstruction can be regarded as an update of the initial 3D model (Model-based with update). Each of these methods has advantages and disadvantages. In the context of this thesis, we propose a solution in order to unify all these localization methods in a single framework referred to as the constrained SLAM, by taking parts of their benefits and limiting their disadvantages. We, particularly, consider that the camera moves in a partially known environment, i.e. for which a 3D model (geometric or photometric) of a static object in the scene is available. The objective is then to accurately estimate the pose (position and orientation) of the camera with respect to this object. The absolute information provided by the 3D model of the object is used to improve the localization of the SLAM by directly including this additional information in the bundle adjustment process. In order to manage a wide range of 3D objets and scenes, various types of constraints are proposed in this study and grouped into two approaches. The first one allows to unify the SLAM and Model-based methods by constraining the trajectory of the camera through the projection, in the images, of the 3D primitives extracted from the model. The second one unifies the SLAM and Model-based with update methods, by constraining the reconstructed 3D primitives of the object to belong to the surface of the model (unification SLAM and model update). The benefits of the constrained bundle adjustment framework in terms of accuracy, stability, robustness to occlusions, are demonstrated on synthetic and real data. Real time applications of augmented reality are also presented on different types of 3D objects. This work has been the subject of four international publications, two national publications and one patent.

[pt] SLAM (Mapeamento e Localização Simultânea) é uma das áreas mais pesquisadas na Robótica móvel. Trata-se do problema, num robô móvel, de construir um mapa sem conhecimento prévio do ambiente e ao mesmo tempo manter a sua localização nele. Embora a tecnologia ofereça sensores cada vez mais precisos, pequenos erros na medição são acumulados comprometendo a precisão na localização, sendo estes evidentes quando o robô retorna a uma posição inicial depois de percorrer um longo caminho. Assim, para melhoria do desempenho do SLAM é necessário representar a sua formulação usando teoria das probabilidades. O SLAM com Filtro Extendido de Kalman (EKF-SLAM) é uma solução básica, e apesar de suas limitações é a técnica mais popular. O Fast SLAM, por outro lado, resolve algumas limitações do EKF-SLAM usando uma instância do filtro de partículas conhecida como Rao-Blackwellized. Outra solução bem sucedida é o DP-SLAM, o qual usa uma representação do mapa em forma de grade de ocupação, com um algoritmo hierárquico que constrói mapas 2D bastante precisos. Todos estes algoritmos usam informação de dois tipos de sensores: odômetros e sensores de distância. O Laser Range Finder (LRF) é um medidor laser de distância por varredura, e pela sua precisão é bastante usado na correção do erro em odômetros. Este trabalho apresenta uma detalhada implementação destas três soluções para o SLAM, focalizado em ambientes fechados e estruturados. Apresenta-se a construção de mapas 2D e 3D em terrenos planos tais como em aplicações típicas de ambientes fechados. A representação dos mapas 2D é feita na forma de grade de ocupação. Por outro lado, a representação dos mapas 3D é feita na forma de nuvem de pontos ao invés de grade, para reduzir o custo computacional. É considerado um robô móvel equipado com apenas um LRF, sem nenhuma informação de odometria. O alinhamento entre varreduras laser é otimizado fazendo o uso de Algoritmos Genéticos. Assim, podem-se construir mapas e ao mesmo tempo localizar o robô sem necessidade de odômetros ou outros sensores. Um simulador em Matlab é implementado para a geração de varreduras virtuais de um LRF em um ambiente 3D (virtual). A metodologia proposta é validada com os dados simulados, assim como com dados experimentais obtidos da literatura, demonstrando a possibilidade de construção de mapas 3D com apenas um sensor LRF. / [en] Simultaneous Localization and Mapping (SLAM) is one of the most widely researched areas of Robotics. It addresses the mobile robot problem of generating a map without prior knowledge of the environment, while keeping track of its position. Although technology offers increasingly accurate position sensors, even small measurement errors can accumulate and compromise the localization accuracy. This becomes evident when programming a robot to return to its original position after traveling a long distance, based only on its sensor readings. Thus, to improve SLAM s performance it is necessary to represent its formulation using probability theory. The Extended Kalman Filter SLAM (EKF-SLAM) is a basic solution and, despite its shortcomings, it is by far the most popular technique. Fast SLAM, on the other hand, solves some limitations of the EKFSLAM using an instance of the Rao-Blackwellized particle filter. Another successful solution is to use the DP-SLAM approach, which uses a grid representation and a hierarchical algorithm to build accurate 2D maps. All SLAM solutions require two types of sensor information: odometry and range measurement. Laser Range Finders (LRF) are popular range measurement sensors and, because of their accuracy, are well suited for odometry error correction. Furthermore, the odometer may even be eliminated from the system if multiple consecutive LRF scans are matched. This works presents a detailed implementation of these three SLAM solutions, focused on structured indoor environments. The implementation is able to map 2D environments, as well as 3D environments with planar terrain, such as in a typical indoor application. The 2D application is able to automatically generate a stochastic grid map. On the other hand, the 3D problem uses a point cloud representation of the map, instead of a 3D grid, to reduce the SLAM computational effort. The considered mobile robot only uses a single LRF, without any odometry information. A Genetic Algorithm is presented to optimize the matching of LRF scans taken at different instants. Such matching is able not only to map the environment but also localize the robot, without the need for odometers or other sensors. A simulation program is implemented in Matlab to generate virtual LRF readings of a mobile robot in a 3D environment. Both simulated readings and experimental data from the literature are independently used to validate the proposed methodology, automatically generating 3D maps using just a single LRF.

Made available in DSpace on 2014-12-17T14:54:48Z (GMT). No. of bitstreams: 1 DiogoPFP_Tese.pdf: 4402228 bytes, checksum: 17eacb6b5f1731f405518c976d32f701 (MD5) Previous issue date: 2006-05-19 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior / The objective of this thesis is proposes a method for a mobile robot to build a hybrid map of an indoor, semi-structured environment. The topological part of this map deals with spatial relationships among rooms and corridors. It is a topology-based map, where the edges of the graph are rooms or corridors, and each link between two distinct edges represents a door. The metric part of the map consists in a set of parameters. These parameters describe a geometric figure which adapts to the free space of the local environment. This figure is calculated by a set of points which sample the boundaries of the local free space. These points are obtained with range sensors and with knowledge about the robot s pose. A method based on generalized Hough transform is applied to this set of points in order to obtain the geomtric figure. The building of the hybrid map is an incremental procedure. It is accomplished while the robot explores the environment. Each room is associated with a metric local map and, consequently, with an edge of the topo-logical map. During the mapping procedure, the robot may use recent metric information of the environment to improve its global or relative pose / Esta tese tem o objetivo de propor uma metodologia para constru??o de um mapa h?brido de um ambiente interno. A parte topol?gica da representa??o trata das rela??es de conectividade existentes entre as salas e corredores, sendo assim um grafo que representa a topologia do ambiente global. A parte m?trica consiste em armazenar um conjunto de par?metros que descreve uma figura geom?trica plana que melhor se ajusta ao espa?o livre local. Esta figura ? calculada atrav?s do conhecimento de pontos, ou amostras, dos limites do espa?o livre. Estes pontos s?o obtidos com sensores de dist?ncia e a informa??o ? complementada com a estimativa da pose do rob?. Uma vez que estes pontos est?o determinados, o rob? ent?o aplica uma ferramenta baseada na transformada generalizada de Hough para obter a figura em quest?o. O processo de constru??o do mapa ? incremental e totalmente realizado enquanto o rob? explora o ambiente. Cada sala ? representada por este mapa local e cada n? do grafo que representa a topologia do ambiente est? associado a este mapa. Durante o mapeamento o rob? pode utilizar as informa??es rec?m-adquiridas do ambiente para obter uma melhor estimativa de sua pose global ou relativa a uma sala ou corredor

Dans la communauté robotique aérienne, un croissant intérêt pour les systèmes multirobot (SMR) est apparu ces dernières années. Cela a été motivé par i) les progrès technologiques, tels que de meilleures capacités de traitement à bord des robots et des performances de communication plus élevées, et ii) les résultats prometteurs du déploiement de SMR tels que l’augmentation de la zone de couverture en un minimum de temps. Le développement d’une flotte de véhicules aériens sans pilote (UAV: Unmanned Aerial Vehicle) et de véhicules aériens de petite taille (MAV: Micro Aerial Vehicle) a ouvert la voie à de nouvelles applications à grande échelle nécessitant les caractéristiques de tel système de systèmes dans des domaines tels que la sécurité, la surveillance des catastrophes et des inondations, la recherche et le sauvetage, l’inspection des infrastructures, et ainsi de suite. De telles applications nécessitent que les robots identifient leur environnement et se localisent. Ces tâches fondamentales peuvent être assurées par la mission d’exploration. Dans ce contexte, cette thèse aborde l’exploration coopérative d’un environnement inconnu en utilisant une équipe de drones avec vision intégrée. Nous avons proposé un système multi-robot où le but est de choisir des régions spécifiques de l’environnement à explorer et à cartographier simultanément par chaque robot de manière optimisée, afin de réduire le temps d’exploration et, par conséquent, la consommation d’énergie. Chaque UAV est capable d’effectuer une localisation et une cartographie simultanées (SLAM: Simultaneous Localization And Mapping) à l’aide d’un capteur visuel comme principale modalité de perception. Pour explorer les régions inconnues, les cibles – choisies parmi les points frontières situés entre les zones libres et les zones inconnues – sont assignées aux robots en considérant un compromis entre l’exploration rapide et l’obtention d’une carte détaillée. À des fins de prise de décision, les UAVs échangent habituellement une copie de leur carte locale, mais la nouveauté dans ce travail est d’échanger les points frontières de cette carte, ce qui permet d’économiser la bande passante de communication. L’un des points les plus difficiles du SMR est la communication inter-robot. Nous étudions cette partie sous les aspects topologiques et typologiques. Nous proposons également des stratégies pour faire face à l’abandon ou à l’échec de la communication. Des validations basées sur des simulations étendues et des bancs d’essai sont présentées. / In the aerial robotic community, a growing interest for Multi-Robot Systems (MRS) appeared in the last years. This is thanks to i) the technological advances, such as better onboard processing capabilities and higher communication performances, and ii) the promising results of MRS deployment, such as increased area coverage in minimum time. The development of highly efficient and affordable fleet of Unmanned Aerial Vehicles (UAVs) and Micro Aerial Vehicles (MAVs) of small size has paved the way to new large-scale applications, that demand such System of Systems (SoS) features in areas like security, disaster surveillance, inundation monitoring, search and rescue, infrastructure inspection, and so on. Such applications require the robots to identify their environment and localize themselves. These fundamental tasks can be ensured by the exploration mission. In this context, this thesis addresses the cooperative exploration of an unknown environment sensed by a team of UAVs with embedded vision. We propose a multi-robot framework where the key problem is to cooperatively choose specific regions of the environment to be simultaneously explored and mapped by each robot in an optimized manner in order to reduce exploration time and, consequently, energy consumption. Each UAV is able to performSimultaneous Localization And Mapping (SLAM) with a visual sensor as the main input sensor. To explore the unknown regions, the targets – selected from the computed frontier points lying between free and unknown areas – are assigned to robots by considering a trade-off between fast exploration and getting detailed grid maps. For the sake of decision making, UAVs usually exchange a copy of their local map; however, the novelty in this work is to exchange map frontier points instead, which allow to save communication bandwidth. One of the most challenging points in MRS is the inter-robot communication. We study this part in both topological and typological aspects. We also propose some strategies to cope with communication drop-out or failure. Validations based on extensive simulations and testbeds are presented.

Les robots et véhicules autonomes représentent le futur des modes de déplacements et de production. Les enjeux de l’avenir reposent sur la robustesse de leurs perceptions et flexibilité face aux environnements changeant et situations inattendues. Les capteurs stéréoscopiques sont des capteurs passifs qui permettent d'obtenir à la fois image et information 3D de la scène à la manière de la vision humaine. Dans ces travaux nous avons développé un système de localisation, par odométrie visuelle permettant de déterminer la position dans l'espace du capteur de façon efficace et performante en tirant partie de la carte de profondeur dense mais également associé à un système de SLAM, rendant la localisation robuste aux perturbations et aux décalages potentiels. Nous avons également développé plusieurs solutions de cartographie et interprétation d’obstacles, à la fois pour le véhicule aérien et terrestre. Ces travaux sont en partie intégrés dans des produits commerciaux. / Autonomous vehicles and robots represent the future of transportation and production industries. The challenge ahead will come from the robustness of perception and flexibility from unexpected situations and changing environments. Stereoscopic cameras are passive sensors that provide color images and depth information of the scene by correlating 2 images like the human vision. In this work, we developed a localization system, by visual odometry that can determine efficiently the position in space of the sensor by exploiting the dense depth map. It is also combined with a SLAM system that enables robust localization against disturbances and potentials drifts. Additionally, we developed a few mapping and obstacles detections solutions, both for aerial and terrestrial vehicles. These algorithms are now partly integrated into commercial products.

In der heutigen Zeit spielt die Automatisierung eine immer bedeutendere Rolle, speziell im Bereich der Robotik entwickeln sich immer neue Einsatzgebiete, in denen der Mensch durch autonome Fahrzeuge ersetzt wird. Dabei orientiert sich der Großteil der eingesetzten Roboter an Streckenmarkierungen, die in den Einsatzumgebungen installiert sind. Bei diesen Systemen gibt es jedoch einen hohen Installationsaufwand, was die Entwicklung von Robotersystemen, die sich mithilfe ihrer verbauten Sensorik orientieren, vorantreibt. Es existiert zwar eine Vielzahl an Robotern die dafür verwendet werden können. Die Entwicklung der Steuerungssoftware ist aber immer noch Teil der Forschung. Für die Steuerung wird eine Umgebungskarte benötigt, an der sich der Roboter orientieren kann. Hierfür eignen sich besonders SLAM-Verfahren, die simultanes Lokalisieren und Kartographieren durchführen. Dabei baut der Roboter während seiner Bewegung durch den Raum mithilfe seiner Sensordaten eine Umgebungskarte auf und lokalisiert sich daran, um seine Position auf der Karte exakt zu bestimmen. Im Laufe dieser Arbeit wurden über 30 verschiedene SLAM Implementierungen bzw. Umsetzungen gefunden die das SLAM Problem lösen. Diese sind jedoch größtenteils an spezielle Systembzw. Umgebungsvoraussetzungen angepasste eigenständige Implementierungen. Es existiert keine öffentlich zugängliche Übersicht, die einen Vergleich aller bzw. des Großteils der Verfahren, z.B. in Bezug auf ihre Funktionsweise, Systemvoraussetzungen (Sensorik, Roboterplattform), Umgebungsvoraussetzungen (Indoor, Outdoor, ...), Genauigkeit oder Geschwindigkeit, gibt. Viele dieser SLAMs besitzen Implementierungen und Dokumentationen in denen ihre Einsatzgebiete, Testvoraussetzungen oder Weiterentwicklungen im Vergleich zu anderen SLAMVerfahren beschrieben werden, was aber bei der großen Anzahl an Veröffentlichungen das Finden eines passenden SLAM-Verfahrens nicht erleichtert. Bei einer solchen Menge an SLAM-Verfahren und Implementierungen stellen sich aus softwaretechnologischer Sicht folgende Fragen: 1. Besteht die Möglichkeit einzelne Teile des SLAM wiederzuverwenden? 2. Besteht die Möglichkeit einzelne Teile des SLAM dynamisch auszutauschen? Mit dieser Arbeit wird das Ziel verfolgt, diese beiden Fragen zu beantworten. Hierfür wird zu Beginn eine Übersicht über alle gefundenen SLAMs aufgebaut um diese in ihren grundlegenden Eigenschaften zu unterscheiden. Aus der Vielzahl von Verfahren werden die rasterbasierten Verfahren, welche Laserscanner bzw. Tiefenbildkamera als Sensorik verwenden, als zu untersuchende Menge ausgewählt. Diese Teilmenge an SLAM-Verfahren wird hinsichtlich ihrer nichtfunktionalen Eigenschaften genauer untersucht und versucht in Komponenten zu unterteilen, welche in mehreren verschiedenen Implementierungen wiederverwendet werden können. Anhand der extrahierten Komponenten soll ein Featurebaum aufgebaut werden, der dem Anwender einen Überblick und die Möglichkeit bereitstellt SLAM-Verfahren nach speziellen Kriterien (Systemvoraussetzungen, Umgebungen, ...) zusammenzusetzen bzw. zur Laufzeit anzupassen. Dafür müssen die verfügbaren SLAM Implementierungen und dazugehörigen Dokumentationen in Bezug auf ihre Gemeinsamkeiten und Unterschiede analysiert werden.

Underwater navigation is a key technology for exploring the oceans and exploiting their resources. For autonomous underwater vehicles (AUVs) to explore the marine environment efficiently and securely, underwater simultaneous localization and mapping (SLAM) systems are often indispensable due to the lack of the global positioning system (GPS). In an underwater SLAM system, an AUV maps its surroundings and estimates its own pose at the same time. The pose of the AUV can be predicted by dead reckoning, but navigation errors accumulate over time. Therefore, sensors are needed to calibrate the state of the AUV. Among various sensors, the multibeam echosounder (MBES) is one of the most popular ones for underwater SLAM since it can acquire bathymetric point clouds with depth information of the surroundings. However, there are difficulties in data association for seabeds without distinct landmarks. Previous studies have focused more on traditional computer vision methods, which have limited performance on bathymetric data. In this thesis, a novel method based on deep learning is proposed to facilitate underwater perception. We conduct two experiments on place recognition and point cloud registration using data collected during a survey. The results show that, compared with the traditional methods, the proposed neural network is able to detect loop closures and register point clouds more efficiently. This work provides a better data association solution for designing underwater SLAM systems. / Undervattensnavigering är en viktig teknik för att utforska haven och utnyttja deras resurser. För att autonoma undervattensfordon (AUV) ska kunna utforska havsmiljön effektivt och säkert är underwater simultaneous localization and mapping (SLAM) system ofta oumbärliga på grund av bristen av det globala positioneringssystemet (GPS). I ett undervattens SLAM-system kartlägger ett AUV sin omgivning och uppskattar samtidigt sin egen position. AUV:s position kan förutsägas med hjälp av dödräkning, men navigeringsfel ackumuleras med tiden. Därför behövs sensorer för att kalibrera AUV:s tillstånd. Bland olika sensorer är multibeam ekolod (MBES) en av de mest populära för undervattens-SLAM eftersom den kan samla in batymetriska punktmoln med djupinformation om omgivningen. Det finns dock svårigheter med dataassociation för havsbottnar utan tydliga landmärken. Tidigare studier har fokuserat mer på traditionella datorvisionsmetoder som har begränsad prestanda för batymetriska data. I den här avhandlingen föreslås en ny metod baserad på djup inlärning för att underlätta undervattensuppfattning. Vi genomför två experiment på punktmolnregistrering med hjälp av data som samlats in under en undersökning. Resultaten visar att jämfört med de traditionella metoderna kan det föreslagna neurala nätverket upptäcka slingförslutningar och registrera punktmoln mer effektivt. Detta arbete ger en bättre lösning för dataassociation för utformning av undervattens SLAM-system.

<p>Simultaneous localization and mapping (SLAM) is a general device localization technique that uses realtime sensor measurements to develop a virtualization of the sensor's environment while also using this growing virtualization to determine the position and orientation of the sensor. This is useful for augmented reality (AR), in which a user looks through a head-mounted display (HMD) or viewfinder to see virtual components integrated into the real world. Visual SLAM (i.e., SLAM in which the sensor is an optical camera) is used in AR to determine the exact device/headset movement so that the virtual components can be accurately redrawn to the screen, matching the perceived motion of the world around the user as the user moves the device/headset. However, many potential AR applications may need access to more than device localization data in order to be useful; they may need to leverage environment data as well. Additionally, most SLAM solutions make the naive assumption that the environment surrounding the system is completely static (non-moving). Given these circumstances, it is clear that AR may benefit substantially from utilizing a SLAM solution that detects objects that move in the scene and ultimately provides localization data for each of these objects. This problem is known as the dynamic SLAM problem. Current attempts to address the dynamic SLAM problem often use machine learning to develop models that identify the parts of the camera image that belong to one of many classes of potentially-moving objects. The limitation with these approaches is that it is impractical to train models to identify every possible object that moves; additionally, some potentially-moving objects may be static in the scene, which these approaches often do not account for. Some other attempts to address the dynamic SLAM problem also localize the moving objects they detect, but these systems almost always rely on depth sensors or stereo camera configurations, which have significant limitations in real-world use cases. This dissertation presents a novel approach for registering and localizing unknown moving objects in the context of markerless, monocular, keyframe-based SLAM with no required prior information about object structure, appearance, or existence. This work also details a novel deep learning solution for determining SLAM map initialization suitability in structure-from-motion-based initialization approaches. This dissertation goes on to validate these approaches by implementing them in a markerless, monocular SLAM system called LUMO-SLAM, which is built from the ground up to demonstrate this approach to unknown moving object registration and localization. Results are collected for the LUMO-SLAM system, which address the accuracy of its camera localization estimates, the accuracy of its moving object localization estimates, and the consistency with which it registers moving objects in the scene. These results show that this solution to the dynamic SLAM problem, though it does not act as a practical solution for all use cases, has an ability to accurately register and localize unknown moving objects in such a way that makes it useful for some applications of AR without thwarting the system's ability to also perform accurate camera localization.</p>

The Simultaneous Localisation And Mapping (SLAM) problem is an open fundamental problem in autonomous mobile robotics. One of the latest most researched techniques used to enhance the SLAM methods is instance segmentation. In this thesis, we implement an instance segmentation system using Swin Transformer combined with two of the state of the art methods of instance segmentation namely Cascade Mask RCNN and Mask RCNN. Instance segmentation is a technique that simultaneously solves the problem of object detection and semantic segmentation. We show that depth information enhances the average precision (AP) by approximately 7%. We also show that the Swin Transformer backbone model can work well with depth images. Our results also show that Cascade Mask RCNN outperforms Mask RCNN. However, the results are to be considered due to the small size of the NYU-depth v2 dataset. Most of the instance segmentation researches use the COCO dataset which has a hundred times more images than the NYU-depth v2 dataset but it does not have the depth information of the image.

In the field of robotics and self-navigation, Simultaneous Localization and Mapping (SLAM) is a technique crucial for estimating poses while concurrently creating a map of the environment. Robotics applications often rely on various sensors for pose estimation, including cameras, inertial measurement units (IMUs), and more. Traditional discrete SLAM, utilizing stereo camera pairs and inertial measurement units, faces challenges such as time offsets between sensors. A solution to this issue is the utilization of continuous-time models for pose estimation. This thesis delves into the exploration and implementation of a continuous-time SLAM system, investigating the advantages of multi-modal sensor fusion over discrete stereo vision models. The findings indicate that incorporating an IMU into the system enhances pose estimation, providing greater robustness and accuracy compared to relying solely on visual SLAM. Furthermore, leveraging the continuous model's derivative and smoothness allows for decent pose estimation with fewer measurements, reducing the required quantity of measurements and computational resources.