With the advancement of autonomous driving research, 3D detection based on LiDAR point cloud has gradually become one of the top research topics in the field of artificial intelligence. Compared with RGB cameras, LiDAR point cloud can provide depth information, while RGB images can provide denser resolution. Features from LiDAR and cameras are considered to be complementary. However, due to the sparsity of the LiDAR point clouds, a dense and accurate RGB/3D projective relationship is difficult to establish especially for distant scene points. Recent works try to solve this problem by designing a network that learns missing points or dense point density distribution to compensate for the sparsity of the LiDAR point cloud. During the master’s exploration, we consider addressing this problem from two aspects. The first is to design a GAN(Generative Adversarial Network)-based module to reconstruct point clouds, and the second is to apply regional point cloud enhancement based on motion maps. In the first aspect, we propose to use an imagine-and-locate process, called UYI. The objective of this module is to improve the point cloud quality and is independent of the detection stage used for inference. We accomplish this task through a GAN-based cross-modality module that uses image as input to infer a dense LiDAR shape. In another aspect, inspired by the attention mechanism of human eyes, we use motion maps to perform random augmentation on point clouds in a targeted manner named motion map-assisted enhancement, MAE. Boosted by our UYI and MAE module, our experiments show a significant performance improvement in all tested baseline models. In fact, benefiting from the plug-and-play characteristics of our module, we were able to push the performance of the existing state-of-the-art model to a new height. Our method not only has made great progress in the detection performance of vehicle objects but also achieved an even bigger leap forward in the pedestrian category. In future research, we will continue to explore the feasibility of spatio-temporal correlation methods in 3D detection, and 3D detection related to motion information extraction could be a promising direction.

Sensor Fusion for 3D data is a popular topic. Multisensor data combination enhance the qualities of each other while single sensor lacks accuracy. In this thesis, an evaluation platform for Multimodal 3D data from Kinect v2 and Microphone Array is designed and implemented by using ReactJS. In automotive industry and computer vision area, 3D detection and localization are widely used. Solutions of 3D detection and localization using different measurement systems are discussed in a large number of papers. Data Fusion systems are normally using ultrasound based, radio waves based, Time-of-Flight, structured light, stereo cameras and sound based sensors. All of these measurement systems might provide different 3D data models. And each system works fine separately. However, in some cases, multiple measurement systems need to work together. Their 3D data sets are different and could not be compared and combined directly. In order to simplify the design process of multiple measurement systems, this web based evaluation platform is focused on comparison and combination of 3D data sets from different coordinate systems. It provides a quick and easy development method between multiple measurement systems. In this thesis, an evaluation platform which based on Kinect v2 body detection and microphone array sound detection systems will be discussed. First an introduction about project overview is given. The second section of this paper deals with several project related technologies. The third section provides the concept of this project. The forth section describes development and implement detail. The next section is about data visualization and statistical analysis. Further the final results, evaluation and discussion are given.

L'intérêt toujours grandissant pour les données cartographiques fiables, notamment en milieu urbain, a motivé le développement de systèmes de cartographie mobiles terrestres. Ces systèmes sont conçus pour l'acquisition de données de très haute précision, telles que des nuages de points LiDAR 3D et des images optiques. La multitude de données, ainsi que leur diversité, rendent complexe le traitement des données issues de ce type de systèmes. Cette thèse se place dans le contexte du traitement de l'image appliqué au nuages de points LiDAR 3D issus de ce type de système.Premièrement, nous nous intéressons à des images issues de la projection de nuages de points LiDAR dans des grilles de pixels 2D régulières. Ces projections créent généralement des images éparses, dans lesquelles l'information de certains pixels n'est pas connue. Nous proposons alors différentes méthodes pour des applications telles que la génération d'orthoimages haute résolution, l'imagerie RGB-D et l'estimation de la visibilité des points d'un nuage.De plus, nous proposons d'exploiter la topologie d'acquisition des capteurs LiDAR pour produire des images de faible résolution: les range-images. Ces images offrent une représentation efficace et canonique du nuage de points, tout en étant directement accessibles à partir du nuage de points. Nous montrons comment ces images peuvent être utilisées pour simplifier, voire améliorer, des méthodes pour le recalage multi-modal, la segmentation, la désoccultation et la détection 3D. / The ever growing demand for reliable mapping data, especially in urban environments, has motivated the development of "close-range" Mobile Mapping Systems (MMS). These systems acquire high precision data, and in particular 3D LiDAR point clouds and optical images. The large amount of data, along with their diversity, make MMS data processing a very complex task. This thesis lies in the context of 2D image processing applied to 3D LiDAR point clouds acquired with MMS.First, we focus on the projection of the LiDAR point clouds onto 2D pixel grids to create images. Such projections are often sparse because some pixels do not carry any information. We use these projections for different applications such as high resolution orthoimage generation, RGB-D imaging and visibility estimation in point clouds.Moreover, we exploit the topology of LiDAR sensors in order to create low resolution images, named range-images. These images offer an efficient and canonical representation of the point cloud, while being directly accessible from the point cloud. We show how range-images can be used to simplify, and sometimes outperform, methods for multi-modal registration, segmentation, desocclusion and 3D detection.