Advanced Driver Assistance System (ADAS) was widely learned nowadays. As crucial parts of ADAS, lane markings detection, as well as other objects detection, have become more popular than before. However, most methods implemented in such areas cannot perfectly balance the performance of accuracy versus efficiency, and the mainstream methods (e.g. Machine Learning) suffer from several limitations which can hardly break the wall between partial autonomous and fully autonomous driving. This thesis proposed a real-time lane marking detection framework for ADAS, which included 4-extreme points set descriptor and a rule-based cascade classifier. By analyzing the behavior of lane markings on the road surface, a characteristic of markings was discovered, i.e., standard markings can sustain their shape in the perpendicular plane of the driving direction. By employing this feature, a 4-extreme points set descriptor was applied to describe the shape of each marking first. Specifically, after processing Maximally Stable Extremal Region (MSER) and Hough transforms on a 2-D image, several contours of interest are obtained. A bounding box, with borders parallel to the image coordinate, intersected with each contour at 4 points in the edge, which was named 4-extreme points set. Afterward, to verify consistency of each contour and standard marking, some rules abstracted from construction manual are employed such as Area Filter, Colour Filter, Relative Location Filter, Convex Filter, etc. To reduce the errors caused by changes in driving direction, an enhanced module was then introduced. By tracking the vanishing point as well as other key points of the road net, a method for 3-D reconstruction, with respect to the optical axis between vanishing point and camera center, is possible. The principle of such algorithm was exhibited, and a description about how to obtain the depth information from this model was also provided. Among all of these processes, a key-point based classification method is the main contribution of this paper because of its function in eliminating the deformation of the object caused by inverse perspective mapping. Several experiments were conducted in highway and urban roads in Ottawa. The detection rate of the markings by the proposed algorithm reached an average accuracy rate of 96.77% while F1 Score (harmonic mean of precision and recall) also attained a rate of 90.57%. In summary, the proposed method exhibited a state-of-the-art performance and represents a significant advancement of understanding.

<p dir="ltr">Road safety analysis typically relies on the correlation between road surface conditions, lane marking status, or lane width and crash data. Traditionally, this data is surveyed in the field after road construction or car accidents, which is labor-intensive, time-consuming, and hazardous. With the development of mobile mapping systems (MMS) in recent years, the ability to collect lane marking retroreflectivity or lane width information has been greatly improved. By utilizing Light Detection and Ranging (LiDAR) point clouds and RGB images captured by MMS, it is possible to establish lane marking inventory that includes the conditions of pavement markers (such as lane marking retroreflectivity and lane width) for road safety analysis.</p><p dir="ltr">This dissertation aims to develop a comprehensive framework to extract lane markings and report their characteristics using MMS datasets for transportation safety. The proposed approaches include geometric/morphological and deep learning-based approaches based on the LiDAR point clouds acquired by MMS. A normalization strategy is developed to ensure consistent intensity values across laser beams/LiDAR units/MMS for the same objects, thereby enhancing the lane marking extraction. In addition, an image-aided LiDAR approach is proposed to improve the extraction process further. Following the extraction, lane marking classification and characterization, including intensity profile generation and lane width estimation, are conducted to establish comprehensive lane marking inventory.</p><p dir="ltr">To evaluate the proposed strategies, lane marking extraction with and without intensity normalization is also conducted. The results show that the proposed intensity normalization significantly improves the performance of lane marking extraction, regardless of the approach or data used. The geometric approach using normalized intensity achieves F1-scores higher than 90%, outperforming the learning-based models. Furthermore, the intensity derived from two different MMS is compared for performance evaluation, and the agreement of normalized intensity values is within a range of 3.1 to 3.8 (the used MMS provide intensity as an integer number within 0 to 255). Through the normalization, a positive linear relationship between LiDAR normalized intensity and retroreflectivity is found, with the strongest relationship providing an R² of 0.72 and a Pearson's correlation coefficient of 0.85. A comparison of the correlation between original/normalized intensity and retroreflectivity revealed a stronger correlation between original intensity and retroreflectivity. For image-aided LiDAR approach, the image information indeed enhanced the LiDAR-based lane marking extraction approach, as evidenced by the highest F1-score (92.5%) of the image-aided LiDAR approach, outperforming the LiDAR-based (90.3%) and image-based (77.8%) ones. Specifically, the recall increases by 4.0% – from 87.6% (LiDAR-based) to 91.6% (image-aided LiDAR) – surpasses the slight improvement in the precision of 0.2% – from 93.2% (LiDAR-based) to 93.4% (image-aided LiDAR).</p><p dir="ltr">Finally, a Potree-based web portal is developed to visualize the results derived through the proposed lane marking extraction/classification/characterization strategies. This portal includes a function that enables the projection between 2D images and 3D point clouds, as well as tools for displaying intensity profiles and lane width estimates. It is capable of rendering a large dataset of {approximately 4.2 billion LiDAR points} in around ten seconds and allows for the visualization of intensity profiles and lane width estimates. Users can select points of interest in an intensity profile/lane width plot. This selection will result in the corresponding point being showcased in the LiDAR data on the web portal. Furthermore, the LiDAR point can be projected onto the corresponding image.</p><p dir="ltr">The above proposed strategies facilitate the investigation of the relationship between LiDAR intensity and mobile retroreflectivity. To ensure quality control, lane markings derived from geometric and learning-based extraction approaches were compared. These strategies were evaluated using two MMS (equipped with multiple imaging and LiDAR sensors), covering extensive road segments exceeding 400 miles. Furthermore, a reporting mechanism based on multi-modal data from various MMS sensors was implemented to visualize the results derived from the proposed strategies and to serve as a quality control tool. Consequently, the proposed strategies are easily adaptable for different MMS or the regular updating of lane marking inventory.</p>

Les applications pour véhicules autonomes et les systèmes d’aide avancée à la conduite (Advanced Driving Assistance Systems - ADAS) mettent en oeuvre des processus permettant à des systèmes haut niveau de réaliser une prise de décision. Pour de tels systèmes, la connaissance du positionnement précis (ou localisation) du véhicule dans son environnement est un pré-requis nécessaire. Cette thèse s’intéresse à la détection de la structure de scène, au processus de localisation ainsi qu’à la modélisation d’erreurs. A partir d’un large spectre fonctionnel de systèmes de vision, de l’accessibilité d’un système de cartographie ouvert (Open Geographical Information Systems - GIS) et de la large diffusion des systèmes de positionnement dans les véhicules (Global Positioning System - GPS), cette thèse étudie la performance et la fiabilité d’une méthode de localisation utilisant ces différentes sources. La détection de marquage sur la route réalisée par caméra monoculaire est le point de départ permettant de connaître la structure de la scène. En utilisant, une détection multi-noyau avec pondération hiérarchique, la méthode paramétrique proposée effectue la détection et le suivi des marquages sur la voie du véhicule en temps réel. La confiance en cette source d’information a été quantifiée par un indicateur de vraisemblance. Nous proposons ensuite un système de localisation qui fusionne des informations de positionnement (GPS), la carte (GIS) et les marquages détectés précédemment dans un cadre probabiliste basé sur un filtre particulaire. Pour ce faire, nous proposons d’utiliser les marquages détectés non seulement dans l’étape de mise en correspondance des cartes mais aussi dans la modélisation de la trajectoire attendue du véhicule. La fiabilité du système de localisation, en présence d’erreurs inhabituelles dans les différentes sources d’information, est améliorée par la prise en compte de différents indicateurs de confiance. Ce mécanisme est par la suite utilisé pour identifier les sources d’erreur. Cette thèse se conclut par une validation expérimentale des méthodes proposées dans des situations réelles de conduite. Leurs performances ont été quantifiées en utilisant un véhicule expérimental et des données en libre accès sur internet. / Autonomous Vehicles (AV) applications and Advanced Driving Assistance Systems (ADAS) relay in scene understanding processes allowing high level systems to carry out decision marking. For such systems, the localization of a vehicle evolving in a structured dynamic environment constitutes a complex problem of crucial importance. Our research addresses scene structure detection, localization and error modeling. Taking into account the large functional spectrum of vision systems, the accessibility of Open Geographical Information Systems (GIS) and the widely presence of Global Positioning Systems (GPS) onboard vehicles, we study the performance and the reliability of a vehicle localization method combining such information sources. Monocular vision–based lane marking detection provides key information about the scene structure. Using an enhanced multi-kernel framework with hierarchical weights, the proposed parametric method performs, in real time, the detection and tracking of the ego-lane marking. A self-assessment indicator quantifies the confidence of this information source. We conduct our investigations in a localization system which tightly couples GPS, GIS and lane makings in the probabilistic framework of Particle Filter (PF). To this end, it is proposed the use of lane markings not only during the map-matching process but also to model the expected ego-vehicle motion. The reliability of the localization system, in presence of unusual errors from the different information sources, is enhanced by taking into account different confidence indicators. Such a mechanism is later employed to identify error sources. This research concludes with an experimental validation in real driving situations of the proposed methods. They were tested and its performance was quantified using an experimental vehicle and publicly available datasets.

Curbs and road markings were designed to provide a visual low-level spatial perception of road environments. In this sense, a perception system capable of detecting those road features is of utmost importance for an autonomous vehicle. In vision-based approaches, few works have been developed for curb detection, and most of the advances on road marking detection have aimed lane markings only. Therefore, to detect all these road features, multiple algorithms running simultaneously would be necessary. Alternatively, as the main contribution of this work, it was proposed to employ an architecture of Fully Convolutional Neural Network (FCNN), denominated as 3CSeg-Multinet, to detect curbs and road markings in a single inference. Since there was no labeled dataset available for training and validation, a new one was generated with Brazilian urban scenes, and they were manually labeled. By visually analyzing experimental results, the proposed approach has shown to be effective and robust against most of the clutter present on images, running at around 10 fps in a Graphics Processing Unit (GPU). Moreover, with the intention of granting spatial perception, stereo vision techniques were used to project the detected road features in a point cloud. Finally, as a way to validate the applicability of the proposed perception system on a vehicle, it was also introduced a vision-based metric localization model for the urban scenario. In an experiment, compared to the ground truth, this localization method has revealed consistency on its pose estimations in a map generated by LIDAR. / Guias e sinalizações horizontais foram projetados para fornecer a percepção visual de baixo nível do espaço das vias urbanas. Deste modo, seria de extrema importância para um veículo autônomo ter um sistema de percepção capaz de detectar tais características visuais. Em abordagens baseadas em visão, poucos trabalhos foram desenvolvidos para detecção de guias, e a maioria dos avanços em detecção de sinalizações horizontais foi focada na detecção de faixas apenas. Portanto, para que fosse possível detectar todas essas características visuais, seria necessário executar diversos algoritmos simultaneamente. Alternativamente, como sendo a principal contribuição deste trabalho, foi proposto a adoção de uma Rede Neural Totalmente Convolutiva, denominado 3CSeg-Multinet, para detectar guias e sinalizações horizontais em apenas uma inferência. Como não havia um conjunto de dados rotulados disponível para treinar e validar a rede, foi gerado um novo conjunto com imagens capturadas em ambiente urbano brasileiro, e foi realizado a rotulação manual. Através de uma análise visual dos resultados experimentais obtidos, o método proposto mostrou-se eficaz e robusto contra a maioria dos fatores que causam confusão nas imagens, executando a aproximadamente 10 fps em uma GPU. Ainda, com o intuito de garantir a percepção espacial, foram usados métodos de visão estéreo para projetar as características detectadas em núvem de pontos. Finalmente, foi apresentado também um modelo de localização métrica baseado em visão para validar a aplicabilidade do sistema de percepção proposto em um veículo. Em um experimento, este método de localização revelou-se capaz de manter as estimativas consistentes com a verdadeira pose do veículo em um mapa gerado a partir de um sensor LIDAR.

<p>A complete prototype system for measuring vehicle lateral position has been set up during the course of this master’s thesis project. In the development of the software, images acquired from a back-ward looking video camera mounted on the roof of the vehicle were used. </p><p>The problem of using computer vision to measure lateral position can be divided into road marking detection and lateral position extraction. Since the strongest characteristic of a road marking image are the edges of the road markings, the road marking detection step is based on edge detection. For the detection of the straight edge lines a Hough based method was chosen. Due to peak spreading in Hough space, the difficulty of detecting the correct peak in Hough space was encountered. A flexible Hough peak detection algorithm was developed based on an adaptive window that takes peak spreading into account. The road marking candidate found by the system is verified before the lateral position data is generated. A good performance of the road marking tracking algorithm was obtained by exploiting temporal correlation to update a search region within the image. A camera calibration made the extraction of real-world lateral position information and yaw angle data possible. </p><p>This vision-based method proved to be very accurate. The standard deviation of the error in the position detection is 0.012 m within an operating range of ±2 m from the image centre. During continuous road markings the rate of valid data is on average 96 %, whereas it drops to around 56 % for sections with intermittent road markings. The system performs well during lane change manoeuvres, which is an indication that the system tracks the correct road marking. This prototype system is a robust and automatic measurement system, which will benefit VTI in its many driving behaviour research programs.</p>

A complete prototype system for measuring vehicle lateral position has been set up during the course of this master’s thesis project. In the development of the software, images acquired from a back-ward looking video camera mounted on the roof of the vehicle were used. The problem of using computer vision to measure lateral position can be divided into road marking detection and lateral position extraction. Since the strongest characteristic of a road marking image are the edges of the road markings, the road marking detection step is based on edge detection. For the detection of the straight edge lines a Hough based method was chosen. Due to peak spreading in Hough space, the difficulty of detecting the correct peak in Hough space was encountered. A flexible Hough peak detection algorithm was developed based on an adaptive window that takes peak spreading into account. The road marking candidate found by the system is verified before the lateral position data is generated. A good performance of the road marking tracking algorithm was obtained by exploiting temporal correlation to update a search region within the image. A camera calibration made the extraction of real-world lateral position information and yaw angle data possible. This vision-based method proved to be very accurate. The standard deviation of the error in the position detection is 0.012 m within an operating range of ±2 m from the image centre. During continuous road markings the rate of valid data is on average 96 %, whereas it drops to around 56 % for sections with intermittent road markings. The system performs well during lane change manoeuvres, which is an indication that the system tracks the correct road marking. This prototype system is a robust and automatic measurement system, which will benefit VTI in its many driving behaviour research programs.

Localization is one of the fundamental components of autonomous vehicles by enabling tasks as overtaking, lane keeping and self-navigation. Urban canyons and bad weather interfere with the reception of GPS satellite signal which prohibits the exclusive use of such technology for vehicle localization in urban places. Alternatively, map-aided localization methods have been employed to enable position estimation without the dependence on GPS devices. In this solution, the vehicle position is given as the place that best matches the sensor measurement to the environment map. Before building the maps, feature sof the environment must be extracted from sensor measurements. In vehicle localization, curbs and road markings have been extensively employed as mapping features. However, most of the urban mapping methods rely on a street free of obstacles or require repetitive measurements of the same place to avoid occlusions. The construction of an accurate representation of the environment is necessary for a proper match of sensor measurements to the map during localization. To prevent the necessity of a manual process to remove occluding obstacles and unobserved areas, a vehicle localization method that supports maps built from partial observations of the environment is proposed. In this localization system,maps are formed by curb and road markings extracted from multilayer laser sensor measurements. Curb structures are detected even in the presence of vehicles that occlude the roadsides, thanks to the use of robust regression. Road markings detector employs Otsu thresholding to analyze infrared remittance data which makes the method insensitive to illumination. Detected road features are stored in two map representations: occupancy grid map (OGM) and Gaussian process occupancy map (GPOM). The first approach is a popular map structure that represents the environment through fine-grained grids. The second approach is a continuous representation that can estimate the occupancy of unseen areas. The Monte Carlo localization (MCL) method was adapted to support the obtained maps of the urban environment. In this sense, vehicle localization was tested in an MCL that supports OGM and an MCL that supports GPOM. Precisely, for MCL based on GPOM, a new measurement likelihood based on multivariate normal probability density function is formulated. Experiments were performed in real urban environments. Maps were built using sparse laser data to verify there ronstruction of non-observed areas. The localization system was evaluated by comparing the results with a high precision GPS device. Results were also compared with localization based on OGM. / No contexto de veículos autônomos, a localização é um dos componentes fundamentais, pois possibilita tarefas como ultrapassagem, direção assistida e navegação autônoma. A presença de edifícios e o mau tempo interferem na recepção do sinal de GPS que consequentemente dificulta o uso de tal tecnologia para a localização de veículos dentro das cidades. Alternativamente, a localização com suporte aos mapas vem sendo empregada para estimar a posição sem a dependência do GPS. Nesta solução, a posição do veículo é dada pela região em que ocorre a melhor correspondência entre o mapa do ambiente e a leitura do sensor. Antes da criação dos mapas, características dos ambientes devem ser extraídas a partir das leituras dos sensores. Dessa forma, guias e sinalizações horizontais têm sido largamente utilizados para o mapeamento. Entretanto, métodos de mapeamento urbano geralmente necessitam de repetidas leituras do mesmo lugar para compensar as oclusões. A construção de representações precisas dos ambientes é essencial para uma adequada associação dos dados dos sensores como mapa durante a localização. De forma a evitar a necessidade de um processo manual para remover obstáculos que causam oclusão e áreas não observadas, propõe-se um método de localização de veículos com suporte aos mapas construídos a partir de observações parciais do ambiente. No sistema de localização proposto, os mapas são construídos a partir de guias e sinalizações horizontais extraídas a partir de leituras de um sensor multicamadas. As guias podem ser detectadas mesmo na presença de veículos que obstruem a percepção das ruas, por meio do uso de regressão robusta. Na detecção de sinalizações horizontais é empregado o método de limiarização por Otsu que analisa dados de reflexão infravermelho, o que torna o método insensível à variação de luminosidade. Dois tipos de mapas são empregados para a representação das guias e das sinalizações horizontais: mapa de grade de ocupação (OGM) e mapa de ocupação por processo Gaussiano (GPOM). O OGM é uma estrutura que representa o ambiente por meio de uma grade reticulada. OGPOM é uma representação contínua que possibilita a estimação de áreas não observadas. O método de localização por Monte Carlo (MCL) foi adaptado para suportar os mapas construídos. Dessa forma, a localização de veículos foi testada em MCL com suporte ao OGM e MCL com suporte ao GPOM. No caso do MCL baseado em GPOM, um novo modelo de verossimilhança baseado em função densidade probabilidade de distribuição multi-normal é proposto. Experimentos foram realizados em ambientes urbanos reais. Mapas do ambiente foram gerados a partir de dados de laser esparsos de forma a verificar a reconstrução de áreas não observadas. O sistema de localização foi avaliado por meio da comparação das posições estimadas comum GPS de alta precisão. Comparou-se também o MCL baseado em OGM com o MCL baseado em GPOM, de forma a verificar qual abordagem apresenta melhores resultados.

Localization is one of the fundamental components of autonomous vehicles by enabling tasks as overtaking, lane keeping and self-navigation. Urban canyons and bad weather interfere with the reception of GPS satellite signal which prohibits the exclusive use of such technology for vehicle localization in urban places. Alternatively, map-aided localization methods have been employed to enable position estimation without the dependence on GPS devices. In this solution, the vehicle position is given as the place that best matches the sensor measurement to the environment map. Before building the maps, feature sof the environment must be extracted from sensor measurements. In vehicle localization, curbs and road markings have been extensively employed as mapping features. However, most of the urban mapping methods rely on a street free of obstacles or require repetitive measurements of the same place to avoid occlusions. The construction of an accurate representation of the environment is necessary for a proper match of sensor measurements to the map during localization. To prevent the necessity of a manual process to remove occluding obstacles and unobserved areas, a vehicle localization method that supports maps built from partial observations of the environment is proposed. In this localization system,maps are formed by curb and road markings extracted from multilayer laser sensor measurements. Curb structures are detected even in the presence of vehicles that occlude the roadsides, thanks to the use of robust regression. Road markings detector employs Otsu thresholding to analyze infrared remittance data which makes the method insensitive to illumination. Detected road features are stored in two map representations: occupancy grid map (OGM) and Gaussian process occupancy map (GPOM). The first approach is a popular map structure that represents the environment through fine-grained grids. The second approach is a continuous representation that can estimate the occupancy of unseen areas. The Monte Carlo localization (MCL) method was adapted to support the obtained maps of the urban environment. In this sense, vehicle localization was tested in an MCL that supports OGM and an MCL that supports GPOM. Precisely, for MCL based on GPOM, a new measurement likelihood based on multivariate normal probability density function is formulated. Experiments were performed in real urban environments. Maps were built using sparse laser data to verify there ronstruction of non-observed areas. The localization system was evaluated by comparing the results with a high precision GPS device. Results were also compared with localization based on OGM. / No contexto de veículos autônomos, a localização é um dos componentes fundamentais, pois possibilita tarefas como ultrapassagem, direção assistida e navegação autônoma. A presença de edifícios e o mau tempo interferem na recepção do sinal de GPS que consequentemente dificulta o uso de tal tecnologia para a localização de veículos dentro das cidades. Alternativamente, a localização com suporte aos mapas vem sendo empregada para estimar a posição sem a dependência do GPS. Nesta solução, a posição do veículo é dada pela região em que ocorre a melhor correspondência entre o mapa do ambiente e a leitura do sensor. Antes da criação dos mapas, características dos ambientes devem ser extraídas a partir das leituras dos sensores. Dessa forma, guias e sinalizações horizontais têm sido largamente utilizados para o mapeamento. Entretanto, métodos de mapeamento urbano geralmente necessitam de repetidas leituras do mesmo lugar para compensar as oclusões. A construção de representações precisas dos ambientes é essencial para uma adequada associação dos dados dos sensores como mapa durante a localização. De forma a evitar a necessidade de um processo manual para remover obstáculos que causam oclusão e áreas não observadas, propõe-se um método de localização de veículos com suporte aos mapas construídos a partir de observações parciais do ambiente. No sistema de localização proposto, os mapas são construídos a partir de guias e sinalizações horizontais extraídas a partir de leituras de um sensor multicamadas. As guias podem ser detectadas mesmo na presença de veículos que obstruem a percepção das ruas, por meio do uso de regressão robusta. Na detecção de sinalizações horizontais é empregado o método de limiarização por Otsu que analisa dados de reflexão infravermelho, o que torna o método insensível à variação de luminosidade. Dois tipos de mapas são empregados para a representação das guias e das sinalizações horizontais: mapa de grade de ocupação (OGM) e mapa de ocupação por processo Gaussiano (GPOM). O OGM é uma estrutura que representa o ambiente por meio de uma grade reticulada. OGPOM é uma representação contínua que possibilita a estimação de áreas não observadas. O método de localização por Monte Carlo (MCL) foi adaptado para suportar os mapas construídos. Dessa forma, a localização de veículos foi testada em MCL com suporte ao OGM e MCL com suporte ao GPOM. No caso do MCL baseado em GPOM, um novo modelo de verossimilhança baseado em função densidade probabilidade de distribuição multi-normal é proposto. Experimentos foram realizados em ambientes urbanos reais. Mapas do ambiente foram gerados a partir de dados de laser esparsos de forma a verificar a reconstrução de áreas não observadas. O sistema de localização foi avaliado por meio da comparação das posições estimadas comum GPS de alta precisão. Comparou-se também o MCL baseado em OGM com o MCL baseado em GPOM, de forma a verificar qual abordagem apresenta melhores resultados.