During the last decades, great steps have been taken to decrease passenger fatality in cars. Systems such as ABS and airbags have been developed for this purpose alone. But not much effort has been put into pedestrian safety. In traffic today, pedestrians are one of the most endangered participants and in recent years, there has been an increased demand for pedestrian safety from the European Enhanced Vehicle safety Committee and the European New Car Assessment Programme has thereby developed tests where pedestrian safety is rated. With this, detection of pedestrians has arised as a part in the automotive safety research. This thesis provides some of this research available in the area and a brief introduction to some of the sensors readily available. The objective of this work is to detect pedestrians in front of a vehicle by using thermoelectric infrared sensors fused with short range radar sensors and also to minimize any missed detections or false alarms. There has already been extensive work performed with the thermoelectric infrared sensors for this sole purpose and this thesis is based on that work. Information is provided about the sensors used and an explanation of how they are set up during this work. Methods used for classifying objects are given and the assumptions made about pedestrians in this system. A basic tracking algorithm is used to track radar detected objects in order to provide the fusion system with better data. The approach chosen for the sensor fusion is a central-level fusion where the probabilities for a pedestrian from the radars and the thermoelectric infrared sensors are combined using Dempster-Shafer Theory and accumulated over time in the Occupancy Grid framework. Theories that are extensively used in this thesis are explained in detail and discussed accordingly in different chapters. Finally the experiments undertaken and the results attained from the presented system are shown. A comparison is made with the previous detection system, which only uses thermoelectric infrared sensors and of which this work continues on. Conclusions regarding what this system is capable of are drawn with its inherent strengths and weaknesses.

<p>During the last decades, great steps have been taken to decrease passenger fatality in cars. Systems such as ABS and airbags have been developed for this purpose alone. But not much effort has been put into pedestrian safety. In traffic today, pedestrians are one of the most endangered participants and in recent years, there has been an increased demand for pedestrian safety from the European Enhanced Vehicle safety Committee and the European New Car Assessment Programme has thereby developed tests where pedestrian safety is rated. With this, detection of pedestrians has arised as a part in the automotive safety research.</p><p>This thesis provides some of this research available in the area and a brief introduction to some of the sensors readily available. The objective of this work is to detect pedestrians in front of a vehicle by using thermoelectric infrared sensors fused with short range radar sensors and also to minimize any missed detections or false alarms. There has already been extensive work performed with the thermoelectric infrared sensors for this sole purpose and this thesis is based on that work.</p><p>Information is provided about the sensors used and an explanation of how they are set up during this work. Methods used for classifying objects are given and the assumptions made about pedestrians in this system. A basic tracking algorithm is used to track radar detected objects in order to provide the fusion system with better data. The approach chosen for the sensor fusion is a central-level fusion where the probabilities for a pedestrian from the radars and the thermoelectric infrared sensors are combined using Dempster-Shafer Theory and accumulated over time in the Occupancy Grid framework. Theories that are extensively used in this thesis are explained in detail and discussed accordingly in different chapters.</p><p>Finally the experiments undertaken and the results attained from the presented system are shown. A comparison is made with the previous detection system, which only uses thermoelectric infrared sensors and of which this work continues on. Conclusions regarding what this system is capable of are drawn with its inherent strengths and weaknesses.</p>

Many automotive safety applications in modern cars make use of cameras and object detection to analyze the surrounding environment. Pedestrians, animals and other vehicles can be detected and safety actions can be taken before dangerous situations arise. To detect occurrences of the different objects, these systems are traditionally trained to learn a classification model using a set of images that carry labels corresponding to their content. To obtain high performance with a variety of object appearances, the required amount of data is very large. Acquiring unlabeled images is easy, while the manual work of labeling is both time-consuming and costly. Semi-supervised learning refers to methods that utilize both labeled and unlabeled data, a situation that is highly desirable if it can lead to improved accuracy and at the same time alleviate the demand of labeled data. This has been an active area of research in the last few decades, but few studies have investigated the performance of these algorithms in larger systems. In this thesis, we investigate if and how semi-supervised learning can be used in a large-scale pedestrian detection system. With the area of application being automotive safety, where real-time performance is of high importance, the work is focused around boosting classifiers. Results are presented on a few publicly available UCI data sets and on a large data set for pedestrian detection captured in real-life traffic situations. By evaluating the algorithms on the pedestrian data set, we add the complexity of data set size, a large variety of object appearances and high input dimension. It is possible to find situations in low dimensions where an additional set of unlabeled data can be used successfully to improve a classification model, but the results show that it is hard to efficiently utilize semi-supervised learning in large-scale object detection systems. The results are hard to scale to large data sets of higher dimensions as pair-wise computations are of high complexity and proper similarity measures are hard to find.

La démocratisation de la « vidéosurveillance intelligente » nécessite le développement d’outils automatiques et temps réel d’analyse vidéo. Parmi ceux-ci, la détection de piétons joue un rôle majeur car de nombreux systèmes reposent sur cette technologie. Les approches classiques de détection de piétons utilisent la reconnaissance de formes et l’apprentissage statistique. Elles souffrent donc d’une dégradation des performances quand l’apparence des piétons ou des éléments de la scène est trop différente de celle étudiée lors de l’apprentissage. Pour y remédier, une solution appelée « contextualisation du détecteur » est étudiée lorsque la caméra est fixe. L’idée est d’enrichir le système à l’aide d’informations provenant de la scène afin de l’adapter aux situations qu’il risque de fréquemment rencontrer. Ce travail a été réalisé en deux temps. Tout d’abord, l’architecture d’un détecteur et les différents outils utiles à sa construction sont présentés dans un état de l’art. Puis la problématique de la contextualisation est abordée au travers de diverses expériences validant ou non les pistes d’amélioration envisagées. L’objectif est d’identifier toutes les briques du système pouvant bénéficier de cet apport afin de contextualiser complètement le détecteur. Pour faciliter l’exploitation d’un tel système, la contextualisation a été entièrement automatisée et s’appuie sur des algorithmes d’apprentissage semi-supervisé. Une première phase consiste à collecter le maximum d’informations sur la scène. Différents oracles sont proposés afin d’extraire l’apparence des piétons et des éléments du fond pour former une base d’apprentissage dite contextualisée. La géométrie de la scène, influant sur la taille et l’orientation des piétons, peut ensuite être analysée pour définir des régions, dans lesquelles les piétons, tout comme le fond, restent visuellement proches. Dans la deuxième phase, toutes ces connaissances sont intégrées dans le détecteur. Pour chaque région, un classifieur est construit à l’aide de la base contextualisée et fonctionne indépendamment des autres. Ainsi chaque classifieur est entraîné avec des données ayant la même apparence que les piétons qu’il devra détecter. Cela simplifie le problème de l’apprentissage et augmente significativement les performances du système. / With the rise of videosurveillance systems comes a logical need for automatic and real-time processes to analyze the huge amount of generated data. Among these tools, pedestrian detection algorithms are essential, because in videosurveillance locating people is often the first step leading to more complex behavioral analyses. Classical pedestrian detection approaches are based on machine learning and pattern recognition algorithms. Thus they generally underperform when the pedestrians’ appearance observed by a camera tends to differ too much from the one in the generic training dataset. This thesis studies the concept of the contextualization of such a detector. This consists in introducing scene information into a generic pedestrian detector. The main objective is to adapt it to the most frequent situations and so to improve its overall performances. The key hypothesis made here is that the camera is static, which is common in videosurveillance scenarios.This work is split into two parts. First a state of the art introduces the architecture of a pedestrian detector and the different algorithms involved in its building. Then the problem of the contextualization is tackled and a series of experiments validates or not the explored leads. The goal is to identify every part of the detector which can benefit from the approach in order to fully contextualize it. To make the contextualization process easier, our method is completely automatic and is based on semi-supervised learning methods. First of all, data coming from the scene are gathered. We propose different oracles to detect some pedestrians in order to catch their appearance and to form a contextualized training dataset. Then, we analyze the scene geometry, which influences the size and the orientation of the pedestrians and we divide the scene into different regions. In each region, pedestrians as well as background elements share a similar appearance.In the second step, all this information is used to build the final detector which is composed of several classifiers, one by region. Each classifier independently scans its dedicated piece of image. Thus, it is only trained with a region-specific contextualized dataset, containing less appearance variability than a global one. Consequently, the training stage is easier and the overall detection results on the scene are improved.

The purpose of this thesis was to determine which perceptual cues drivers use to identify pedestrians that may constitute a risk in traffic. Methods chosen were recordings of pedestrian behaviour in Linköping by means of a stationary video camera as well as video camera mounted in a car. Interviews on the recordings from the mobile camera were conducted with taxi drivers and driving instructors. Results include that drivers not only react to pedestrians they believe will behave in a dangerous way, but also react to pedestrians that probably not will behave in such a way, but where the possibility still exists. The study concluded that it was not possible to determine how risky a pedestrian is considered to be by only using behavioural factors such as trajectory or position on the sidewalk, and distance. It is necessary also to include environmental factors, mainly where the pedestrian and car are positioned in relation to the side of the road, so that the behaviour of the pedestrian can be interpreted.

Pedestrian detection is an important field with applications in active safety systems for cars as well as autonomous driving. Since autonomous driving and active safety are becoming technically feasible now the interest for these applications has dramatically increased.The aim of this thesis is to investigate convolutional neural networks (CNN) for pedestrian detection. The reason for this is that CNN have recently beensuccessfully applied to several different computer vision problems. The main applications of pedestrian detection are in real time systems. For this reason,this thesis investigates strategies for reducing the computational complexity offorward propagation for CNN.The approach used in this thesis for extracting pedestrians is to use a CNN tofind a probability map of where pedestrians are located. From this probabilitymap bounding boxes for pedestrians are generated. A method for handling scale invariance for the objects of interest has also been developed in this thesis. Experiments show that using this method givessignificantly better results for the problem of pedestrian detection.The accuracy which this thesis has managed to achieve is similar to the accuracy for some other works which use CNN.

Because of the increase in the number of security cameras, there is more video footage available than a human could efficiently process. In combination with the fact that computers are getting more efficient, it is getting more and more interesting to solve the problem of detecting and recognizing people automatically. Therefore a method is proposed for estimating a 3D-path of a person of interest in multiple, non overlapping, monocular cameras. This project is a collaboration between two master theses. This thesis will focus on recognizing a person of interest from several possible candidates, as well as estimating the 3D-position of a person and providing a graphical user interface for the system. The recognition of the person of interest includes keeping track of said person frame by frame, and identifying said person in video sequences where the person of interest has not been seen before. The final product is able to both detect and recognize people in video, as well as estimating their 3D-position relative to the camera. The product is modular and any part can be improved or changed completely, without changing the rest of the product. This results in a highly versatile product which can be tailored for any given situation.

In this thesis, a novel Pedestrian Protection System (PPS), composed of the Pedestrian Detection System (PDS) and the Pedestrian Tracking System (PTS), was proposed. The PPS is a supplementary application for the Advanced Driver Assistance System, which is used to avoid collisions between vehicles and pedestrians. The Pedestrian Detection System (PDS) is used to detect pedestrians from near to far ranges with the feature-classi er-based detection method (HOG + SVM). To achieve pedestrian detection from near to far ranges, a novel structure was proposed. The structure of our PDS consists of two cameras (called CS and CL separately). The CS is equipped with a short focal length lens to detect pedestrians in near-to-mid range; and, the CL is equipped with a long focal length lens to detect pedestrians in mid-to-far range. To accelerate the processing speed of pedestrian detection, the parallel computing capacity of GPU was utilized in the PDS. The synchronization algorithm is also introduced to synchronize the detection results of CS and CL. Based on the novel pedestrian detection structure, the detection process can reach a distance which is more than 130 meters away without decreasing detection accuracy. The detection range can be extended more than 100 meters without decreasing the processing speed of pedestrian detection. Afterwards, an algorithm to eliminate duplicate detection results is proposed to improve the detection accuracy. The Pedestrian Tracking System (PTS) is applied following the Pedestrian Detection System. The PTS is used to track the movement trajectory of pedestrians and to predict the future motion and movement direction. A C + + class (called pedestrianTracking class, which is short for PTC) was generated to operate the tracking process for every detected pedestrian. The Kalman lter is the main algorithm inside the PTC. During the operation of PPS, the nal detection results of each frame from PDS will be transmitted to the PTS to enable the tracking process. The new detection results will be used to update the existing tracking results in the PTS. Moreover, if there is a newly detected pedestrian, a new process will be generated to track the pedestrian in the PTS. Based on the tracking results in PTS, the movement trajectory of pedestrians can be obtained and their future motion and movement direction can be predicted. Two kinds of alerts are generated based on the predictions: warning alert and dangerous alert. These two alerts represent di erent situations; and, they will alert drivers to the upcoming situations. Based on the predictions and alerts, the collisions can be prevented e ectively. The safety of pedestrians can be guaranteed.

Tato práce se zabývá detekcí chodců pomocí konvolučních neuronových sítí z pohledu autonomního vozidla. A to zejména jejich otestováním ve smyslu nalezení vhodné praxe tvorby datasetu pro machine learning modely. V práci bylo natrénováno celkem deset machine learning modelů meta architektur Faster R-CNN s ResNet 101 jako feature extraktorem a SSDLite s feature extraktorem MobileNet_v2. Tyto modely byly natrénovány na datasetech o různých velikostech. Nejlépší výsledky byly dosaženy na datasetu o velikosti 5000 snímků. Kromě těchto modelů byl vytvořen nový dataset zaměřující se na chodce v noci. Dále byla vytvořena knihovna Python funkcí pro práci s datasety a automatickou tvorbu datasetu.

L’intérêt principal des systèmes d’aide à la conduite (ADAS) est d’accroître la sécurité de tous les usagers de la route. Le domaine du véhicule intelligent porte une attention particulière au piéton,l’une des catégories la plus vulnérable. Bien que ce sujet ait été étudié pendant près de cinquante ans par des chercheurs, une solution parfaite n’existe pas encore. Nous avons exploré dans ce travail de thèse différents aspects de la détection et la classification du piéton. Plusieurs domaines du spectre (Visible, Infrarouge proche, Infrarouge lointain et stéréovision) ont été explorés et comparés.Parmi la multitude des systèmes imageurs existants, les capteurs infrarouge lointain (FIR),capables de capturer la température des différents objets, reste particulièrement intéressants pour la détection de piétons. Les piétons ont, le plus souvent, une température plus élevée que les autres objets. En raison du manque d’accessibilité publique aux bases de données d’images thermiques, nous avons acquis et annoté une base de donnée, nommé RIFIR, contenant à la fois des images dans le visible et dans l’infrarouge lointain. Cette base nous a permis de comparer les performances de plusieurs attributs présentés dans l’état de l’art dans les deux domaines.Nous avons proposé une méthode générant de nouvelles caractéristiques adaptées aux images FIR appelées « Intensity Self Similarity (ISS) ». Cette nouvelle représentation est basée sur la similarité relative des intensités entre différents sous-blocks dans la région d’intérêt contenant le piéton.Appliquée sur différentes bases de données, cette méthode a montré que, d’une manière générale,le spectre infrarouge donne de meilleures performances que le domaine du visible. Néanmoins, la fusion des deux domaines semble beaucoup plus intéressante.La deuxième modalité d’image à laquelle nous nous sommes intéressé est l’infrarouge très proche (SWIR, Short Wave InfraRed). Contrairement aux caméras FIR, les caméras SWIR sont capables de recevoir le signal même à travers le pare-brise d’un véhicule. Ce qui permet de les embarquer dans l’habitacle du véhicule. De plus, les imageurs SWIR ont la capacité de capturer une scène même à distance lointaine. Ce qui les rend plus appropriées aux applications liées au véhicule intelligent. Dans le cadre de cette thèse, nous avons acquis et annoté une base de données, nommé RISWIR, contenant des images dans le visible et dans le SWIR. Cette base a permis une comparaison entre différents algorithmes de détection et de classification de piétons et entre le visible et le SWIR. Nos expérimentations ont montré que les systèmes SWIR sont prometteurs pour les ADAS. Les performances de ces systèmes semblent meilleures que celles du domaine du visible.Malgré les performances des domaines FIR et SWIR, le domaine du visible reste le plus utilisé grâce à son bas coût. Les systèmes imageurs monoculaires classiques ont des difficultés à produire une détection et classification de piétons en temps réel. Pour cela, nous avons l’information profondeur (carte de disparité) obtenue par stéréovision afin de réduire l’espace d’hypothèses dans l’étape de classification. Par conséquent, une carte de disparité relativement correcte est indispensable pour mieux localiser le piéton. Dans ce contexte, une multitude de fonctions coût ont été proposées, robustes aux distorsions radiométriques, pour le calcul de la carte de disparité.La qualité de la carte de disparité, importante pour l’étape de classification, a été affinée par un post traitement approprié aux scènes routières.Les performances de différentes caractéristiques calculées pour différentes modalités (Intensité,profondeur, flot optique) et domaines (Visible et FIR) ont été étudiées. Les résultats ont montré que les systèmes les plus robustes sont ceux qui prennent en considération les trois modalités,plus particulièrement aux occultations. / The main purpose of constructing Intelligent Vehicles is to increase the safety for all traffic participants. The detection of pedestrians, as one of the most vulnerable category of road users, is paramount for any Advance Driver Assistance System (ADAS). Although this topic has been studied for almost fifty years, a perfect solution does not exist yet. This thesis focuses on several aspects regarding pedestrian classification and detection, and has the objective of exploring and comparing multiple light spectrums (Visible, ShortWave Infrared, Far Infrared) and modalities (Intensity, Depth by Stereo Vision, Motion).From the variety of images, the Far Infrared cameras (FIR), capable of measuring the temperature of the scene, are particular interesting for detecting pedestrians. These will usually have higher temperature than the surroundings. Due to the lack of suitable public datasets containing Thermal images, we have acquired and annotated a database, that we will name RIFIR, containing both Visible and Far-Infrared Images. This dataset has allowed us to compare the performance of different state of the art features in the two domains. Moreover, we have proposed a new feature adapted for FIR images, called Intensity Self Similarity (ISS). The ISS representation is based on the relative intensity similarity between different sub-blocks within a pedestrian region of interest. The experiments performed on different image sequences have showed that, in general, FIR spectrum has a better performance than the Visible domain. Nevertheless, the fusion of the two domains provides the best results. The second domain that we have studied is the Short Wave Infrared (SWIR), a light spectrum that was never used before for the task of pedestrian classification and detection. Unlike FIRcameras, SWIR cameras can image through the windshield, and thus be mounted in the vehicle’s cabin. In addition, SWIR imagers can have the ability to see clear at long distances, making it suitable for vehicle applications. We have acquired and annotated a database, that we will name RISWIR, containing both Visible and SWIR images. This dataset has allowed us to compare the performance of different pedestrian classification algorithms, along with a comparison between Visible and SWIR. Our tests have showed that SWIR might be promising for ADAS applications,performing better than the Visible domain on the considered dataset. Even if FIR and SWIR have provided promising results, Visible domain is still widely used due to the low cost of the cameras. The classical monocular imagers used for object detectionand classification can lead to a computational time well beyond real-time. Stereo Vision providesa way of reducing the hypothesis search space through the use of depth information contained in the disparity map. Therefore, a robust disparity map is essential in order to have good hypothesis over the location of pedestrians. In this context, in order to compute the disparity map, we haveproposed different cost functions robust to radiometric distortions. Moreover, we have showed that some simple post-processing techniques can have a great impact over the quality of the obtained depth images.The use of the disparity map is not strictly limited to the generation of hypothesis, and couldbe used for some feature computation by providing complementary information to color images.We have studied and compared the performance of features computed from different modalities(Intensity, Depth and Flow) and in two domains (Visible and FIR). The results have showed that the most robust systems are the ones that take into consideration all three modalities, especially when dealing with occlusions.