This report covers the implementation and evaluation of a stereo vision corre- spondence-based depth estimation algorithm on a GPU. The results and feed- back are used for a Multi-view camera system in combination with Jetson TK1 devices for parallelized image processing and the aim of this system is to esti- mate the depth of the scenery in front of it. The performance of the algorithm plays the key role. Alongside the implementation, the objective of this study is to investigate the advantages of parallel acceleration inter alia the differences to the execution on a CPU which are significant for all the function, the imposed overheads particular for a GPU application like memory transfer from the CPU to the GPU and vice versa as well as the challenges for real-time and concurrent execution. The study has been conducted with the aid of CUDA on three NVIDIA GPUs with different characteristics and with the aid of knowledge gained through extensive literature study about different depth estimation algo- rithms but also stereo vision and correspondence as well as CUDA in general. Using the full set of components of the algorithm and expecting (near) real-time execution is utopic in this setup and implementation, the slowing factors are in- ter alia the semi-global matching. Investigating alternatives shows that results for disparity maps of a certain accuracy are also achieved by local methods like the Hamming Distance alone and by a filter that refines the results. Further- more, it is demonstrated that the kernel launch configuration and the usage of GPU memory types like shared memory is crucial for GPU implementations and has an impact on the performance of the algorithm. Just concurrency proves to be a more complicated task, especially in the desired way of realization. For the future work and refinement of the algorithm it is therefore recommended to invest more time into further optimization possibilities in regards of shared memory and into integrating the algorithm into the actual pipeline.

Depuis 2011, la réévaluation de plus d’une vingtaine de mesures de flux de neutrinos de réacteur mesurés à courtes distances suggère la présence d’un déficit de 7 % par rapport aux prédictions. Cette anomalie est statistiquement significative à environ 3 sigmas, tout comme une autre anomalie similaire correspondant à l’observation d’un déficit de neutrinos émis par des sources intenses de désintégrations beta. Ces anomalies peuvent être expliquées soit par un biais dans la prédiction des flux de neutrinos, soit par l’existence d’un nouvel état de neutrino, un neutrino stérile léger, vers lequel les trois neutrinos du modèle standard pourraient osciller. Ce nouveau neutrino, n’interagissant pas par interaction faible, serait uniquement « visible » grâce à ces oscillations. L’objectif de l’expérience STEREO est de répondre à la question de l’existence d'un neutrino stérile avec une masse de l’ordre de 1 eV, en utilisant le flux de neutrinos émis par le réacteur de recherche de l’« Institut Laue-Langevin » (ILL) à Grenoble, en France. Pour cela, le détecteur STEREO est segmenté en 6 cellules afin de mesurer l’hypothétique distorsion du spectre en énergie des neutrinos en fonction de la distance au réacteur (entre 9 et 11m de distance). Après quelques années de construction du détecteur et de préparation, l’expérience STEREO a pu commencer la prise de données en novembre 2016 avec une premier période d’acquisition qui s’est terminée en mars 2017. Cette thèse s’inscrit dans ces deux périodes distinctes de l’expérience STEREO. Une première partie a été consacrée à la caractérisation et la validation de l’électronique tout au long du processus de développement jusqu’à sa conception finale. Dans ce cadre, une méthode de mesure de la linéarité de la réponse en charge des photomultiplicateurs et de leurs embases a été développée avec une précision meilleure que 1 %, pour répondre au exigence de sensibilité à l’hypothèse d’un neutrino stérile. Cette exigence a d'ailleurs été validée par une analyse statistique de l’expérience. Une deuxième partie de la thèse a concerné l’analyse des premières données et plus particulièrement, l’identification des différents types de bruit de fond et la recherche des candidats neutrinos. / Since 2011, the re-evaluation of about twenty neutrino flux measurements at short distance from reactors suggests a deficit of about 7 % with respect to predictions. This anomaly has a 3 sigmas significance, as another similar anomaly which corresponds to the deficit of neutrinos emitted from intense source of beta-decay. These anomalies could be explain either by a bias in the flux prediction or by the existence of a new neutrino state, a light sterile neutrino, possibly mixing with the three neutrino states of the standard model. This new neutrino, with no ordinary weak interaction, could only be “visible” via the neutrino oscillation. The STEREO experiment goal is to answer the question of the sterile neutrino existence with mass around 1 eV, using neutrino flux emitted from the research reactor of the « Institut Laue-Langevin » (ILL) at Grenoble, France. To achieve this, the STEREO detector is segmented in 6 cells to measure a hypothetical distortion of the neutrino energy spectrum at different distances from the reactor (between 9 et 11m). Afters few years of detector construction and preparation, the STEREO experiment has started taking data in November 2016 with a first period over in Mars 2017. This thesis took place during two distinct phases of the STEREO experiment. A first part was dedicated to the characterisation and the validation of the electronics, during all its development process until its final conception. In this context, a method to measure the charge response linearity of photomultipliers and their bases, with a precision better than 1 %, has been developed, in order to fulfill the desired sensitivity to the light sterile neutrino hypothesis. This requirement on the precision has been validated by a statistical analysis of the experiment. The second part of the thesis was devoted to the analysis of the first data and more particularly to the identification of the different kinds of background and to the search of neutrino candidates.

Dans cette thèse nous avons l'intention d'enquêter sur la complémentarité des données auditives et visuelles sensorielles pour la construction d'une interprétation de haut niveau d'une scène. L'audiovisuel (AV) d'entrée reçus par le robot est une fonction à la fois l'environnement extérieur et de la localisation réelle du robot qui est étroitement liée à ses actions. La recherche actuelle dans AV analyse de scène a eu tendance à se concentrer sur les observateurs fixes. Toutefois, la preuve psychophysique donne à penser que les humains utilisent petite tête et les mouvements du corps, afin d'optimiser l'emplacement de leurs oreilles à l'égard de la source. De même, en marchant ou en tournant, le robot mai être en mesure d'améliorer les données entrantes visuelle. Par exemple, dans la perception binoculaire, il est souhaitable de réduire la distance de vue à un objet d'intérêt. Cela permet à la structure 3D de l'objet à analyser à une profondeur de résolution supérieure. / In this thesis we plan to investigate the complementarity of auditory and visual sensory data for building a high-level interpretation of a scene. The audiovisual (AV) input received by the robot is a function of both the external environment and of the robot's actual localization which is closely related to its actions. Current research in AV scene analysis has tended to focus on fixed perceivers. However, psychophysical evidence suggests that humans use small head and body movements, in order to optimize the location of their ears with respect to the source. Similarly, by walking or turning, the robot may be able to improve the incoming visual data. For example, in binocular perception, it is desirable to reduce the viewing distance to an object of interest. This allows the 3D structure of the object to be analyzed at a higher depth-resolution.

En vision industrielle, de nombreuses applications de mesure et de contrôle qualité évoluent vers des problématiques tri-dimensionnelles. Les systèmes de stéréovision sont des solutions technologiques qui attirent les industriels par leur simplicité mécanique. Deux caméras statiques disposées à des endroits stratégiques peut s'avérer suffisantes pour répondre à cette problématique bien que les contraintes industrielles imposent de respecter des temps de traitement courts et des mesures précises. La diversité des applications nous amènent à envisager deux approches afin de répondre à deux types d'application. La première technique consiste en la reconstruction 3D à partir de paires de points images qui se correspondent dans les deux images. Elle est destinée à répondre à la problématique de mesure 3D. Les méthodes de calibration monoculaire et de calcul 3D par triangulation sont la base de la reconstruction 3D. Nous étudions la précision de mesure et son évolution selon la pose du système de capture par rapport à la scène observée. La seconde technique consiste à construire des images de disparité afin de répondre à des problématiques de construction de profil et d'analyse de défaut. La contrainte d'alignement des caméras, nécessaire pour accélérer le processus de mise en correspondance, implique d'utiliser des méthodes de calibration stéréoscopique et de rectification des images. Nous étudions l'impact de l'alignement sur la qualité de la rectification. La production de carte dense de disparité se base sur les techniques de stéréo-corrélation. Nous montrons les limites de l'utilisation d'un noyau de corrélation carré et proposons une alternative par production de deux cartes denses de disparité à partir de deux noyaux mono-directionnels, améliorant la mesure de disparité sur les zones de contours et d'occultations. / In industrial vision, many applications for measuring and quality control are moving to three-dimensional problems. Stereovision systems are technological solutions that attract industry by their mechanical simplicity. Two static cameras placed at strategic locations may be sufficient to address this problem although the industrial constraints imposed to respect a short processing time and precise measurements. The diversity of applications lead us to consider two approaches to resolve the two types of application. The first technique consists in the 3D reconstruction from pairs of image points which correspond in both images. It is intended to address the problem of 3D measurement. The methods of monocular calibration and 3D triangulation are the basis of 3D reconstruction. We study the accuracy and its evolution according to the capture system pose compared to the observed scene. The second technique is to construct disparity maps to address problems of building profile and default analysis. The alignment constraint of cameras needed to accelerate the process of matching involves the use of methods of stereoscopic calibration and image rectification. We study the impact of alignment on the quality of the rectification. The production of dense disparity map is based on the stereo-correlation techniques. We show the limits of the use of a squared correlation kernel and propose an alternative production of two dense disparity maps from two mono-directional kernels, improving the measurement of disparity around edges and occlusions.

<p>This thesis describes the development of a low-cost, low-power, accurate sensor designed for precise, feedback control of an autonomous vehicle to a hitch. Few studies have been completed on the hitching problem, yet it is an important challenge to be solved for vehicles in the agricultural and transportation industries. Existing sensor solutions are high cost, high power, and require modification to the hitch in order to work. Other potential sensor solutions such as LiDAR and Digital Fringe Projection suffer from these same fundamental problems. </p> <p>The solution that has been developed uses an active stereo vision system, combining classical stereo vision with a laser speckle projection system, which solves the correspondence problem experienced by classic stereo vision sensors. A third camera is added to the sensor for texture mapping. As a whole, the system cost is $188, with a power usage of 2.3 W.</p> <p>To test the system, a model test of the hitching problem was developed using an RC car and a target to represent a hitch. In the application, both the stereo system and the texture camera are used for measurement of the hitch, and a control system is implemented to precisely control the vehicle to the hitch. The system can successfully control the vehicle from within 35⁰ of perpendicular to the hitch, to a final position with an overall standard deviation of 3.0 mm of lateral error and 1.5⁰ of angular error. Ultimately, this is believed to be the first low power, low cost hitching system that does not require modification of the hitch in order to sense it. </p>

Stereo vision is one of the most active research areas in computer vision. While hundreds of stereo reconstruction algorithms have been developed, little work has been done on the evaluation of such algorithms and almost none on evaluation on Near-Infrared (NIR) images. Of almost a hundred examined, we selected a set of 15 stereo algorithms, mostly with real-time performance, which were then categorized and evaluated on several NIR image datasets, including single stereo pair and stream datasets. The accuracy and run time of each algorithm are measured and compared, giving an insight into which categories of algorithms perform best on NIR images and which algorithms may be candidates for real-time applications. Our comparison indicates that adaptive support-weight and belief propagation algorithms have the highest accuracy of all fast methods, but also longer run times (2-3 seconds). On the other hand, faster algorithms (that achieve 30 or more fps on a single thread) usually perform an order of magnitude worse when measuring the per-centage of incorrectly computed pixels.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第23312号 / 情博第748号 / 新制||情||128(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)准教授 飯山 将晃, 教授 西野 恒, 教授 中村 裕一, 教授 美濃 導彦(京都大学 名誉教授) / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

This thesis presents a new method for improved resolution of stereoscopic 3D terrain mapping by local dense bundling of disparity maps. The Unmanned Systems Lab (USL) at Virginia Tech is designing an unmanned aerial vehicle (UAV) first-response system capable of 3D terrain mapping in the wake of a nuclear event. The UAV is a helicopter, and is equipped with a stereo boom imaging system, GPS, and an inertial measurement system (IMU) for low-altitude aerial mapping. Previous 3D reconstruction algorithms on the project used two-frame rectified stereo correspondence to create a local 3D map, which was geo-located by raw GPS and IMU data. The new local dense bundling algo-rithm combines multiple pairs of stereo images by SURF feature point matching, image rectification, matching of dense points with semi-global block matching, and optimization of camera pose and dense 3D point location using a stereo-constrained local bundle adjustment. The performance of the algorithm is evaluated numerically on synthetic im-agery and qualitatively on real aerial flight data. Results indicate the algorithm produces marked improvement in accuracy and vertical resolution, given a proper helicopter flight path and sufficient image overlap. With synthetic imagery and precise pose supplied, the algorithm shows a 1.2x to 6x reduction in vertical error. / Master of Science

This thesis presents online 3D reconstruction and ground segmentation using unmanned aerial vehicle (UAV) based stereo vision. For this purpose, a long baseline stereo vision system has been designed and built. Application of this system is to work as part of an air and ground based multi-robot autonomous terrain surveying project at Unmanned Systems Lab (USL), Virginia Tech, to act as a first responder robotic system in disaster situations. Areas covered by this thesis are design of long baseline stereo vision system, study of stereo vision raw output, techniques to filter out outliers from raw stereo vision output, a 3D reconstruction method and a study to improve running time by controlling the density of point clouds. Presented work makes use of filtering methods and implementations in Point Cloud Library (PCL) and feature matching on graphics processing unit (GPU) using OpenCV with CUDA. Besides 3D reconstruction, the challenge in the project was speed and several steps and ideas are presented to achieve it. Presented 3D reconstruction algorithm uses feature matching in 2D images, converts keypoints to 3D using disparity images, estimates rigid body transformation between matched 3D keypoints and fits point clouds. To correct and control orientation and localization errors, it fits re-projected UAV positions on GPS recorded UAV positions using iterative closest point (ICP) algorithm as the correction step. A new but computationally intensive process of use of superpixel clustering and plane fitting to increase resolution of disparity images to sub-pixel resolution is also presented. Results section provides accuracy of 3D reconstruction results. The presented process is able to generate application acceptable semi-dense 3D reconstruction and ground segmentation at 8-12 frames per second (fps). In 3D reconstruction of an area of size 25 x 40 m2, with UAV flight altitude of 23 m, average obstacle localization error and average obstacle size/dimension error is found to be of 17 cm and 3 cm, respectively. / MS / This thesis presents near real-time, called online, visual reconstruction in 3-dimensions (3D) using ground facing camera system on an unmanned aerial vehicle. Another result of this thesis is separating ground from obstacles on the ground. To do this the camera system using two cameras, called stereo vision system, with the cameras being positioned comparatively far away from each other at 60 cm was designed as well as an algorithm and software to do the visual 3D reconstruction was developed. Application of this system is to work as part of an air and ground based multi-robot autonomous terrain surveying project at Unmanned Systems Lab, Virginia Tech, to act as a first responder robotic system in disaster situations. Presented work makes use of Point Cloud Library and library functions on graphics processing unit using OpenCV with CUDA, which are popular Computer Vision libraries. Besides 3D reconstruction, the challenge in the project was speed and several steps and ideas are presented to achieve it. Presented 3D reconstruction algorithm is based on feature matching, which is a popular way to mathematically identify unique pixels in an image. Besides using image features in 3D reconstruction, the algorithm also presents a correction step to correct and control orientation and localization errors using iterative closest point algorithm. A new but computationally intensive process to improve resolution of disparity images, which is an output of the developed stereo vision system, from single pixel accuracy to sub-pixel accuracy is also presented. Results section provides accuracy of 3D reconstruction results. The presented process is able to generate application acceptable 3D reconstruction and ground segmentation at 8-12 frames per second. In 3D reconstruction of an area of size 25 x 40 m2 , with UAV flight altitude of 23 m, average obstacle localization error and average obstacle size/dimension error is found to be of 17 cm and 3 cm, respectively.

Diese Arbeit befasst sich mit der Entwicklung eines echtzeitfähigen Systems zur Erstellung von Tiefeninformation aus Stereobildpaaren, das in einer Reihe von Anwendungen zur dreidimensionalen Vermessung des Raumes herangezogen werden kann. Als Hauptanwendungsgebiete sind in erster Linie mobile Robotikapplikationen vorgesehen, die sehr strenge Anforderungen sowohl bezüglich des Ressourcenverbrauchs als auch im Hinblick auf die Messeigenschaften und das Laufzeitverhalten stellen. Ein Merkmal des in dieser Arbeit entworfenen Systems ist die in Echtzeit stattfindende Ausführung der verwendeten Algorithmen in Kombination mit sehr guten Messeigenschaften. Das verwendete Stereo-Matching-Verfahren basiert auf einem globalen Ansatz und liefert im Vergleich zu den alternativen echtzeitfähigen Methoden sehr gute Ergebnisse. Im Vordergrund steht dabei der Semi-Global-Matching-Algorithmus. Aufgrund der Komplexität globaler Ansätze finden in Echtzeitapplikationen nur lokale Stereo-Verfahren Verwendung. Lokale Verfahren liefern jedoch im Vergleich zu den globalen Methoden qualitativ schlechte Disparitätskarten. Ein neuer globaler Matching-Algorithmus Efficient-Semi-Global-Matching (eSGM) wird vorgestellt und in das Konzept für mobile Robotikanwendungen umgesetzt. Wegen der begrenzten Ressourcen der realen Hardware wurde eine Weiterentwicklung des eSGM-Algorithmus für die Realisierung genutzt. Abschließend wird das System anhand der drei Kerneigenschaften Laufzeit, Ressourcenverbrauch und Qualität der Tiefeninformation gegenüber den Verfahren nach dem Stand der Technik bewertet. Der in dieser Arbeit vorgestellte FPGA-Ansatz, die eingesetzte Entwurfsmethode und die vorgestellten Algorithmen ermöglichten es, ein leistungsfähiges Stereo-Bildverarbeitungssystem zu entwickeln, das den hohen Anforderungen bezüglich des Laufzeitverhaltens und der Qualität des Ergebnisses gerecht wird. / This work presents a realtime stereo image matching system that takes advantage of a global image matching method. The system is designed to provide depth information for mobile robotic applications. Typical tasks of the proposed system are to assist in obstacle avoidance, SLAM and path planning of mobile robots, that pose strong requirements on the size, energy consumption, reliability, frame rate and quality of the calculated depth map. Current available systems either rely on active sensors or on local stereo-image matching algorithms. The first are only suitable in controlled environments while the second suffer from low quality depth-maps. Top ranking quality results are only achieved by an iterative approach using global image matching and colour segmentation techniques which are computationally demanding and therefore difficult to be executed in real time. Attempts were made to still reach real-time performance with global methods by simplifying the routines but led to degraded depth maps which are at the end almost comparable with local methods. An equally named semi-global algorithm was proposed earlier, that shows both very good image matching results and relatively simple execution at the same time. A memory efficient variant of the Semi-Global Matching algorithm is presented and adopted for an implementation based on reconfigurable hardware that is suitable for real-time operations in the field of robotics. It will be shown that the modified version of the efficient Semi-Global matching method is delivering equivalent result compared to the original algorithm. The complete design has been implemented within a hardware development framework that is also reviewed.