Project shows calibration stereoscopic sensor. Also describes basic methods stereo-corespodation using library OpenCV. Project contains calculations of disparity maps on CPU or graphic card (using library OpenCL).

Submitted by Liliane Ferreira (ljuvencia30@gmail.com) on 2018-11-26T13:24:36Z No. of bitstreams: 2 Dissertação - Gabriel da Silva Vieira - 2018.pdf: 13740412 bytes, checksum: ddb7d4353e4f2d7650b087dd0d4bd796 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Approved for entry into archive by Luciana Ferreira (lucgeral@gmail.com) on 2018-11-26T13:43:18Z (GMT) No. of bitstreams: 2 Dissertação - Gabriel da Silva Vieira - 2018.pdf: 13740412 bytes, checksum: ddb7d4353e4f2d7650b087dd0d4bd796 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Made available in DSpace on 2018-11-26T13:43:18Z (GMT). No. of bitstreams: 2 Dissertação - Gabriel da Silva Vieira - 2018.pdf: 13740412 bytes, checksum: ddb7d4353e4f2d7650b087dd0d4bd796 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) Previous issue date: 2018-10-05 / Outro / Disparity maps are key components of a stereo vision system. Autonomous navigation, 3D reconstruction, and mobility are examples of areas of research which use disparity maps as an important element. Although a lot of work has been done in the stereo vision field, it is not easy to build stereo systems with concepts such as reuse and extensible scope. In this study, we explore this gap and it presents a software architecture that can accommodate different stereo methods through a standard structure. Firstly, it introduces some scenarios that illustrate use cases of disparity maps and it shows a novel architecture that foments code reuse. A Disparity Computation Framework (DCF) is presented and we discuss how its components are structured. Then we developed a prototype which closely follows the proposal architecture and we prepared some test cases to be performed. Furthermore, we have implemented disparity methods for validation purposes and to evaluate our disparity refinement method. This refinement method, named as Segmented Consistency Check (SCC), was designed to increase the robustness of stereo matching algorithms. It consists of a segmentation process, statistical analysis of grouping areas and a support weighted function to find and to fill in unknown disparities. The experimental results show that the DCF can satisfy different scenarios on-demand. Besides, they show that SCC method is an efficient approach that can make some enhancements in disparity maps, as reducing the disparity error measure. / Mapas de disparidade são elementos cruciais em sistemas de visão estéreo. Navegação autônoma, reconstrução 3D e mobilidade são exemplos de área de pesquisa que utilizam mapas de disparidade como elementos-chave. Embora muitos trabalhos têm sido feitos na área de visão estéreo, ainda assim, não é trivial construir sistemas estéreos com aplicação de conceitos como reutilização e escopo extensível. Neste estudo, exploramos essa lacuna e apresentamos uma arquitetura de software capaz de acomodar diferentes métodos de visão estéreo através de uma estrutura bem definida. Inicialmente, cenários que ilustram usos de mapa de disparidade são introduzidos e uma arquitetura que fomenta reutilização de código é apresentada. Dessa forma, um Framework de Cálculo de Disparidade (FCD) é apresentado e seus componentes são discutidos a fim de especificar a sua estrutura. Em seguida, um protótipo que segue a arquitetura proposta é apresentado e alguns casos de teste são preparados e executados. Além disso, métodos de cálculo de disparidade foram implementados para propostas de validação e para avaliar o método de refinamento de disparidade proposto pelos autores. Esse método de refinamento, chamado de Checagem de Consistência de Segmento (CCS), foi projetado para aumentar a robustez de algoritmos de combinação estéreo. Trata-se de um método que utiliza um processo de segmentação preliminar, análise estatística de áreas definidas e função ponderada de suporte para encontrar e preencher disparidades marcadas como desconhecidas. Os resultados dos experimentos realizados apontam que o FCD pode satisfazer diferentes cenários sob demanda. Além disso, os resultados mostram que o método CCS é uma abordagem eficiente que pode trazer certos melhoramentos em mapas de disparidade, como reduzir a medida de erro no cálculo de correspondências estéreo.

This master's thesis will take us through theoretical procedure that allows us to determine the distance of an object by stereoscopic sensor. Part of this work presents the description of the steps to achieve image of objects, calibraton and rectification. At the next part our study provides an overview of algortihms for creating disparity maps and determining the distance of the object from sensor. In the following part of thesis deals with the implementation of these processes into aplication which aim is to measure the distance.

This master's thesis will take us through theoretical procedure that allows us to determine the distance of an object by stereoscopic sensor. Part of this work presents the description of the steps to achieve image of objects, calibraton and rectification. At the next part our study provides an overview of algortihms for creating disparity maps and determining the distance of the object from sensor. In the following part of thesis deals with the implementation of these processes into aplication which aim is to measure the distance.

La stéréovision est depuis longtemps étudiée dans le domaine de la vision par ordinateur et a connu dernièrement un regain d'intérêt. Dans le domaine du divertissement un nombre important de films sont maintenant disponibles en version relief. Ma thèse prend place dans ce contexte, dans le cadre d'une convention CIFRE entre le LITIS et l'entreprise StereoLabs dont le cœur de métier est le développement de logiciels d'aides à la production de films stéréoscopiques. Je présente ainsi mes travaux sur le filtrage de cartes de disparités, l'estimation en temps réel ne permet pas d'utiliser les approches les plus performantes, cela conduisant à des résultats bruités et/ou erronés. Je propose alors une méthode de filtrage adaptée à cette contrainte de temps. Mes travaux ont également mis en évidence la problématique du paramétrage de l'algorithme de calcul de carte de disparités. Le nombre élevé de paramètres et la diversité des images nous ont conduit à chercher une méthode automatique pour la détermination des valeurs de ces paramètres. Je propose ainsi une méthode d'optimisation des paramètres de l'algorithme, l'objectif étant de trouver les valeurs qui conduisent aux cartes de disparités les plus fiables. Mes travaux concernent également la partie production de contenus S3D, j'ai développé différentes applications liées à l'utilisation de l'information de disparité. Je propose ainsi une méthode de gestion automatique du relief afin de garantir le confort visuel du spectateur. De plus l'information de disparité est utilisée pour une application de correction chromatique des images pour assurer une concordance des couleurs entre les deux images. / Stereo Vision has been studied for a long time in the field of computer vision and became trendy in the past few years. In the field of entertainment a large amount of movies are now released in relief version. My thesis take place in this context, as a part of an agreement between the LITIS and the StereoLabs company, which develop 3D productions' assistant software. So, here I present my work on disparity map filtering, real time computing constraint don't allow to use strong algorithms which lead to noisy or/and erroneous results. In this case I propose a filtering method dedicated to this computational time constraint. My works also pointed out the problematic of assigning the value for the parameters of our disparity map estimation algorithm. The numerous parameters and images diversity lead to look for an automated approach to assign the parameters values. I propose an approach based on an optimization algorithm, the goal is to find the parameters values which lead to reliable disparity map. My work also deal with the stereoscopic contents production, I have developed different applications based on the usage of the disparity information. I proposed an automated method to handle the stereoscopic volume to warrant the visual comfort of the viewer. I also use the disparity information in a color correction application designed to ensure the color constancy across the two stereoscopics images.

De nos jours, les réseaux de capteurs multimédia sans fils sont prometteurs pour différentes applications et domaines, en particulier avec le développement de l’IoT et des capteurs de caméra efficaces et bon marché. La stéréo vision est également très importante pour des objectifs multiples comme la Cinématographie, les jeux, la Réalité Virtuelle, la Réalité Augmentée, etc. Cette thèse vise à développer un système de reconstruction de scène en 3D prouvant l’utilisation de cartes de disparités stéréoscopiques multi-angles dans le contexte des réseaux de capteurs multimedia. Notre travail peut être divisé en trois parties. La première se concentre sur l’étude de toutes les applications, composants, topologies, contraintes et limitations de ces réseaux. En plus, les méthodes de calcul de disparité de vision stéréoscopique afin de choisir la ou les meilleures méthodes pour réaliser une reconstruction en 3D sur le réseau à faible coût en termes de complexité et de consommation d’énergie. Dans la deuxième partie, nous expérimentons et simulons différents calculs de cartes de disparités sur quelques nœuds en changeant les scénarios (intérieur et extérieur), les distances de couverture, les angles, le nombre de nœuds et les algorithmes. Dans la troisième partie, nous proposons un modèle de réseau basé sur l’arbre pour calculer des cartes de disparités précises sur des nœuds de capteurs de caméra multicouches qui répond aux besoins du serveur pour faire une reconstruction de scène 3D de la scène ou de l’objet d’intérêt. Les résultats sont acceptables et assurent la preuve du concept d’utilisation des cartes de disparités dans le contexte des réseaux de capteurs multimédia. / Nowadays, the WMSNs are promising for different applications and fields, specially with the development of the IoT and cheap efficient camera sensors. The stereo vision is also very important for multiple purposes like Cinematography, games, Virtual Reality, Augmented Reality, etc. This thesis aim to develop a 3D scene reconstruction system that proves the concept of using multiple view stereo disparity maps in the context of WMSNs. Our work can be divided in three parts. The first one concentrates on studying all WMSNs applications, components, topologies, constraints and limitations. Adding to this stereo vision disparity map calculations methods in order to choose the best method(s) to make a 3d reconstruction on WMSNs with low cost in terms of complexity and power consumption. In the second part, we experiment and simulate different disparity map calculations on a couple of nodes by changing scenarios (indoor and outdoor), coverage distances, angles, number of nodes and algorithms. In the third part, we propose a tree-based network model to compute accurate disparity maps on multi-layer camera sensor nodes that meets the server needs to make a 3d scene reconstruction of the scene or object of interest. The results are acceptable and ensure the proof of the concept to use disparity maps in the context of WMSNs.

La multiplication des contenus stéréoscopique augmente les risques de piratage numérique. La solution technologique par tatouage relève ce défi. En pratique, le défi d’une approche de tatouage est d'atteindre l’équilibre fonctionnel entre la transparence, la robustesse, la quantité d’information insérée et le coût de calcul. Tandis que la capture et l'affichage du contenu 3D ne sont fondées que sur les deux vues gauche/droite, des représentations alternatives, comme les cartes de disparité devrait également être envisagée lors de la transmission/stockage. Une étude spécifique sur le domaine d’insertion optimale devient alors nécessaire. Cette thèse aborde les défis mentionnés ci-dessus. Tout d'abord, une nouvelle carte de disparité (3D video-New Three Step Search- 3DV-SNSL) est développée. Les performances des 3DV-NTSS ont été évaluées en termes de qualité visuelle de l'image reconstruite et coût de calcul. En comparaison avec l'état de l'art (NTSS et FS-MPEG) des gains moyens de 2dB en PSNR et 0,1 en SSIM sont obtenus. Le coût de calcul est réduit par un facteur moyen entre 1,3 et 13. Deuxièmement, une étude comparative sur les principales classes héritées des méthodes de tatouage 2D et de leurs domaines d'insertion optimales connexes est effectuée. Quatre méthodes d'insertion appartenant aux familles SS, SI et hybride (Fast-IProtect) sont considérées. Les expériences ont mis en évidence que Fast-IProtect effectué dans la nouvelle carte de disparité (3DV-NTSS) serait suffisamment générique afin de servir une grande variété d'applications. La pertinence statistique des résultats est donnée par les limites de confiance de 95% et leurs erreurs relatives inférieurs er <0.1 / The explosion in stereoscopic video distribution increases the concerns over its copyright protection. Watermarking can be considered as the most flexible property right protection technology. The watermarking applicative issue is to reach the trade-off between the properties of transparency, robustness, data payload and computational cost. While the capturing and displaying of the 3D content are solely based on the two left/right views, some alternative representations, like the disparity maps should also be considered during transmission/storage. A specific study on the optimal (with respect to the above-mentioned properties) insertion domain is also required. The present thesis tackles the above-mentioned challenges. First, a new disparity map (3D video-New Three Step Search - 3DV-NTSS) is designed. The performances of the 3DV-NTSS were evaluated in terms of visual quality of the reconstructed image and computational cost. When compared with state of the art methods (NTSS and FS-MPEG) average gains of 2dB in PSNR and 0.1 in SSIM are obtained. The computational cost is reduced by average factors between 1.3 and 13. Second, a comparative study on the main classes of 2D inherited watermarking methods and on their related optimal insertion domains is carried out. Four insertion methods are considered; they belong to the SS, SI and hybrid (Fast-IProtect) families. The experiments brought to light that the Fast-IProtect performed in the new disparity map domain (3DV-NTSS) would be generic enough so as to serve a large variety of applications. The statistical relevance of the results is given by the 95% confidence limits and their underlying relative errors lower than er<0.1

Kasmet daugybė žmonių būna sužalojami autoįvykiuose, iš kurių dalis sužalojimų būna rimti arba pasibaigia mirtimi. Dedama vis daugiau pastangų kuriant įvairias sistemas, kurios padėtų mažinti nelaimių skaičių kelyje. Tokios sistemos gebėtų perspėti vairuotojus apie galimus pavojus, atpažindamos eismo dalyvius ir sekdamos jų padėtį kelyje. Eismo dalyvių kelyje atpažinimas iš vaizdo yra pakankamai sudėtinga, daug skaičiavimų reikalaujanti problema. Šiame darbe šiai problemai spręsti pasitelkti stereo vaizdai, nesugretinamumo žemėlapis bei konvoliuciniai neuroniniai tinklai. Konvoliuciniai neuroniniai tinklai reikalauja daug skaičiavimų, todėl jie optimizuoti pasitelkus grafikos procesorių ir OpenCL. Gautas iki 33,4% spartos pagerėjimas lyginant su centriniu procesoriumi. Stereo vaizdai ir nesugretinamumo žemėlapis leidžia atmesti didelius kadro regionus, kurių nereikia klasifikuoti su konvoliuciniu neuroniniu tinklu. Priklausomai nuo scenos vaizde, reikalingų klasifikavimo operacijų skaičius sumažėja vidutiniškai apie 70-95% ir tai leidžia kadrą apdoroti atitinkamai greičiau. / Many people are injured during auto accidents each year, some injures are serious or end in death. Many efforts are being put in developing various systems, which could help to reduce accidents on the road. Such systems could warn drivers of a potential danger, while recognizing on-road vehicles and tracking their position on the road. On-road vehicle recognition on image is a complex and computationally very intensive problem. In this paper, to solve this problem, stereo images, disparity map and convolutional neural networks are used. Convolutional neural networks are very computational intensive, so to optimize it GPU and OpenCL are used. 33.4% speed improvement was achieved compared to the central processor. Stereo images and disparity map allows to discard large areas of the image, which are not needed to be classified using convolutional neural networks. Depending on the scene of the image, the number of the required classification operations decreases on average by 70-95% and this allows to process the image accordingly faster.

The explosion in stereoscopic video distribution increases the concerns over its copyright protection. Watermarking can be considered as the most flexible property right protection technology. The watermarking applicative issue is to reach the trade-off between the properties of transparency, robustness, data payload and computational cost. While the capturing and displaying of the 3D content are solely based on the two left/right views, some alternative representations, like the disparity maps should also be considered during transmission/storage. A specific study on the optimal (with respect to the above-mentioned properties) insertion domain is also required. The present thesis tackles the above-mentioned challenges. First, a new disparity map (3D video-New Three Step Search - 3DV-NTSS) is designed. The performances of the 3DV-NTSS were evaluated in terms of visual quality of the reconstructed image and computational cost. When compared with state of the art methods (NTSS and FS-MPEG) average gains of 2dB in PSNR and 0.1 in SSIM are obtained. The computational cost is reduced by average factors between 1.3 and 13. Second, a comparative study on the main classes of 2D inherited watermarking methods and on their related optimal insertion domains is carried out. Four insertion methods are considered; they belong to the SS, SI and hybrid (Fast-IProtect) families. The experiments brought to light that the Fast-IProtect performed in the new disparity map domain (3DV-NTSS) would be generic enough so as to serve a large variety of applications. The statistical relevance of the results is given by the 95% confidence limits and their underlying relative errors lower than er<0.1

Image registration is a fundamental problem that can be found in a diverse range of fields within the research community. It is used in areas such as engineering, science, medicine, robotics, computer vision and image processing, which often require the process of developing a spatial mapping between sets of data. Registration plays a crucial role in the medical imaging field where continual advances in imaging modalities, including MRI, CT and PET, allow the generation of 3D images that explicitly outline detailed in vivo information of not only human anatomy, but also human function. Mutual Information (MI) is a popular entropy-based similarity measure which has found use in a large number of image registration applications. Stemming from information theory, this measure generally outperforms most other intensity-based measures in multimodal applications as it does not assume the existence of any specific relationship between image intensities. It only assumes a statistical dependence. The basic concept behind any approach using MI is to find a transformation, which when applied to an image, will maximise the MI between two images. This thesis presents research using MI in three major topics encompassed by the computer vision and medical imaging field: rigid image registration, stereo vision, and non-rigid image registration. In the rigid domain, a novel gradient-based registration algorithm (MIGH) is proposed that uses Parzen windows to estimate image density functions and Gauss-Hermite quadrature to estimate the image entropies. The use of this quadrature technique provides an effective and efficient way of estimating entropy while bypassing the need to draw a second sample of image intensities (a procedure required in previous Parzen-based MI registration approaches). It is possible to achieve identical results with the MIGH algorithm when compared to current state of the art MI-based techniques. These results are achieved using half the previously required sample sizes, thus doubling the statistical power of the registration algorithm. Furthermore, the MIGH technique improves algorithm complexity by up to an order of N, where N represents the number of samples extracted from the images. In stereo vision, a popular passive method of depth perception, new extensions have been pro- posed in order to increase the robustness of MI-based stereo matching algorithms. Firstly, prior probabilities are incorporated into the MI measure to considerably increase the statistical power of the matching windows. The statistical power, directly related to the number of samples, can become too low when small matching windows are utilised. These priors, which are calculated from the global joint histogram, are tuned to a two level hierarchical approach. A 2D match surface, in which the match score is computed for every possible combination of template and matching windows, is also utilised to enforce left-right consistency and uniqueness constraints. These additions to MI-based stereo matching significantly enhance the algorithms ability to detect correct matches while decreasing computation time and improving the accuracy, particularly when matching across multi-spectra stereo pairs. MI has also recently found use in the non-rigid domain due to a need to compute multimodal non-rigid transformations. The viscous fluid algorithm is perhaps the best method for re- covering large local mis-registrations between two images. However, this model can only be used on images from the same modality as it assumes similar intensity values between images. Consequently, a hybrid MI-Fluid algorithm is proposed to compute a multimodal non-rigid registration technique. MI is incorporated via the use of a block matching procedure to generate a sparse deformation field which drives the viscous fluid algorithm, This algorithm is also compared to two other popular local registration techniques, namely Gaussian convolution and the thin-plate spline warp, and is shown to produce comparable results. An improved block matching procedure is also proposed whereby a Reversible Jump Markov Chain Monte Carlo (RJMCMC) sampler is used to optimally locate grid points of interest. These grid points have a larger concentration in regions of high information and a lower concentration in regions of small information. Previous methods utilise only a uniform distribution of grid points throughout the image.