With the rapid growth in infrared sensor technology and its drastic cost reduction, the potential of application of these imaging technologies in computer vision systems has increased. One potential application for IR imaging is depth from stereo. Discerning depth from stereopsis is difficult because the quality of un-cooled sensors is not sufficient for generating dense depth maps. In this thesis, we investigate the production of sparse disparity maps from un-calibrated infrared stereo images and agree that a dense depth field may not be attained directly from IR stereo images, but perhaps a sparse depth field may be obtained that can be interpolated to produce a dense/semi-dense depth field. In our proposed technique, the sparse disparity map is produced by a robust features-based stereo matching method capable of dealing with the problems of infrared images, such as low resolution and high noise; initially, a set of stable features are extracted from stereo pairs using the phase congruency model, which contrary to the gradient-based feature detectors, provides features that are invariant to geometric transformations. Then, a set of Log-Gabor wavelet coefficients at different orientations and frequencies is used to analyze and describe the extracted features for matching. The resulted sparse disparity map is then refined by triangular and epipolar geometrical constraints. In densifying the sparse map, a watershed transformation is applied to divide the image into several segments, where the disparity inside each segment is assumed to vary smoothly. The surface of each segment is then reconstructed independently by fitting a spline to its known disparities; Experiments on a set of indoor and outdoor IR stereo pairs lend credibility to the robustness of our IR stereo matching and surface reconstruction techniques and hold promise for low-resolution stereo images which don’t have a large amount of texture and local details.

With the rapid growth in infrared sensor technology and its drastic cost reduction, the potential of application of these imaging technologies in computer vision systems has increased. One potential application for IR imaging is depth from stereo. Discerning depth from stereopsis is difficult because the quality of un-cooled sensors is not sufficient for generating dense depth maps. In this thesis, we investigate the production of sparse disparity maps from un-calibrated infrared stereo images and agree that a dense depth field may not be attained directly from IR stereo images, but perhaps a sparse depth field may be obtained that can be interpolated to produce a dense/semi-dense depth field. In our proposed technique, the sparse disparity map is produced by a robust features-based stereo matching method capable of dealing with the problems of infrared images, such as low resolution and high noise; initially, a set of stable features are extracted from stereo pairs using the phase congruency model, which contrary to the gradient-based feature detectors, provides features that are invariant to geometric transformations. Then, a set of Log-Gabor wavelet coefficients at different orientations and frequencies is used to analyze and describe the extracted features for matching. The resulted sparse disparity map is then refined by triangular and epipolar geometrical constraints. In densifying the sparse map, a watershed transformation is applied to divide the image into several segments, where the disparity inside each segment is assumed to vary smoothly. The surface of each segment is then reconstructed independently by fitting a spline to its known disparities; Experiments on a set of indoor and outdoor IR stereo pairs lend credibility to the robustness of our IR stereo matching and surface reconstruction techniques and hold promise for low-resolution stereo images which don’t have a large amount of texture and local details.

No description available.

Dense stereo matching is one of the most extensively investigated topics in computer vision, since it plays an important role in many applications such as 3D scene reconstruction. In this thesis, a novel dense stereo matching method is proposed based on edge-aware truncated minimum spanning tree (T-MST). Instead of employing non-local cost aggregation on traditional MST which is only generated from color differences of neighbouring pixels, a new tree structure, "Edge-Aware T-MST", is proposed to aggregate the cost according to the image texture. Specifically, cost aggregations are strongly enforced in large planar textureless regions due to the truncated edge weights. Meanwhile, the "edge fatten" effect is suppressed by employing a novel hybrid edge-prior which combines edge-prior and superpixel-prior to locate the potential disparity edges. Then a widely used Winner-Takes-All (WTA) strategy is performed to establish initial disparity map. An adaptive non-local refinement is also performed based on the stability of initial disparity estimation. Given the stereo images from Middlebury benchmark, we estimate the disparity maps by using our proposed method and other five state-of-the-art tree-based non-local matching methods. The experimental results show that the proposed method successfully produced reliable disparity values within large planar textureless regions and around object disparity boundaries. Performance comparisons demonstrate that our proposed non-local stereo matching method based on edge-aware T-MST outperforms current non-local tree-based state-of-the-art stereo matching methods in most cases, especially in large textureless planar regions and around disparity bounaries.

The problem of estimating depth from two or more images is a fundamental problem in computer vision, which is commonly referred as to stereo matching. The applications of stereo matching range from 3D reconstruction to autonomous robot navigation. Stereo matching is particularly attractive for applications in real life because of its simplicity and low cost, especially compared to costly laser range finders/scanners, such as for the case of 3D reconstruction. However, stereo matching has its very unique problems like convergence issues in the optimisation methods, and challenges to find matches accurately due to changes in lighting conditions, occluded areas, noisy images, etc. It is precisely because of these challenges that stereo matching continues to be a very active field of research. In this thesis we develop a binocular stereo matching algorithm that works with rectified images (i.e. scan lines in two images are aligned) to find a real valued displacement (i.e. disparity) that best matches two pixels. To accomplish this our research has developed techniques to efficiently explore a 3D space, compare potential matches, and an inference algorithm to assign the optimal disparity to each pixel in the image. The proposed approach is also extended to the trinocular case. In particular, the trinocular extension deals with a binocular set of images captured at the same time and a third image displaced in time. This approach is referred as to t +1 trinocular stereo matching, and poses the challenge of recovering camera motion, which is addressed by a novel technique we call baseline recovery. We have extensively validated our binocular and trinocular algorithms using the well known KITTI and Middlebury data sets. The performance of our algorithms is consistent across different data sets, and its performance is among the top performers in the KITTI and Middlebury datasets.

Les applications basées sur 3D tels que les films 3D, l’impression 3D, la cartographie 3D, la reconnaissance 3D, sont de plus en plus présentes dans notre vie quotidienne; elles exigent une reconstruction 3D qui apparaît alors comme une technique clé. Dans cette thèse, nous nous intéressons à l’appariement stéréo qui est au coeur de l’acquisition 3D. Malgré les nombreuses publications traitant de l’appariement stéréo, il demeure un défi en raison des contraintes de précision et de temps de calcul: la conduite autonome requiert le temps réel; la modélisation d’objets 3D exige une précision et une résolution élevées. La méthode de pondération adaptative des pixels de support (adaptative-supportweight), basée sur le bien connu filtre bilatéral, est une méthode de l’état de l’art, de catégorie locale, qui en dépit de ses potentiels atouts peine à lever l’ambiguïté induite par des pixels voisins, de disparités différentes mais avec des couleurs similaires. Notre première contribution, à base de filtre trilatéral, est une solution pertinente qui tout en conservant les avantages du filtre bilatéral permet de lever l’ambiguïté mentionnée. Evaluée sur le corpus de référence, communément acceptée, Middlebury, elle se positionne comme étant la plus précise au moment où nous écrivons ces lignes. Malgré ces performances, la complexité de notre première contribution est élevée. Elle dépend en effet de la taille de la fenêtre support. Nous avons proposé alors une implémentation récursive du filtre trilatérale, inspirée par les filtres récursifs. Ici, les coûts bruts en chaque pixel sont agrégés à travers une grille organisée en graphe. Quatre passages à une dimension permettent d’atteindre une complexité en O(N), indépendante cette fois de la taille de la fenêtre support. C’est-à-dire des centaines de fois plus rapide que la méthode originale. Pour le calcul des pondérations des pixels du support, notre méthode basée sur le filtre trilatéral introduit un nouveau terme, qui est une fonction d’amplitude du gradient. Celui-ci est remarquable aux bords des objets, mais aussi en cas de changement de couleurs et de texture au sein des objets. Or, le premier cas est déterminant dans l’estimation de la profondeur. La dernière contribution de cette thèse vise alors à distinguer les contours des objets de ceux issus du changement de couleur au sein de l’objet. Les évaluations, sur Middlebury, prouvent l’efficacité de la méthode proposée. Elle est en effet plus précise que la méthode basée sur le filtre trilatéral d’origine, mais aussi d’autres méthodes locales. / 3D-related applications are becoming more and more popular in our daily life, such as 3D movies, 3D printing, 3D maps, 3D object recognition, etc. Many applications require realistic 3D models and thus 3D reconstruction is a key technique behind them. In this thesis, we focus on a basic problem of 3D reconstruction, i.e. stereo matching, which searches for correspondences in a stereo pair or more images of a 3D scene. Although various stereo matching methods have been published in the past decades, it is still a challenging task since the high requirement of accuracy and efficiency in practical applications. For example, autonomous driving demands realtime stereo matching technique; while 3D object modeling demands high quality solution. This thesis is dedicated to develop efficient and accurate stereo matching method. The well-known bilateral filter based adaptive support weight method represents the state-of-the-art local method, but it hardly sorts the ambiguity induced by nearby pixels at different disparities but with similar colors. Therefore, we proposed a novel trilateral filter based method that remedies such ambiguities by introducing a boundary strength term. As evaluated on the commonly accepted Middlebury benchmark, the proposed method is proved to be the most accurate local stereo matching method at the time of submission (April 2013). The computational complexity of the trilateral filter based method is high and depends on the support window size. In order to enhance its computational efficiency, we proposed a recursive trilateral filter method, inspired by recursive filter. The raw costs are aggregated on a grid graph by four one-dimensional aggregations and its computational complexity proves to be O(N), which is independent of the support window size. The practical runtime of the proposed recursive trilateral filter based method processing 375 _ 450 resolution image is roughly 260ms on a PC with a 3:4 GHz Inter Core i7 CPU, which is hundreds times faster than the original trilateral filter based method. The trilateral filter based method introduced a boundary strength term, which is computed from color edges, to handle the ambiguity induced by nearby pixels at different disparities but with similar colors. The color edges consist of two types of edges, i.e. depth edges and texture edges. Actually, only depth edges are useful for the boundary strength term. Therefore, we presented a depth edge detection method, aiming to pick out depth edges and proposed a depth edge trilateral filter based method. Evaluation on Middlebury benchmark proves the effectiveness of the proposed depth edge trilateral filter method, which is more accurate than the original trilateral filter method and other local stereo matching methods.

The prevalence of obesity has necessitated developing safe and convenient tools for timely assessing and monitoring this condition for a broad range of population. Three-dimensional (3D) body imaging has become a new mean for obesity assessment. Moreover, it generates body shape information that is meaningful for fitness, ergonomics, and personalized clothing. In the previous work of our lab, we developed a prototype active stereo vision system that demonstrated a potential to fulfill this goal. But the prototype required four computer projectors to cast artificial textures on the body which facilitate the stereo-matching on texture-deficient images (e.g., skin). This decreases the mobility of the system when used to collect a large population data. In addition, the resolution of the generated 3D~images is limited by both cameras and projectors available during the project. The study reported in this dissertation highlights our continued effort in improving the capability of 3Dbody imaging through simplified hardware for passive stereo and advanced computation techniques. The system utilizes high-resolution single-lens reflex (SLR) cameras, which became widely available lately, and is configured in a two-stance design to image the front and back surfaces of a person. A total of eight cameras are used to form four pairs of stereo units. Each unit covers a quarter of the body surface. The stereo units are individually calibrated with a specific pattern to determine cameras' intrinsic and extrinsic parameters for stereo matching. The global orientation and position of each stereo unit within a common world coordinate system is calculated through a 3Dregistration step. The stereo calibration and 3Dregistration procedures do not need to be repeated for a deployed system if the cameras' relative positions have not changed. This property contributes to the portability of the system, and tremendously alleviates the maintenance task. The image acquisition time is around two seconds for a whole-body capture. The system works in an indoor environment with a moderate ambient light. Advanced stereo computation algorithms are developed by taking advantage of high-resolution images and by tackling the ambiguity problem in stereo matching. A multi-scale, coarse-to-fine matching framework is proposed to match large-scale textures at a low resolution and refine the matched results over higher resolutions. This matching strategy reduces the complexity of the computation and avoids ambiguous matching at the native resolution. The pixel-to-pixel stereo matching algorithm follows a classic, four-step strategy which consists of matching cost computation, cost aggregation, disparity computation and disparity refinement. The system performance has been evaluated on mannequins and human subjects in comparison with other measurement methods. It was found that the geometrical measurements from reconstructed 3Dbody models, including body circumferences and whole volume, are highly repeatable and consistent with manual and other instrumental measurements (CV < 0.1$%, R2>0.99). The agreement of percent body fat (%BF) estimation on human subjects between stereo and dual-energy X-ray absorptiometry (DEXA) was found to be improved over the previous active stereo system, and the limits of agreement with 95% confidence were reduced by half. Our achieved %BF estimation agreement is among the lowest ones of other comparative studies with commercialized air displacement plethysmography (ADP) and DEXA. In practice, %BF estimation through a two-component model is sensitive to body volume measurement, and the estimation of lung volume could be a source of variation. Protocols for this type of measurement should still be created with an awareness of this factor. / text

No description available.

Alpha matting is one of the key techniques in image processing and is used to extract accurate foreground from a still image or video sequences. Chroma keying is a special case of alpha matting with a solid background color. Color spill is one of the difficulties in chroma keying, and it has not been effectively solved by current methods. Sometimes, an image contains both reflected regions and transparent regions. When the foreground in such images is chroma keyed, reflection on the foreground is often falsely treated as transparency and causes unreal foreground extraction and composition. This problem is called color spill. Color spill suppression aims to extract the opaque foreground with the correct transparency descriptor (i.e. alpha value) and remove the reflected background color on it. When the background color presented on the foreground is simultaneously caused by reflection and transparency, color spill suppression becomes extremely challenging. It is because that the reflection removal and the actual transparency estimation is a dilemma. Our proposed method for color spill suppression is to separate reflected regions from transparent regions, and process reflected regions as foreground while keeping transparency unchanged at the same time. In this thesis, we propose a novel method for color spill suppression for chroma keying. The quality of the estimated alpha matte could be significantly improved. In our approach, we suppress color spill by using the polarization and the optical flow algorithm based on disparity estimation. Specifically, we make the assumption that reflection changes more than transparency when the scene is captured by a binocular camera with a polaroid filter. Based on this assumption, we took stereo images with polarization filter, registered stereo images by optical flow and conducted the variance analysis on histograms of input images to separate transparency and reflection. Our experiments show that the opaque foreground with background color spill can be reliably extracted while the real transparency can be kept.

Diese Arbeit befasst sich mit der Entwicklung eines echtzeitfähigen Systems zur Erstellung von Tiefeninformation aus Stereobildpaaren, das in einer Reihe von Anwendungen zur dreidimensionalen Vermessung des Raumes herangezogen werden kann. Als Hauptanwendungsgebiete sind in erster Linie mobile Robotikapplikationen vorgesehen, die sehr strenge Anforderungen sowohl bezüglich des Ressourcenverbrauchs als auch im Hinblick auf die Messeigenschaften und das Laufzeitverhalten stellen. Ein Merkmal des in dieser Arbeit entworfenen Systems ist die in Echtzeit stattfindende Ausführung der verwendeten Algorithmen in Kombination mit sehr guten Messeigenschaften. Das verwendete Stereo-Matching-Verfahren basiert auf einem globalen Ansatz und liefert im Vergleich zu den alternativen echtzeitfähigen Methoden sehr gute Ergebnisse. Im Vordergrund steht dabei der Semi-Global-Matching-Algorithmus. Aufgrund der Komplexität globaler Ansätze finden in Echtzeitapplikationen nur lokale Stereo-Verfahren Verwendung. Lokale Verfahren liefern jedoch im Vergleich zu den globalen Methoden qualitativ schlechte Disparitätskarten. Ein neuer globaler Matching-Algorithmus Efficient-Semi-Global-Matching (eSGM) wird vorgestellt und in das Konzept für mobile Robotikanwendungen umgesetzt. Wegen der begrenzten Ressourcen der realen Hardware wurde eine Weiterentwicklung des eSGM-Algorithmus für die Realisierung genutzt. Abschließend wird das System anhand der drei Kerneigenschaften Laufzeit, Ressourcenverbrauch und Qualität der Tiefeninformation gegenüber den Verfahren nach dem Stand der Technik bewertet. Der in dieser Arbeit vorgestellte FPGA-Ansatz, die eingesetzte Entwurfsmethode und die vorgestellten Algorithmen ermöglichten es, ein leistungsfähiges Stereo-Bildverarbeitungssystem zu entwickeln, das den hohen Anforderungen bezüglich des Laufzeitverhaltens und der Qualität des Ergebnisses gerecht wird. / This work presents a realtime stereo image matching system that takes advantage of a global image matching method. The system is designed to provide depth information for mobile robotic applications. Typical tasks of the proposed system are to assist in obstacle avoidance, SLAM and path planning of mobile robots, that pose strong requirements on the size, energy consumption, reliability, frame rate and quality of the calculated depth map. Current available systems either rely on active sensors or on local stereo-image matching algorithms. The first are only suitable in controlled environments while the second suffer from low quality depth-maps. Top ranking quality results are only achieved by an iterative approach using global image matching and colour segmentation techniques which are computationally demanding and therefore difficult to be executed in real time. Attempts were made to still reach real-time performance with global methods by simplifying the routines but led to degraded depth maps which are at the end almost comparable with local methods. An equally named semi-global algorithm was proposed earlier, that shows both very good image matching results and relatively simple execution at the same time. A memory efficient variant of the Semi-Global Matching algorithm is presented and adopted for an implementation based on reconfigurable hardware that is suitable for real-time operations in the field of robotics. It will be shown that the modified version of the efficient Semi-Global matching method is delivering equivalent result compared to the original algorithm. The complete design has been implemented within a hardware development framework that is also reviewed.