Image interpolation is the process of generating a new image utilizing a set of available images. The available images may be taken with a camera at different times, or with multiple cameras and from different viewpoints. Usually, the interpolation problem in the first scenario is called Frame Rate-Up Conversion (FRUC), and the second one view synthesis. This thesis focuses on image interpolation and addresses both FRUC and view synthesis problems. We propose a novel FRUC method using optical flow motion estimation and a patch-based reconstruction scheme. FRUC interpolates new frames between original frames of a video to increase the number of frames, and increases motion continuity. In our approach first, forward and backward motion vectors are obtained using an optical flow algorithm, and reconstructed versions of the current and previous frames are generated by our patch-based reconstruction scheme. Using the original and reconstructed versions of the current and previous frames, two mismatch masks are obtained. Then two versions of the middle frame are generated using a patch-based scheme, with estimated motion vectors and the current and previous frames. Finally, a middle mask, which identifies the mismatch areas of the two middle frames is reconstructed. Using these three masks, the best candidates for interpolation are selected and fused to obtain the final middle frame. Due to the patch-based nature of our interpolation scheme most of the holes and cracks will be filled. Although there is always a probability of having holes, the size and number of such holes are much smaller than those that would be generated using pixel-based mapping. The rare holes are filled using existing hole-filling algorithms. With fewer and smaller holes, simpler hole-filling algorithms can be applied to the image and the overall complexity of the required post processing decreases. View synthesis is the process of generating a new (virtual) view using available ones. Depending on the amount of available geometric information, view synthesis techniques can be divided into three categories: Image Based Rendering (IBR), Depth Image Based Rendering (DIBR), and Model Based Rendering (MBR). We introduce an adaptive version, patch-based scheme for IBR. This patch-based scheme reduces the size and number of holes during reconstruction. The size of patch is determined in response to edge information for better reconstruction, especially near the boundaries. In the first stage of the algorithm, disparity is obtained using optical flow estimation. Then, a reconstructed version of the left and right views are generated using our adaptive patch-based algorithm. The mismatches between each view and its reconstructed version are obtained in the mismatch detection steps. This stage results in two masks as outputs, which help with the refinement of disparities and the selection of the best patches for final synthesis. Finally, the remaining holes are filled using our simple hole filling scheme and the refined disparities. The adaptive version still benefits from the overlapping effect of the patches for hole reduction. However, compared with our fixed-size version, it results in better reconstruction near the edges, object boundaries, and inside the highly textured areas. We also propose an adaptive patch-based scheme for DIBR. The proposed method avoids unnecessary warping which is a computationally expensive step in DIBR. We divide nearby views into blocks, and only warp the center of each block. To have a better reconstruction near the edges and depth discontinuities, the block size is selected adaptively. In the blending step, an approach is introduced to calculate and refine the blending weights. Many of the existing DIBR schemes warp all pixels of nearby views during interpolation which is unnecessary. We show that using our adaptive patch-based scheme, it is possible to reduce the number of required warping without degrading the overall quality compared with existing schemes. / Thesis / Doctor of Philosophy (PhD)

An image-based technique for passive acquisition of three-dimensional (3-D) range data is proposed. The distance is extracted, in this technique, from the estimation of focus conditions on images produced through a monocular imaging system under natural illumination. The image taken from a 3-D object is generally out-of-focus (defocused). For each surface point, the severity of defocus on the image depends upon how far away the point is from the imaging system and how camera (optical) parameters are adjusted. Each setting of the parameters can be recorded physically, and associated in object-space with the inverse of a distance that corresponds to the position for the sharpest imaging under this setting. Therefore, for a given surface point the defocus severity is a function of such an inverse object-distance. It can be shown that this function is symmetrical to, and monotonic on both sides of, a point corresponding to the inverse distance of the surface point. To estimate the parameters of the function (one of which is the inverse distance of the surface point), 3~4 images need to be taken under different camera settings with known associated inverse distances in object-space, determined through a once-for-all calibration procedure. Defocus severity is evaluated from a calculation on the window image that corresponds to a small area around the surface point, and the inverse variance in the window is suggested in this technique for the best performance. The 3-D surface geometry is acquired by applying the algorithm, in parallel, to all surface points in the field of view. Various aspects of the technique are discussed and several algorithms are developed. The technique is implemented on an opto-digital imaging system and evaluated under different conditions. A number of objects are tested to demonstrate its performance. / Thesis / Master of Engineering (ME)

京都大学 / 新制・課程博士 / 博士(工学) / 甲第24593号 / 工博第5099号 / 新制||工||1976(附属図書館) / 京都大学大学院工学研究科都市社会工学専攻 / (主査)教授 角 哲也, 准教授 竹門 康弘, 准教授 Kantoush Sameh / 学位規則第4条第1項該当 / Doctor of Philosophy (Engineering) / Kyoto University / DFAM

Image-based 3D reconstruction has been employed in industrial metrology for micro measurements and quality control purposes. However, generating a highly-detailed and reliable 3D reconstruction of non-collaborative surfaces (textureless, shiny, and transparent) is still an open issue. This thesis presents various methodologies to successfully generate a highly-detailed and reliable 3D reconstruction of non-collaborative objects using the proposed photometric stereo image acquisition system. The first proposed method employs geometric construction to integrate photogrammetry and photometric stereo in order to overcome each technique's limitations and to leverage each technique's strengths in order to reconstruct an accurate and high-resolution topography of non-collaborative surfaces. This method uses accurate photogrammetric 3D measurements to rectify the global shape deviation of photometric stereo meanwhile uses photometric stereo to recover the high detailed topography of the object. The second method combines the high spatial frequencies of photometric stereo depth map with the low frequencies of photogrammetric depth map in frequency domain to produce accurate low frequencies while retaining high frequencies. For the third approach, we utilize light directionality to improve texture quality by leveraging shade and shadow phenomena using the proposed image-capturing system that employs several light sources for highlighting roughness and microstructures on the surface. And finally, we present two methods that effectively orient images by leveraging the low-contrast textures highlighted on object surfaces (roughness and 3D microstructures) using proper lighting system. Various objects with different surface characteristics including textureless, reflective, and transparent are used to evaluate different proposed approaches. To assess the accuracy of each approach, a comprehensive comparison between reference data and generated 3D points is provided.

This thesis explores the use of computer vision to facilitate three different processes of a quarry's operation. The first is the blasting process. This is where operators determine where to drill in order to execute an efficient and safe blast. Having an operator manually determine the drilling angles and positions can lead to inefficient and dangerous blasts. By using two cameras, oriented vertically, and separated by a fixed baseline, Structure from Motion techniques can be used to create a scaled 3D model of a bench. This can then be analyzed to provide operators with borehole locations and drilling angles in relation to fixed reference targets. The second process explored is the crushing process, where the rocks pass through different crushers that reduce the rocks into smaller sizes. The crushed rocks are then dropped onto a moving conveyor belt. The maximum dimension of the rocks exiting the crushers should not exceed size thresholds that are specific to each crusher. This thesis presents a 2D vision system capable of estimating the size distribution of the rocks by attempting to segment the rocks in each image. The size distribution, based on the maximum dimension of each rock, is estimated by finding the maximum dimension in the image in pixels and converting that to inches. The third process of the quarry operations explored is where the final product is piled up to form stockpiles. For inventory purposes, operators often carry out a manual estimation of the size of a the stockpile. This thesis presents a vision system capable of providing a more accurate estimate for the size of the stockpile by using Structure from Motion techniques to create a 3D reconstruction. User interaction helps to find the points that are relevant to the stockpile in the resulting point cloud, which are then used to estimate the volume. / Master of Science

Sample collection is a common task for mobile robots and there are a variety of manipulators available to perform this operation. This thesis presents a novel scoop sample collection system design which is able to both collect and contain a sample using the same hardware. To ease the operator burden during sampling the scoop system is paired with new semi-autonomous and fully autonomous collection techniques. These are derived from data provided by colored 3D point clouds produced via image-based 3D reconstructions. A custom robotic mobility platform, the Scoopbot, is introduced to perform completely automated imaging of the sampling area and also to pick up the desired sample. The Scoopbot is wirelessly controlled by a base station computer which runs software to create and analyze the 3D point cloud models. Relevant sample parameters, such as dimensions and volume, are calculated from the reconstruction and reported to the operator. During tests of the system in full (48 images) and fast (6-8 images) modes the Scoopbot was able to identify and retrieve a sample without any human intervention. Finally, a new building crack detection algorithm (CDA) is created to use the 3D point cloud outputs from image sets gathered by a mobile robot. The CDA was shown to successfully identify and color-code several cracks in a full-scale concrete building element. / Master of Science

Depth-Image-Based Rendering (DIBR) est une technologie fondamentale dans plusieurs applications liées à la 3D, telles que la vidéo en mode point de vue libre (FVV), la réalité virtuelle (VR) et la réalité augmentée (AR). Cependant, l'évaluation de la qualité des vues synthétisées par DIBR a également posé de nouveaux problèmes, car ce processus induit de nouveaux types de distorsions, qui sont intrinsèquement différentes des distorsions provoquées par le codage vidéo. Ce travail est destiné à mieux évaluer la qualité des vues synthétisées par DIBR en multimédia immersif. Au chapitre 2, nous proposons deux métriques complètements sans référence (NR). Le principe de la première métrique NR NIQSV consiste à utiliser plusieurs opérations morphologiques d’ouverture et de fermeture pour détecter et mesurer les distorsions, telles que les régions floues et l’effritement. Dans la deuxième métrique NR NIQSV+, nous améliorons NIQSV en ajoutant un détecteur de “black hole” et une détection “stretching”.Au chapitre 3, nous proposons deux métriques de référence complète pour traiter les distorsions géométriques à l'aide d'un masque de désocclusion et d'une méthode de correspondance de blocs multi-résolution. Au chapitre 4, nous présentons une nouvelle base de données d'images synthétisée par DIBR avec ses scores subjectifs associés. Ce travail se concentre sur les distorsions uniquement induites par différentes méthodes de synthèse de DIBR qui déterminent la qualité d’expérience (QoE) de ces applications liées à DIBR. En outre, nous effectuons également une analyse de référence des mesures d'évaluation de la qualité objective de pointe pour les vues synthétisées par DIBR sur cette base de données. Le chapitre 5 conclut les contributions de cette thèse et donne quelques orientations pour les travaux futurs. / Depth-Image-Based Rendering (DIBR) is a fundamental technology in several 3D-related applications, such as Free viewpoint video (FVV), Virtual Reality (VR) and Augmented Reality (AR). However, new challenges have also been brought in assessing the quality of DIBR-synthesized views since this process induces some new types of distortions, which are inherently different from the distortions caused by video coding. This work is dedicated to better evaluate the quality of DIBRsynthesized views in immersive multimedia. In chapter 2, we propose a completely No-reference (NR) metric. The principle of the first NR metrics NIQSV is to use a couple of opening and closing morphological operations to detect and measure the distortions, such as “blurry regions” and “crumbling”. In the second NR metric NIQSV+, we improve NIQSV by adding a “black hole” and a “stretching” detection. In chapter 3, we propose two Fullreference metrics to handle the geometric distortions by using a dis-occlusion mask and a multi-resolution block matching methods.In chapter 4, we present a new DIBR-synthesized image database with its associated subjective scores. This work focuses on the distortions only induced by different DIBR synthesis methods which determine the quality of experience (QoE) of these DIBR related applications. In addition, we also conduct a benchmark of the state-of-the-art objective quality assessment metrics for DIBR-synthesized views on this database. The chapter 5 concludes the contributions of this thesis and gives some directions of future work.

Cílem této práce je demonstrovat možnost renderování srsti přímo do existujících obrazů bez toho, aby bylo po uživateli požadováno překreslení všech pixelů nebo dodání kompletní 3D geometrie a osvětlení. Srst je přidána na povrch objektů pomocí extrakce jejich přibližného tvaru a světelných informací z obrazu a takto získaný objekt je poté přerenderován. Tento přístup je nový v tom, že vysokoúrovňové úpravy obrazu (jako např. přidání srsti), mohou úspěšně vést k vizuálně korektním výsledkům a to i přes omezení nepřesnou geometrií a světelnými podmínkami. Relativně velká množina technik použitých v této práci zahrnuje obrazy s velkým dynamickým rozsahem, metody extrakce 3D tvaru z obrazu, výsledky výzkumu vnímání tvaru a osvětlení a fotorealistické renderování. Hlavním cílem práce je potvrdit koncept popsaný výše. Hlavním implementačním jazykem bylo C++ s použitím knihoven wxWidgets, OpenGL a libTIFF. Renderování bylo realizováno v software 3Delight kompatibilním se standardem Renderman, za pomoci množiny shaderů implementovaných v nativním jazyce Rendermanu.

L'objectif principal de cette thèse est de développer des outils pour la reconstruction de l'environnement urbain à partir d'images. Les entrées typiques de notre travail est un ensemble d'images de façades, des empreintes au sol de bâtiments, et des modèles 3D reconstruits à partir d'images aériennes. Les principales étapes comprennent le calibrage des images,le recalage avec le modèle 3D, la récupération des informations de profondeur ainsi que la sémantique des façades.Pour atteindre cet objectif, nous utilisons des techniques du domaine de vision par ordinateur, reconnaissance de formes et de l'informatique graphique. Les contributions de notre approche sont présentés en deux parties.Dans la première partie, nous nous sommes concentrés sur des techniques de reconstruction multi-vues dans le but de récupérer automatiquement les informations de profondeur de façades à partir un ensemble des photographies non calibrées. Tout d'abord, nous utilisons la technique structure et mouvement pour calibrer automatiquement l'ensemble des photographies. Ensuite, nous proposons des techniques pour le recalage de la reconstruction avec un modèle 3D. Enfin, nous proposons des techniques de reconstruction 3d dense (stéréo multi-vues et voxel coloring) pour produire un maillage 3D texturé d'une scène d'un ensemble d'images calibrées.La deuxième partie est consacrée à la reconstruction à partir d'une seule vue et son objectif est de récupérer la structure sémantique d'une façade d'une image ortho-rectifiée. La nouveauté de cette approche est l'utilisation d'une grammaire stochastique décrivant un style architectural comme modèle pour la reconstruction de façades. nous combinons un ensemble de détecteurs image avec une méthode d'optimisation globale stochastique en utilisant l'algorithme Metropolis-Hastings. / The main goal of this thesis is to develop innovative and practicaltools for the reconstruction of buildings from images. The typical input to our workis a set of facade images, building footprints, and coarse 3d models reconstructedfrom aerial images. The main steps include the calibration of the photographs,the registration with the coarse 3d model, the recovery of depth and sematicinformation, and the refinement of the coarse 3d model.To achieve this goal, we use computer vision, pattern recognition and computergraphics techniques. Contributions in this approach are presented on two parts.In the first part, we focused on multiple view reconstruction techniques withthe aim to automatically recover the depth information of facades from a setof uncalibrated photographs. First, we use structure from motion techniques toautomatically calibrate the set of photographs. Then, we propose techniques for theregistration of the sparse reconstruction to a coarse 3d model. Finally, we proposean accelerated multi-view stereo and voxel coloring framework using graphicshardware to produce a textured 3d mesh of a scene from a set of calibrated images.The second part is dedicated to single view reconstruction and its aim is to recoverthe semantic structure of a facade from an ortho-rectified image. The novelty ofthis approach is the use of a stochastic grammar describing an architectural style asa model for facade reconstruction. we combine bottom-up detection with top-downproposals to optimize the facade structure using the Metropolis-Hastings algorithm.

Title: Interactive Preview Renderer for Complex Camera Models Author: Bohumír Zámečník Department: Department of Software and Computer Science Education Supervisor: Dr. Alexander Wilkie Supervisor's e-mail address: alexander@wilkie.at Abstract: An interactive renderer was implemented that allows users to preview the effects of imaging with lenses, such as depth of field, bokeh (defocus highlights) and tilt-shift lens configurations. It is based on a state-of-the-art method which com- bines the power of GPU rasterization and ray tracing. Many models and interactive visualizations were created. A non-interactive simulation of a complex geometrical lens model has been made which is able to produce optical aberrations. Also a prototype implementation of recent fast spreading filters is available. A thorough summary of the principles of optical image formation, lens models and depth of field rendering methods used in computer graphics is given along with a comparison of the approaches and new insights. New possibilities of representing the behavior of complex lenses are suggested, which could be employed to accelerate the rendering. Keywords: image synthesis, camera models, depth of field, GPU, image-based ray tracing