No description available.

<p>This thesis investigates possibilities to perform image interpolation on an FPGA instead of on a graphics card. The images will be used for 3D display on Setred AB’s screen and an implementation in firmware will hopefully give two major advantages over the existing rendering methods. First, an FPGA can handle big amounts of data and perform a lot of calculations in parallel. Secondly, the amount of data to transfer is drastically increased after the interpolation and with this, a higher bandwith is required to transfer the data at a high speed. By moving the interpolation as close to the projector as possible, the bandwidth requirements can be lowered. Both these points will hopefully be improved, giving a higher frame rate on the screen.</p><p>The thesis consists of three major parts, where the first handles methods to increase the resolution of images. Especially nearest neighbour, bilinear and bicubic interpolation is investigated. Bilinear interpolation was considered to give a good trade off between image quality and calculation cost and was therefore implemented. The second part discusses how a number of perspectives can be interpolated from one or a few captured images and the corresponding depth or disparity maps. Two methods were tested and one was chosen for a final implementation. The last part of the thesis handles Multi Video, a method that can be used to slice the perspectives into a form that is needed for the Scanning Slit display to show them correctly.</p><p>The quality of the images scaled with bilinear interpolation is satisfactory if the scale factor is kept reasonably low. The perspectives interpolated in the second part show good quality with lots of details but suffers from some empty areas. Further improvements of this function is not necessary but would increase the image quality further. An acceptable frame rate has been achieved but further improvements of the speed can be performed. The most important continuation of this thesis is to integrate the implemented parts with the existing firmware and with that enable a real test of the performance.</p>

This thesis investigates possibilities to perform image interpolation on an FPGA instead of on a graphics card. The images will be used for 3D display on Setred AB’s screen and an implementation in firmware will hopefully give two major advantages over the existing rendering methods. First, an FPGA can handle big amounts of data and perform a lot of calculations in parallel. Secondly, the amount of data to transfer is drastically increased after the interpolation and with this, a higher bandwith is required to transfer the data at a high speed. By moving the interpolation as close to the projector as possible, the bandwidth requirements can be lowered. Both these points will hopefully be improved, giving a higher frame rate on the screen. The thesis consists of three major parts, where the first handles methods to increase the resolution of images. Especially nearest neighbour, bilinear and bicubic interpolation is investigated. Bilinear interpolation was considered to give a good trade off between image quality and calculation cost and was therefore implemented. The second part discusses how a number of perspectives can be interpolated from one or a few captured images and the corresponding depth or disparity maps. Two methods were tested and one was chosen for a final implementation. The last part of the thesis handles Multi Video, a method that can be used to slice the perspectives into a form that is needed for the Scanning Slit display to show them correctly. The quality of the images scaled with bilinear interpolation is satisfactory if the scale factor is kept reasonably low. The perspectives interpolated in the second part show good quality with lots of details but suffers from some empty areas. Further improvements of this function is not necessary but would increase the image quality further. An acceptable frame rate has been achieved but further improvements of the speed can be performed. The most important continuation of this thesis is to integrate the implemented parts with the existing firmware and with that enable a real test of the performance.

Esta tese apresenta um novo método para interpolação de vistas em vídeos usando câmeras ao longo de um baseline baseado em uma triangulação 2D. A imagem de referência é primeiramente particionada em regiões triangulares usando informação de bordas e escala, visando colocar vértices ao longo das bordas da imagem e aumentar o número de triângulos em regiões texturadas. Um algoritmo de casamento de regiões é então usado para encontrar a disparidade inicial de cada triângulo, e uma etapa de refinamento é aplicada para mudar a disparidade nos vértices dos triângulos, gerando um mapa de disparidade linear em trechos. Uma simples etapa de pós-processamento é aplicada para conectar os triângulos com disparidade semelhante, gerando uma malha 3D relacionada a cada câmera, que são usadas para gerar novas vistas sintéticas ao longo do mesmo baseline das câmeras. Para gerar vistas com menos artefatos temporais (flickering), foi proposta uma abordagem para atualizar a malha 3D inicial dinamicamente, movendo, removendo e inserindo vértices a cada quadro baseado no fluxo óptico. Esta abordagem permite relacionar triângulos da malha ao longo do tempo, e uma combinação de Modelo Oculto de Markov, aplicado nos triângulos que persistem ao longo do tempo, com Filtro de Kalman, aplicado nos vértices, permite a geração de uma mapa de disparidade com coerência temporal. Com a abordagem proposta, o processo de gerar vistas interpoladas se reduz à trivial tarefa de renderizar uma malha poligonal, algo que pode ser feito muito rapidamente, principalmente quando placas gráficas são utilizadas. Além disso, as vistas geradas não possuem buracos, diferente de muitas técnicas de interpolação de vistas baseadas em pixels que requerem procedimentos de pós-processamento para preencher buracos. Os resultados experimentais indicam que a abordagem proposta foi capaz de gerar vistas interpoladas visualmente coerentes em vídeos desafiadores, com luz natural e movimento de câmera. Além disso, uma avaliação quantitativa usando métricas de qualidade de vídeos mostrou que as sequências de video interpoladas são melhores que abordagens competitivas. / This thesis presents a new method for video view interpolation using multiview linear camera arrays based on 2D domain triangulation. The domain of the reference image is initially partitioned into triangular regions using edge and scale information, aiming to place vertices along image edges and to increase the number of triangles in textured regions. A region-based matching algorithm is then used to find an initial disparity for each triangle, and a refinement stage is applied to change the disparity at the vertices of the triangles, generating a piecewise linear disparity map. A simple post-processing procedure is applied to connect the triangles with similar disparities, generating a full 3D mesh related to each camera (view), which are used to generate the new synthesized views along the cameras baseline. In order to generate views with less temporal flickering artifacts, we propose a scheme to update the initial 3D mesh dynamically, by moving, deleting and inserting vertices at each frame based on optical flow. This approach allows to relate triangles of the mesh across time, and a combination of Hidden Markov Models (HMMs), applied to time-persistent triangles, with the Kalman Filter, applied to vertices, so that temporal consistency can also be obtained. With the proposed framework, view interpolation reduces to the trivial task of rendering polygonal meshes, which can be done very fast, particularly when GPUs are employed. Furthermore, the generated views are hole-free, unlike most point-based view interpolation schemes that require some kind of post-processing procedures to fill holes. Experimental results indicate that our approach was able to generate visually coherent in-between interpolated views for challenging, real-world videos with natural lighting and camera movement.

Esta tese apresenta um novo método para interpolação de vistas em vídeos usando câmeras ao longo de um baseline baseado em uma triangulação 2D. A imagem de referência é primeiramente particionada em regiões triangulares usando informação de bordas e escala, visando colocar vértices ao longo das bordas da imagem e aumentar o número de triângulos em regiões texturadas. Um algoritmo de casamento de regiões é então usado para encontrar a disparidade inicial de cada triângulo, e uma etapa de refinamento é aplicada para mudar a disparidade nos vértices dos triângulos, gerando um mapa de disparidade linear em trechos. Uma simples etapa de pós-processamento é aplicada para conectar os triângulos com disparidade semelhante, gerando uma malha 3D relacionada a cada câmera, que são usadas para gerar novas vistas sintéticas ao longo do mesmo baseline das câmeras. Para gerar vistas com menos artefatos temporais (flickering), foi proposta uma abordagem para atualizar a malha 3D inicial dinamicamente, movendo, removendo e inserindo vértices a cada quadro baseado no fluxo óptico. Esta abordagem permite relacionar triângulos da malha ao longo do tempo, e uma combinação de Modelo Oculto de Markov, aplicado nos triângulos que persistem ao longo do tempo, com Filtro de Kalman, aplicado nos vértices, permite a geração de uma mapa de disparidade com coerência temporal. Com a abordagem proposta, o processo de gerar vistas interpoladas se reduz à trivial tarefa de renderizar uma malha poligonal, algo que pode ser feito muito rapidamente, principalmente quando placas gráficas são utilizadas. Além disso, as vistas geradas não possuem buracos, diferente de muitas técnicas de interpolação de vistas baseadas em pixels que requerem procedimentos de pós-processamento para preencher buracos. Os resultados experimentais indicam que a abordagem proposta foi capaz de gerar vistas interpoladas visualmente coerentes em vídeos desafiadores, com luz natural e movimento de câmera. Além disso, uma avaliação quantitativa usando métricas de qualidade de vídeos mostrou que as sequências de video interpoladas são melhores que abordagens competitivas. / This thesis presents a new method for video view interpolation using multiview linear camera arrays based on 2D domain triangulation. The domain of the reference image is initially partitioned into triangular regions using edge and scale information, aiming to place vertices along image edges and to increase the number of triangles in textured regions. A region-based matching algorithm is then used to find an initial disparity for each triangle, and a refinement stage is applied to change the disparity at the vertices of the triangles, generating a piecewise linear disparity map. A simple post-processing procedure is applied to connect the triangles with similar disparities, generating a full 3D mesh related to each camera (view), which are used to generate the new synthesized views along the cameras baseline. In order to generate views with less temporal flickering artifacts, we propose a scheme to update the initial 3D mesh dynamically, by moving, deleting and inserting vertices at each frame based on optical flow. This approach allows to relate triangles of the mesh across time, and a combination of Hidden Markov Models (HMMs), applied to time-persistent triangles, with the Kalman Filter, applied to vertices, so that temporal consistency can also be obtained. With the proposed framework, view interpolation reduces to the trivial task of rendering polygonal meshes, which can be done very fast, particularly when GPUs are employed. Furthermore, the generated views are hole-free, unlike most point-based view interpolation schemes that require some kind of post-processing procedures to fill holes. Experimental results indicate that our approach was able to generate visually coherent in-between interpolated views for challenging, real-world videos with natural lighting and camera movement.

Esta tese apresenta um novo método para interpolação de vistas em vídeos usando câmeras ao longo de um baseline baseado em uma triangulação 2D. A imagem de referência é primeiramente particionada em regiões triangulares usando informação de bordas e escala, visando colocar vértices ao longo das bordas da imagem e aumentar o número de triângulos em regiões texturadas. Um algoritmo de casamento de regiões é então usado para encontrar a disparidade inicial de cada triângulo, e uma etapa de refinamento é aplicada para mudar a disparidade nos vértices dos triângulos, gerando um mapa de disparidade linear em trechos. Uma simples etapa de pós-processamento é aplicada para conectar os triângulos com disparidade semelhante, gerando uma malha 3D relacionada a cada câmera, que são usadas para gerar novas vistas sintéticas ao longo do mesmo baseline das câmeras. Para gerar vistas com menos artefatos temporais (flickering), foi proposta uma abordagem para atualizar a malha 3D inicial dinamicamente, movendo, removendo e inserindo vértices a cada quadro baseado no fluxo óptico. Esta abordagem permite relacionar triângulos da malha ao longo do tempo, e uma combinação de Modelo Oculto de Markov, aplicado nos triângulos que persistem ao longo do tempo, com Filtro de Kalman, aplicado nos vértices, permite a geração de uma mapa de disparidade com coerência temporal. Com a abordagem proposta, o processo de gerar vistas interpoladas se reduz à trivial tarefa de renderizar uma malha poligonal, algo que pode ser feito muito rapidamente, principalmente quando placas gráficas são utilizadas. Além disso, as vistas geradas não possuem buracos, diferente de muitas técnicas de interpolação de vistas baseadas em pixels que requerem procedimentos de pós-processamento para preencher buracos. Os resultados experimentais indicam que a abordagem proposta foi capaz de gerar vistas interpoladas visualmente coerentes em vídeos desafiadores, com luz natural e movimento de câmera. Além disso, uma avaliação quantitativa usando métricas de qualidade de vídeos mostrou que as sequências de video interpoladas são melhores que abordagens competitivas. / This thesis presents a new method for video view interpolation using multiview linear camera arrays based on 2D domain triangulation. The domain of the reference image is initially partitioned into triangular regions using edge and scale information, aiming to place vertices along image edges and to increase the number of triangles in textured regions. A region-based matching algorithm is then used to find an initial disparity for each triangle, and a refinement stage is applied to change the disparity at the vertices of the triangles, generating a piecewise linear disparity map. A simple post-processing procedure is applied to connect the triangles with similar disparities, generating a full 3D mesh related to each camera (view), which are used to generate the new synthesized views along the cameras baseline. In order to generate views with less temporal flickering artifacts, we propose a scheme to update the initial 3D mesh dynamically, by moving, deleting and inserting vertices at each frame based on optical flow. This approach allows to relate triangles of the mesh across time, and a combination of Hidden Markov Models (HMMs), applied to time-persistent triangles, with the Kalman Filter, applied to vertices, so that temporal consistency can also be obtained. With the proposed framework, view interpolation reduces to the trivial task of rendering polygonal meshes, which can be done very fast, particularly when GPUs are employed. Furthermore, the generated views are hole-free, unlike most point-based view interpolation schemes that require some kind of post-processing procedures to fill holes. Experimental results indicate that our approach was able to generate visually coherent in-between interpolated views for challenging, real-world videos with natural lighting and camera movement.

<p>In this thesis, we study three different image interpolation applications in high definition (HD) video processing: video de-interlacing, frame rate up-conversion, and view interpolation. We propose novel methods for these applications which are based on the concept of Nonlocal-Means (NL-Means).</p> <p>In the first part of this thesis, we introduce a new de-interlacing method which uses NL-Means algorithm. In this method, every interpolated pixel is set to a weighted average of its neighboring pixels in the current, previous, and the next frames. Weights of the pixels used in this filtering are calculated according to the radiometric distance between the surrounding areas of the pixel being interpolated and the neighboring pixels. One of the main challenges of the NL-Means is finding a suitable size for the neighborhoods (similarity window) that we want to find radiometric distance for them. We address this problem by using a steering kernel in our distance function to adapt the effective size of similarity window to the local information of the image. In order to calculate the weights of the filter, we need to have an estimate of the progressive frames. Therefore, we introduce a low computational initial de-interlacing method. This method interpolates the missing pixel along a direction based on two criteria of having minimum variation and being used by the above or below pixels. This method preserves the edge structures and yields superior visual quality compared to the simple edge-based line-averaging and many other simple iv de-interlacing methods.</p> <p>The second part of this thesis is devoted to the frame rate up-conversion application. Our frame rate up-conversion method is based on two main steps: NL-Means and foreground /background segmentation. In this method, for every pixel being interpolated first we check whether it belongs to the background or foreground. If the pixel belongs to the background and the values of the next and previous frames’ pixels are the same, we simply set the pixel intensity to the intensity of its location in the previous or next frame. If the pixel belongs to the foreground, we use NL-Means based interpolation for it. We adjust the equations of the NL-means for frame rate up-conversion so that we do not need to have the neighborhoods of the intermediate for calculating the weights of the filter. The comparison of our method with other existing methods shows the better performance of our method.</p> <p>In the third part of this thesis, we introduce a novel view interpolation method without using disparity estimation. In this method, we let every pixel in the intermediate view be the output of the NL-means using the pixels in the reference views. The experimental results demonstrate the better quality of our results compared with other algorithms which use disparity estimation.</p> / Master of Applied Science (MASc)

[Truncated abstract] Image transition effects are commonly used on television and human computer interfaces. The transition between images creates a perception of continuity which has aesthetic value in special effects and practical value in visualisation. The work in this thesis demonstrates that better image transition effects are obtained by incorporating properties of projective geometry into image transition algorithms. Current state-of-the-art techniques can be classified into two main categories namely shape interpolation and warp generation. Many shape interpolation algorithms aim to preserve rigidity but none preserve it with perspective effects. Most warp generation techniques focus on smoothness and lack the rigidity of perspective mapping. The affine transformation, a commonly used mapping between triangular patches, is rigid but not able to model perspective effects. Image transition techniques from the view interpolation community are effective in creating transitions with the correct perspective effect, however, those techniques usually require more feature points and algorithms of higher complexity. The motivation of this thesis is to enable different views of a planar surface to be interpolated with an appropriate perspective effect. The projective geometric relationship which produces the perspective effect can be specified by two quadrilaterals. This problem is equivalent to finding a perspectively appropriate interpolation for projective transformation matrices. I present two algorithms that enable smooth perspective transition between planar surfaces. The algorithms only require four point correspondences on two input images. ...The second algorithm generates transitions between shapes that lie on the same plane which exhibits a strong perspective effect. It recovers the perspective transformation which produces the perspective effect and constrains the transition so that the in-between shapes also lie on the same plane. For general image pairs with multiple quadrilateral patches, I present a novel algorithm that is transitionally symmetrical and exhibits good rigidity. The use of quadrilaterals, rather than triangles, allows an image to be represented by a small number of primitives. This algorithm uses a closed form force equilibrium scheme to correct the misalignment of the multiple transitional quadrilaterals. I also present an application for my quadrilateral interpolation algorithm in Seitz and Dyer's view morphing technique. This application automates and improves the calculation of the reprojection homography in the postwarping stage of their technique. Finally I unify different image transition research areas into a common framework, this enables analysis and comparison of the techniques and the quality of their results. I highlight that quantitative measures can greatly facilitate the comparisons among different techniques and present a quantitative measure based on epipolar geometry. This novel quantitative measure enables the quality of transitions between images of a scene from different viewpoints to be quantified by its estimated camera path.