Advancements in three dimensional (3D) technologies are rapidly increasing. Three Dimensional Television (3DTV) aims at creating 3D experience for the home user. Moreover, multiview autostereoscopic displays provide a depth impression without the requirement for any special glasses and can be viewed from multiple locations. One of the key issues in the 3DTV processing chain is the content generation from the available input data format video plus depth and multiview video plus depth. This data allows for the possibility of producing virtual views using depth-image-based rendering. Although depth-image-based rendering is an efficient method, it is known for appearance of artifacts such as cracks, corona and empty regions in rendered images. While several approaches have tackled the problem, reducing the artifacts in rendered images is still an active field of research.   Two problems are addressed in this thesis in order to achieve a better 3D video quality in the context of view rendering: firstly, how to improve the quality of rendered views using a direct approach (i.e. without applying specific processing steps for each artifact), and secondly, how to fill the large missing areas in a visually plausible manner using neighbouring details from around the missing regions. This thesis introduces a new depth-image-based rendering and depth-based texture inpainting in order to address these two problems. The first problem is solved by an edge-aided rendering method that relies on the principles of forward warping and one dimensional interpolation. The other problem is addressed by using the depth-included curvature inpainting method that uses appropriate depth level texture details around disocclusions.   The proposed edge-aided rendering method and depth-included curvature inpainting methods are evaluated and compared with the state-of-the-art methods. The results show an increase in the objective quality and the visual gain over reference methods. The quality gain is encouraging as the edge-aided rendering method omits the specific processing steps to remove the rendering artifacts. Moreover, the results show that large disocclusions can be effectively filled using the depth-included curvature inpainting approach. Overall, the proposed approaches improve the content generation for 3DTV and additionally, for free view point television.

Orientador: Erivaldo Antonio da Silva / Banca: Sebastião Milton Pinheiro da Silva / Banca: Wallace Correa de Oliveira Casaca / Banca: Aylton Pagamisse / Banca: José Roberto Nogueira / Resumo: Imagens orbitais com alta resolução espacial abriram uma nova era na extração de informações, especialmente em ambientes urbanos, em que valiosas informações sobre as superfícies tornaram-se disponíveis e de forma detalhada. No entanto, a obtenção de informação em áreas urbanas complexas pode ser comprometida pela presença de sombras, que chegam a ocupar uma parte significativa da imagem, causando sérias interferências na sua análise. A remoção de sombras é, portanto, um tema importante e ainda não resolvido devido à dificuldade da tarefa, sendo necessária na etapa de pré-processamento de diversas aplicações. Este trabalho propõe o desenvolvimento de uma nova abordagem automática para a restauração de áreas de sombras em imagens de satélite multiespectrais com alta resolução espacial. A abordagem se divide em três etapas principais e sequenciais, sendo a primeira o pré-processamento, que consiste na conversão das imagens para reflectância aparente e na fusão das bandas para a geração de índices espectrais. Na segunda etapa, a detecção das áreas de sombras é realizada pela combinação do top-hat por fechamento com a injunção de um parâmetro de área, calculado em função do índice de sombras NSDVI (Normalized Saturation-Value Difference Index). As regiões de sombras são refinadas para a geração da máscara de sombras, a qual é utilizada na terceira etapa, como guia na restauração pelo inpainting automático. A estratégia de restauração pelo inpainting híbrido, adaptado para o conte... (Resumo completo, clicar acesso eletrônico abaixo) / Abstract: High resolution satellite images has played an important role in information extraction especially in urban areas, since valuable information and higher level of details from surface become available with these images. Nevertheless, the task of extracting information from complex urban environment can be hampered by shadows, which can occupy a significant part into the image and, thus negatively affecting the image analysis. Therefore, due to the complexity of the problem, shadow removal is a crucial as one of the first pre-processing steps to enhance the performance of many subsequent steps and applications. The main goal of this thesis is to present a new automatic method for shadow removal in high spatial resolution satellite multispectral images. The proposed method comprises three main steps: the first one is the pre-processing, which includesthe conversion of the target image to the top of atmosphere (TOA) reflectance and the image pansherpening to spectral indices generation. Secondly, a shadow pixels candidates' identification is performed, combining black-top-hat (BTH) transformation with area injunction driven by the normalized saturation-value difference index (NSDVI) mask. The obtained output is a shadow mask, which is used to properly guide our automatic inpainting-inspired strategy in the restoration step. In the third step, a hybrid inpainting-based strategy specifically adapted for the multispectral imagery context is applied to recover shadow areas, which uni... (Complete abstract click electronic access below) / Doutor

Thesis (M.S.)--Brigham Young University. Dept. of Computer Science, 2005. / Includes bibliographical references (p. 65-68).

This thesis explores the potential of AI generated images as a means to enhance the design, sketching and brainstorming processes in architecture. The study addresses the challenges faced by architects in generating innovative ideas and overcoming cognitive biases during their sketching phase. By examining the integration of the AI inpainting tool, Dall-E 2 developed by OpenAI, into the architectural sketching process, the study explores the possibilities as well as the challenges with such an integration. To do so, a qualitative approach utilizing a case study methodology was employed, conducting a focus group consisting of five architects. The participants were given the task of creating a skyscraper using the inpainting tool individually and to iterate over the sketches in three iterations. Between each iteration, group discussions were held to discuss their experiences and thoughts on the tool itself and the images generated. The data collected from the focus group was transcribed and analyzed using theoretical thematic analysis. The analysis produced four key themes, including human-computer interaction, tool improvement points, evaluation of the inpainting tool, and evaluation of generated images. The results reveal that even though the participants encountered challenges with the inpainting tool’s interaction and output, they still found value in its application to their process. The findings of this study suggest that AI inpainting effectively can be integrated into the early stages of sketching, providing architects with rapid editing capabilities and alternative design options that align with the characteristics of brainstorming.

Removing unwanted scratches or objects from an image in an undetectable manner is a technique that has been researched for its many useful and varied applications, such as removing scratches, defects, super-imposed text, or even entire objects from a scene. Currently there is a wide variety of algorithms that fill in unwanted regions, none of which incorporate user preferences into the structure completion process. By building a framework to incorporate user preferences into the filling-in process, user input can be utilized to more effectively fill in damaged regions in an image. User input can influence the filling-in process in a variety of ways, including identifying the region to remove, guiding the completion of structure in the damaged region, influencing priority in the searching process for texture completion, and picking the best combination of structure and texture completion in the damaged region. The framework to achieve the interactive filling-in process contains five main steps. First, the scratch or deformity is detected. Second, the edges outside the deformity are detected. Third, curves are fit to the detected edges. Fourth, the structure is completed across the damaged region. Finally, texture synthesis constrained by the previously computed curves is used to fill in the intensities in the damaged region. Scratch detection, structure completion, and texture synthesis can be influenced or guided by user input when given. Defects have successfully been removed from images that contain structure, images that contain texture, and images that contain both structure and texture. A user is able to successfully complete images that contain ambiguous structure in more than one viable way by gesturing the cursor in the direction of desired structure completion.

Benefiting from hardware development, deep learning (DL) has become a popular research area in recent decades. Convolutional neural network (CNN) is a critical deep learning tool that has been utilized in many computer vision problems. Moreover, the data-driven approach has unleashed CNN's potential in acquiring impressive learning ability with minimum human supervision. Therefore, many computer vision problems are brought into the spotlight again. In this thesis, we investigate the application of deep-learning-based methods, particularly the role of deep learning features, in two representative visual tasks: image retrieval and image inpainting. Image retrieval aims to find in a dataset images similar to a query image. In the proposed image retrieval method, we use canonical correlation analysis to explore the relationship between matching and non-matching features from pre-trained CNN, and generate compact transformed features. The level of similarity between two images is determined by a hypothesis test regarding the joint distribution of transformed image feature pairs. The proposed approach is benchmarked against three popular statistical analysis methods, Linear Discriminant Analysis (LDA), Principal Component Analysis with whitening (PCAw), and Supervised Principal Component Analysis (SPCA). Our approach is shown to achieve competitive retrieval performances on Oxford5k, Paris6k, rOxford, and rParis datasets. Moreover, an image inpainting framework is proposed to reconstruct the corrupted region in an image progressively. Specifically, we design a feature extraction network inspired by Gaussian and Laplacian pyramid, which is usually used to decompose the image into different frequency components. Furthermore, we use a two-branch iterative inpainting network to progressively recover the corrupted region on high and low-frequency features respectively and fuse both high and low-frequency features from each iteration. Moreover, an enhancement model is introduced to employ neighbouring iterations' features to further improve intermediate iterations' features. The proposed network is evaluated on popular image inpainting datasets such as Paris Streetview, Celeba, and Place2. Extensive experiments prove the validity of the proposed method in this thesis, and demonstrate the competitive performance against the state-of-the-art. / Thesis / Doctor of Philosophy (PhD)

Many scientific data sets are contaminated by noise, either because of data acquisition process, or because of naturally occurring phenomena. A first step in analyzing such data sets is denoising, i.e., removing additive noise from a noisy image. For images, noise suppression is a delicate and a difficult task. A trade of between noise reduction and the preservation of actual image features has to be made in a way that enhances the relevant image content. The beginning chapter in this thesis is introductory in nature and discusses the Popular denoising techniques in spatial and frequency domains. Wavelet transform has wide applications in image processing especially in denoising of images. Wavelet systems are a set of building blocks that represent a signal in an expansion set involving indices for time and scale. These systems allow the multi-resolution representation of signals. Several well known denoising algorithms exist in wavelet domain which penalize the noisy coefficients by threshold them. We discuss the wavelet transform based denoising of images using bit planes. This approach preserves the edges in an image. The proposed approach relies on the fact that wavelet transform allows the denoising strategy to adapt itself according to directional features of coefficients in respective sub-bands. Further, issues related to low complexity implementation of this algorithm are discussed. The proposed approach has been tested on different sets images under different noise intensities. Studies have shown that this approach provides a significant reduction in normalized mean square error (NMSE). The denoised images are visually pleasing. Many of the image compression techniques still use the redundancy reduction property of the discrete cosine transform (DCT). So, the development of a denoising algorithm in DCT domain has a practical significance. In chapter 3, a DCT based denoising algorithm is presented. In general, the design of filters largely depends on the a-priori knowledge about the type of noise corrupting the image and image features. This makes the standard filters to be application and image specific. The most popular filters such as average, Gaussian and Wiener reduce noisy artifacts by smoothing. However, this operation normally results in smoothing of the edges as well. On the other hand, sharpening filters enhance the high frequency details making the image non-smooth. An integrated approach to design filters based on DCT is proposed in chapter 3. This algorithm reorganizes DCT coefficients in a wavelet transform manner to get the better energy clustering at desired spatial locations. An adaptive threshold is chosen because such adaptively can improve the wavelet threshold performance as it allows additional local information of the image to be incorporated in the algorithm. Evaluation results show that the proposed filter is robust under various noise distributions and does not require any a-priori Knowledge about the image. Inpainting is another application that comes under the category of image processing. In painting provides a way for reconstruction of small damaged portions of an image. Filling-in missing data in digital images has a number of applications such as, image coding and wireless image transmission for recovering lost blocks, special effects (e.g., removal of objects) and image restoration (e.g., removal of solid lines, scratches and noise removal). In chapter 4, a wavelet based in painting algorithm is presented for reconstruction of small missing and damaged portion of an image while preserving the overall image quality. This approach exploits the directional features that exist in wavelet coefficients in respective sub-bands. The concluding chapter presents a brief review of the three new approaches: wavelet and DCT based denoising schemes and wavelet based inpainting method.

Digital inpainting is the technique of filling in the missing regions of an image or a video using information from surrounding area. This technique has found widespread use in applications such as restoration, error recovery, multimedia editing, and video privacy protection. This dissertation addresses three significant challenges associated with the existing and emerging inpainting algorithms and applications. The three key areas of impact are 1) Structure completion for image inpainting algorithms, 2) Fast and efficient object based video inpainting framework and 3) Perceptual evaluation of large area image inpainting algorithms. One of the main approach of existing image inpainting algorithms in completing the missing information is to follow a two stage process. A structure completion step, to complete the boundaries of regions in the hole area, followed by texture completion process using advanced texture synthesis methods. While the texture synthesis stage is important, it can be argued that structure completion aspect is a vital component in improving the perceptual image inpainting quality. To this end, we introduce a global structure completion algorithm for completion of missing boundaries using symmetry as the key feature. While existing methods for symmetry completion require a-priori information, our method takes a non-parametric approach by utilizing the invariant nature of curvature to complete missing boundaries. Turning our attention from image to video inpainting, we readily observe that existing video inpainting techniques have evolved as an extension of image inpainting techniques. As a result, they suffer from various shortcoming including, among others, inability to handle large missing spatio-temporal regions, significantly slow execution time making it impractical for interactive use and presence of temporal and spatial artifacts. To address these major challenges, we propose a fundamentally different method based on object based framework for improving the performance of video inpainting algorithms. We introduce a modular inpainting scheme in which we first segment the video into constituent objects by using acquired background models followed by inpainting of static background regions and dynamic foreground regions. For static background region inpainting, we use a simple background replacement and occasional image inpainting. To inpaint dynamic moving foreground regions, we introduce a novel sliding-window based dissimilarity measure in a dynamic programming framework. This technique can effectively inpaint large regions of occlusions, inpaint objects that are completely missing for several frames, change in size and pose and has minimal blurring and motion artifacts. Finally we direct our focus on experimental studies related to perceptual quality evaluation of large area image inpainting algorithms. The perceptual quality of large area inpainting technique is inherently a subjective process and yet no previous research has been carried out by taking the subjective nature of the Human Visual System (HVS). We perform subjective experiments using eye-tracking device involving 24 subjects to analyze the effect of inpainting on human gaze. We experimentally show that the presence of inpainting artifacts directly impacts the gaze of an unbiased observer and this in effect has a direct bearing on the subjective rating of the observer. Specifically, we show that the gaze energy in the hole regions of an inpainted image show marked deviations from normal behavior when the inpainting artifacts are readily apparent.

Submitted by Samara Calçado de Azevedo (samara_calcado@hotmail.com) on 2018-09-17T13:15:08Z No. of bitstreams: 1 Azevedo_SC.pdf: 8234866 bytes, checksum: 4f6512aa127d3ab1d52e793725f3a75a (MD5) / Approved for entry into archive by Claudia Adriana Spindola null (claudia@fct.unesp.br) on 2018-09-17T13:46:15Z (GMT) No. of bitstreams: 1 azevedo_sc_dr_prud.pdf: 7732297 bytes, checksum: bebf4345bd7542863867084e462c9ff9 (MD5) / Made available in DSpace on 2018-09-17T13:46:15Z (GMT). No. of bitstreams: 1 azevedo_sc_dr_prud.pdf: 7732297 bytes, checksum: bebf4345bd7542863867084e462c9ff9 (MD5) Previous issue date: 2018-08-30 / Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) / Imagens orbitais com alta resolução espacial abriram uma nova era na extração de informações, especialmente em ambientes urbanos, em que valiosas informações sobre as superfícies tornaram-se disponíveis e de forma detalhada. No entanto, a obtenção de informação em áreas urbanas complexas pode ser comprometida pela presença de sombras, que chegam a ocupar uma parte significativa da imagem, causando sérias interferências na sua análise. A remoção de sombras é, portanto, um tema importante e ainda não resolvido devido à dificuldade da tarefa, sendo necessária na etapa de pré-processamento de diversas aplicações. Este trabalho propõe o desenvolvimento de uma nova abordagem automática para a restauração de áreas de sombras em imagens de satélite multiespectrais com alta resolução espacial. A abordagem se divide em três etapas principais e sequenciais, sendo a primeira o pré-processamento, que consiste na conversão das imagens para reflectância aparente e na fusão das bandas para a geração de índices espectrais. Na segunda etapa, a detecção das áreas de sombras é realizada pela combinação do top-hat por fechamento com a injunção de um parâmetro de área, calculado em função do índice de sombras NSDVI (Normalized Saturation-Value Difference Index). As regiões de sombras são refinadas para a geração da máscara de sombras, a qual é utilizada na terceira etapa, como guia na restauração pelo inpainting automático. A estratégia de restauração pelo inpainting híbrido, adaptado para o contexto multiespectral, combina a difusão anisotrópica, ordem de preenchimento definida a partir de uma imagem cartoon e equação de transporte, e o preenchimento baseado em blocos por amostragem local. A avaliação experimental do método proposto foi realizada em um conjunto de 215 imagens, fruto do recorte de duas cenas dos satélites WorldView-2 (WV-2) e Pléiades-1B (PL-1B) sobre a área urbana de São Paulo. Os resultados mostraram uma boa performance da detecção de sombras pelo método proposto, alcançando uma acurácia global média de 94,20% e de 90,84%, para as imagens WV-2 e PL-1B, respectivamente, sendo inclusive superior, quando comparado com outros dois métodos propostos na literatura. A remoção dos efeitos das sombras obtido pelo inpainting, proporcionou uma restauração satisfatória e coerente de feições como vias e telhados, que ficaram melhor discerníveis pelo inpainting. Além disso, quando quantidades substanciais de áreas não contamindas por sombras estão disponíveis, o inpainting aumentou as áreas úteis da imagem mais do que a restauração baseada em outras técnicas de transformação de intensidades, como o matching de histogramas. Assim, frente à necessidade de informações extraídas de imagens multiespectrais, espera-se que o método possa contribuir com a detecção e remoção automática dos efeitos de sombras em imagens multiespectrais com alta resolução espacial de áreas urbanas. / High resolution satellite images has played an important role in information extraction especially in urban areas, since valuable information and higher level of details from surface become available with these images. Nevertheless, the task of extracting information from complex urban environment can be hampered by shadows, which can occupy a significant part into the image and, thus negatively affecting the image analysis. Therefore, due to the complexity of the problem, shadow removal is a crucial as one of the first pre-processing steps to enhance the performance of many subsequent steps and applications. The main goal of this thesis is to present a new automatic method for shadow removal in high spatial resolution satellite multispectral images. The proposed method comprises three main steps: the first one is the pre-processing, which includesthe conversion of the target image to the top of atmosphere (TOA) reflectance and the image pansherpening to spectral indices generation. Secondly, a shadow pixels candidates’ identification is performed, combining black-top-hat (BTH) transformation with area injunction driven by the normalized saturation-value difference index (NSDVI) mask. The obtained output is a shadow mask, which is used to properly guide our automatic inpainting-inspired strategy in the restoration step. In the third step, a hybrid inpainting-based strategy specifically adapted for the multispectral imagery context is applied to recover shadow areas, which unifies anisotropic diffusion, filling-in priority based on cartoon image representations, transport equation and block-based pixel replication using local dynamic sampling. The performance of our approach has been evaluated by taking 215 subset images from WorldView-2 (WV-2) and Pléiades-1B (PL-1B) that encompasses the city of São Paulo. The method achieves an overall accuracy on shadow detection up to 94.2%, for WV-2, and 90.84%, for PL-1B. The comparative results indicate that the proposed method outperforms two existing state-of-the-art methods. Shadow effects are mitigated by local inpainting method in which the satisfactory outputs demonstrate good coherence for highway and building rooftops recovery. Moreover, when largely non-contaminated areas are available in the image, the proposed approach improves significantly more areas of the image than the intensity-based transformation techniques, such as the histogram matching method. Once only multispectral imagery is required as input data, the approach can be suitable to support other remote sensing applications as well. / 2013/25257-4

Au cours de ces dernières années, le domaine de la compression vidéo a connu un essor considérable avec le standard H.264/AVC et l'arrivée de son successeur HEVC. La prédiction spatiale de ces standards repose sur la propagation unidirectionnelle de pixels voisins. Bien que très efficace pour étendre des motifs répondants aux mêmes caractéristiques, cette prédiction présente des performances limitées lorsqu'il s'agit de propager des textures complexes. Cette thèse vise à explorer de nouveaux schémas de prédiction spatiale afin d'améliorer les techniques actuelles de prédiction intra, en étendant ces schémas locaux et monodimensionnels à des schémas globaux, multidimensionnels et multi-patches. Une première méthode de prédiction hybride intégrant correspondance de bloc et correspondance de gabarit (template) a été investiguée. Cette approche hybride a ensuite été étendue en prédiction multi-patches de type "neighbor embedding" (NE). L'autre partie de la thèse est dédiée à l'étude des épitomes dans un contexte de compression d'images. L'idée est d'exploiter la redondance spatiale de l'image d'origine afin d'extraire une image résumé contenant les patches de texture les plus représentatifs de l'image, puis ensuite utiliser cette représentation compacte pour reconstruire l'image de départ. Ce concept d'épitome a été intégré dans deux schémas de compression, l'un de ces algorithmes s'avère vraiment en rupture avec les techniques traditionnelles dans la mesure où les blocs de l'image sont traités, à l'encodeur et au décodeur, dans un ordre spatial qui dépend du contenu et cela dans un souci de propagation des structures de l'image. Dans ce dernier algorithme de compression, des modes de prédiction directionnelle intra H.264 étendus et des méthodes avancées de prédiction multi-patches y ont été également introduits. Ces différentes solutions ont été intégrées dans un encodeur de type H.264/AVC afin d'évaluer leurs performances de codage par rapport aux modes intra H.264 et à l'état de l'art relatif à ces différentes techniques. / In recent years, video compression field has increased significantly since the apparition of H.264/AVC standard and of its successor HEVC. Spatial prediction in these standards are based on the unidirectional propagation of neighboring pixels. Although very effective to extend pattern with the same characteristics, this prediction has limited performances to extrapolate complex textures. This thesis aims at exploring new spatial prediction schemes to improve the current intra prediction techniques, by extending these local schemes to global, multidimensional and multi-patches schemes. A hybrid prediction method based on template and block matching is first investigated. This hybrid approach is then extended to multi-patches prediction of type "Neighbor Embedding" (NE). The other part of this thesis is dedicated to the study of epitome image within the scope of image compression. The idea is to exploit spatial redundancies within the original image in order to first extract a summary image containing the texture patches the most representative of the image, and then use this compacted representation to rebuild the original image. The concept of epitome has been incorporated in two compression schemes, one of these algorithms is in rupture with the traditional techniques since the image blocks are processed, both at encoder and decoder sides, in a spatial order that depends on the image content and this in the interest of propagating image structures. In this last compression algorithm, extended H.264 Intra directional prediction modes and advanced multi-patches prediction methods have been also included. These different solutions have been integrated in an H.264/AVC encoder in order to assess their coding performances with respect to H.264 intra modes and the state of the art relative to these different techniques.