In this thesis, we propose a chroma keying system that automatically estimates the alpha map and the reliable intrinsic color of foreground objects in front of solid background. Our system is designed to be capable of distinguishing the transparent foreground from the re flective foreground and shaded background, thereby making the artifacts of the composited image less conspicuous. Speci cally, we assume that the transparent region tends to be with higher saturation and lightness compared with region re ecting background light. With this assumption, a threshold function (TF) on a saturation-lightness plane is de ned according to human visual experiments. The pixels with color mixed with the background light (conventional unknown pixels) are now further categorized into re ective pixels and transparent pixels according to TF. In this case, the re ective and the transparent regions are separated to improve the alpha matte quality. Furthermore, a new color representation model is proposed to estimate the intrinsic color of each pixel according to the global color distribution of the image. The underlying assumption of our proposed model is that all colors in a natural image can be approximated by a limited number of chrominance values (dominant colors). Speci cally, the color statistics are counted by 2D histogram analysis. Then, we approximate the color distribution by the sum of a set of Gaussian mixture functions (GMF), whose centroids are the dominant colors (Dc) of the image. By choosing colors around each Dc, the possible intrinsic colors for each pixel can be comprehensively and e fficiently selected. Considering the fast development of IP-based video broadcasting, we present a data hiding scheme that can protect the chroma keying results when the image/video data is recorded in the JPEG/H.264 format. According to our simulation, the proposed chroma keying system generates high quality composited images that are little a ected by re ecting, background shading, and intrinsic color missing.

This thesis proposes a novel chroma keying method based on stereo images, which can be applied to post-process the alpha matte generated by any existing matting approach. Given a pair of stereo images, a new matting Laplacian matrix is constructed based on the affinities between matching pixels and their neighbors from two frames. Based on the new matting Laplacian matrix, a new cost function is also formulated to estimate alpha values of the reference image through the propagation between stereo images.

Alpha matting is one of the key techniques in image processing and is used to extract accurate foreground from a still image or video sequences. Chroma keying is a special case of alpha matting with a solid background color. Color spill is one of the difficulties in chroma keying, and it has not been effectively solved by current methods. Sometimes, an image contains both reflected regions and transparent regions. When the foreground in such images is chroma keyed, reflection on the foreground is often falsely treated as transparency and causes unreal foreground extraction and composition. This problem is called color spill. Color spill suppression aims to extract the opaque foreground with the correct transparency descriptor (i.e. alpha value) and remove the reflected background color on it. When the background color presented on the foreground is simultaneously caused by reflection and transparency, color spill suppression becomes extremely challenging. It is because that the reflection removal and the actual transparency estimation is a dilemma. Our proposed method for color spill suppression is to separate reflected regions from transparent regions, and process reflected regions as foreground while keeping transparency unchanged at the same time. In this thesis, we propose a novel method for color spill suppression for chroma keying. The quality of the estimated alpha matte could be significantly improved. In our approach, we suppress color spill by using the polarization and the optical flow algorithm based on disparity estimation. Specifically, we make the assumption that reflection changes more than transparency when the scene is captured by a binocular camera with a polaroid filter. Based on this assumption, we took stereo images with polarization filter, registered stereo images by optical flow and conducted the variance analysis on histograms of input images to separate transparency and reflection. Our experiments show that the opaque foreground with background color spill can be reliably extracted while the real transparency can be kept.

This thesis proposes a new chroma keying method that can automatically detect background, foreground, and unknown regions. For background color detection, we use K-means clustering in color space to calculate the limited number of clusters of background colors. We use spatial information to clean the background regions and minimize the unknown regions. Our method only needs minimum inputs from user. For unknown regions, we implement the alpha matte based on Wang's robust matting algorithm, which is considered one of the best algorithms in the literature, if not the best. Wang's algorithm is based on modified random walk. We proposed a better color selection method, which improves matting results in the experiments. In the thesis, a detailed implementation of robust matting is provided. The experimental results demonstrate that our proposed method can handle images with one background color, images with gridded background, and images with difficult regions such as complex hair stripes and semi-transparent clothes.

This thesis proposes a new chroma keying method that can automatically detect background, foreground, and unknown regions. For background color detection, we use K-means clustering in color space to calculate the limited number of clusters of background colors. We use spatial information to clean the background regions and minimize the unknown regions. Our method only needs minimum inputs from user. For unknown regions, we implement the alpha matte based on Wang's robust matting algorithm, which is considered one of the best algorithms in the literature, if not the best. Wang's algorithm is based on modified random walk. We proposed a better color selection method, which improves matting results in the experiments. In the thesis, a detailed implementation of robust matting is provided. The experimental results demonstrate that our proposed method can handle images with one background color, images with gridded background, and images with difficult regions such as complex hair stripes and semi-transparent clothes.

This thesis proposes a new chroma keying method that can automatically detect background, foreground, and unknown regions. For background color detection, we use K-means clustering in color space to calculate the limited number of clusters of background colors. We use spatial information to clean the background regions and minimize the unknown regions. Our method only needs minimum inputs from user. For unknown regions, we implement the alpha matte based on Wang's robust matting algorithm, which is considered one of the best algorithms in the literature, if not the best. Wang's algorithm is based on modified random walk. We proposed a better color selection method, which improves matting results in the experiments. In the thesis, a detailed implementation of robust matting is provided. The experimental results demonstrate that our proposed method can handle images with one background color, images with gridded background, and images with difficult regions such as complex hair stripes and semi-transparent clothes.

This thesis proposes a new chroma keying method that can automatically detect background, foreground, and unknown regions. For background color detection, we use K-means clustering in color space to calculate the limited number of clusters of background colors. We use spatial information to clean the background regions and minimize the unknown regions. Our method only needs minimum inputs from user. For unknown regions, we implement the alpha matte based on Wang's robust matting algorithm, which is considered one of the best algorithms in the literature, if not the best. Wang's algorithm is based on modified random walk. We proposed a better color selection method, which improves matting results in the experiments. In the thesis, a detailed implementation of robust matting is provided. The experimental results demonstrate that our proposed method can handle images with one background color, images with gridded background, and images with difficult regions such as complex hair stripes and semi-transparent clothes.

In Mixed Reality scenarios, background replacement is a common way to immerse a user in a synthetic environment. Properly identifying the background pixels in an image or video is a dif- ficult problem known as matting. Proper alpha mattes usually come from human guidance, special hardware setups, or color dependent algorithms. This is a consequence of the under-constrained nature of the per pixel alpha blending equation. In constant color matting, research identifies and replaces a background that is a single color, known as the chroma key color. Unfortunately, the algorithms force a controlled physical environment and favor constant, uniform lighting. More generic approaches, such as natural image matting, have made progress finding alpha matte solutions in environments with naturally occurring backgrounds. However, even for the quicker algorithms, the generation of trimaps, indicating regions of known foreground and background pixels, normally requires human interaction or offline computation. This research addresses ways to automatically solve an alpha matte for an image in realtime, and by extension a video, using a consumer level GPU. It does so even in the context of noisy environments that result in less reliable constraints than found in controlled settings. To attack these challenges, we are particularly interested in automatically generating trimaps from depth buffers for dynamic scenes so that algorithms requiring more dense constraints may be used. The resulting computation is parallelizable so that it may run on a GPU and should work for natural images as well as chroma key backgrounds. Extra input may be required, but when this occurs, commodity hardware available in most Mixed Reality setups should be able to provide the input. This allows us to provide real-time alpha mattes for Mixed Reality scenarios that take place in relatively controlled environments. As a consequence, while monochromatic backdrops (such as green screens or retro-reflective material) aid the algorithm’s accuracy, they are not an explicit requirement. iii Finally we explore a sub-image based approach to parallelize an existing hierarchical approach on high resolution imagery. We show that locality can be exploited to significantly reduce the memory and compute requirements of previously necessary when computing alpha mattes of high resolution images. We achieve this using a parallelizable scheme that is both independent of the matting algorithm and image features. Combined, these research topics provide a basis for Mixed Reality scenarios using real-time natural image matting on high definition video sources.