Since the birth of charge coupled devices (CCD) and the complementary metal-oxide-semiconductor (CMOS) active pixel sensors, pixel pitch of digital image sensors has been continuously shrinking to meet the resolution and size requirements of the cameras. However, shrinking pixels reduces the maximum number of photons a sensor can hold, a phenomenon broadly known as the full-well capacity limit. The drop in full-well capacity causes drop in signal-to-noise ratio and dynamic range.<div><br></div><div>The Quanta Image Sensor (QIS) is a class of solid-state image sensors proposed by Eric Fossum in 2005 as a potential solution for the limited full-well capacity problem. QIS is envisioned to be the next generation image sensor after CCD and CMOS since it enables sub-diffraction-limit pixels without the inherited problems of pixel shrinking. Equipped with a massive number of detectors that have single-photon sensitivity, the sensor counts the incoming photons and triggers a binary response “1” if the photon count exceeds a threshold, or “0” otherwise. To acquire an image, the sensor oversamples the space and time to generate a sequence of binary bit maps. Because of this binary sensing mechanism, the full-well capacity, signal-to-noise ratio and the dynamic range can all be improved using an appropriate image reconstruction algorithm. The contribution of this thesis is to address three image processing problems in QIS: 1) Image reconstruction, 2) Threshold design and 3) Color filter array design.</div><div><br></div><div>Part 1 of the thesis focuses on reconstructing the latent grayscale image from the QIS binary measurements. Image reconstruction is a necessary step for QIS because the raw binary measurements are not images. Previous methods in the literature use iterative algorithms which are computationally expensive. By modeling the QIS binary measurements as quantized Poisson random variables, a new non-iterative image reconstruction method based on the Transform-Denoise framework is proposed. Experimental results show that the new method produces better quality images while requiring less computing time.</div><div><br></div><div>Part 2 of the thesis considers the threshold design problem of a QIS. A spatially-varying threshold can significantly improve the reconstruction quality and the dynamic range. However, no known method of how to achieve this can be found in the literature. The theoretical analysis of this part shows that the optimal threshold should match with the underlying pixel intensity. In addition, the analysis proves the existence of a set of thresholds around the optimal threshold that give asymptotically unbiased reconstructions. The asymptotic unbiasedness has a phase transition behavior. A new threshold update scheme based on this idea is proposed. Experimentally, the new method can provide good estimates of the thresholds with less computing budget compared to existing methods.</div><div><br></div><div>Part 3 of the thesis extends QIS capabilities to color imaging by studying how a color filter array should be designed. Because of the small pixel pitch of QIS, crosstalk between neighboring pixels is inevitable and should be considered when designing the color filter arrays. However, optimizing the light efficiency while suppressing aliasing and crosstalk in a color filter array are conflicting tasks. A new optimization framework is proposed to solve the problem. The new framework unifies several mainstream design criteria while offering generality and flexibility. Extensive experimental comparisons demonstrate the effectiveness of the framework.</div>

The objective of the proposed research is to develop high performance, low computational complexity resolution enhancement and demosaicing algorithms. Our approach to both problems is to find creative ways to incorporate edge information into the algorithm design. However, in contrast with the usual edge directed approaches, we do not try to detect edge presence and orientation explicitly. For the image interpolation problem, we study the relationship between low resolution and high resolution pixels, and derive a general interpolation formula to be used on all pixels. This simple interpolation algorithm is able to generate sharp edges in any orientation. We also propose a simple 3 by 3 filter that quantifies local luminance transition and apply it to the demosaicing problem. Additionally, we propose a gradient based directional demosaicing method that does not require setting any thresholds. We show that the performance of this algorithm can be improved by using multiscale gradients. Finally, we address the low spectral correlation demosaicing problem by proposing a new family of hybrid color filter array (CFA) patterns and a local algorithm that is two orders of magnitude faster than a comparable non-local solution while offering the same level of performance.

Pour produire des images couleurs nous devons obtenir l'information relative aux trois couleurs primaires (généralement Rouge, Vert et Bleu) à chaque pixels de l'image. Pour capturer cette information la plupart des caméras numériques utilisent une matrice de filtres couleurs (CFA – Color Filter Array en anglais), c'est-à-dire qu'une mosaïque de couleurs recouvre le capteur de manière à ce qu'une seule couleur soit mesurée à chaque position dans l'image.Cette méthode de mesure est similaire à celle du système visuel humain (HVS – Human Visual System en anglais) pour lequel les cônes LMS (sensibles aux longues L, moyenne M et courte S (short en anglais)) forment également une mosaïque à la surface de la rétine. Pour le système visuel, l'arrangement est aléatoire et change entre les individus alors que pour les caméras nous utilisons des arrangements réguliers. Dans les caméras, on doit interpoler les couleurs manquantes pour retrouver une image couleur totalement résolue, méthode appelée démosaïçage. A cause de l'arrangement régulier ou périodique des filtres couleurs, l'image démosaïçée peut faire apparaître des fausses couleurs ou des artefacts. Dans la littérature, les algorithmes de démosaïçage adressent principalement les mosaïques régulières.Dans cette thèse, nous proposons un algorithme de démosaïçage par apprentissage statistique, qui peut être utilisé avec n’importe quelle mosaïque régulière ou aléatoire. De plus, nous optimisons l’arrangement des couleurs dans la mosaïque et proposons des mosaïques qui, avec notre méthode, offrent des performances supérieures aux meilleures méthodes appliquées aux mosaïques régulières. Les images démosaïçées à partir de ces mosaïques ne présentent pas de fausses couleurs ou artefacts.Nous avons étendu l’algorithme pour qu’il ne soit pas limité à trois couleurs mais puisse être utilisé pour un arrangement aléatoire d’un nombre quelconque de filtres spectraux. Avoir plus de trois couleurs permet non seulement de mieux représenter les images mais aussi de mesurer des signatures spectrales de la scène. Ces mosaïques sont appelées matrice de filtres spectraux (SFA – Spectral Filter Array en anglais). Les technologies récentes nous offrent une grande flexibilité pour définir les filtres spectraux et par démosaïçage nous pouvons obtenir des couleurs plus justes et une estimation de la radiance spectrale de la scène. Le substrat silicium dans lequel les photodiodes du capteur sont réalisées est sensible aux radiations proche infra-rouge et donc des filtres visibles et proche infra-rouge peuvent-être combinés dans la même mosaïque. Cette combinaison est particulièrement utile pour le nouveau challenge des caméras numérique d’obtenir des images couleurs en vision de nuit à basse lumière.Nous démontrons l'application de notre algorithme pour plusieurs exemples de cameras récentes équipées d'une matrice de filtres spectraux. Nous montrons que notre méthode est plus performante que les algorithmes actuels en terme de qualité d'image et de vitesse de calcul. Nous proposons également d'optimiser les transmissions des filtres et leur arrangement pour améliorer les résultats selon des métriques ou applications choisies.La méthode, basée sur la minimisation de l'erreur quadratique moyenne est linéaire et par conséquent rapide et applicable en temps réel. Finalement, pour défier la nature linéaire de notre algorithme, nous proposons un deuxième algorithme de démosaïçage par réseaux de neurones qui à des performances légèrement meilleures mais pour un coût de calcul supérieur. / To produce color images we need information of three primary colors (notably Red, Green and Blue) at each pixel point. To capture this information most digital cameras utilize a Color Filter Array (CFA), i.e. a mosaic arrangement of these colors is overlaid on the sensor such that only one color is sampled at one pixel.This arrangement is similar to the Human Visual System (HVS) wherein a mosaic of LMS cones (for sensitivity to Long, Medium and Short wavelength) forms the surface of the retina. For HVS, the arrangement is random and differs between individuals, whereas for cameras we use a regular arrangement of color filters. For digital cameras one needs to interpolate the missing colors to recover the full color image and this process is known as demosaicing. Due to regular or periodic arrangement of color filters the output demosaiced image is susceptible to false colors and artifacts. In literature, the demosaicing algorithms proposed so far cater mainly to regular CFAs.In this thesis, we propose an algorithm for demosaicing which can be used to demosaic any random or regular CFA by learning statistics of an image database. Further, we optimize and propose CFAs such that they outperform even the state of art algorithms on regular CFAs. At the same time the demosaiced images from proposed CFAs are free from false colors and artifacts.We extend our algorithm such that it is not limited to only three colors but can be used for any random arrangement of any number of spectral filters. Having more than three colors allows us to not only record an image but to record a spectral signature of the scene. These mosaics are known as Spectral Filter Arrays (SFAs). Recent technological advances give us greater flexibility in designing the spectral filters and by demosaicing them we can get more accurate colors and also do estimation of spectral radiance of the scene. We know that silicon is inherently sensitive to Near-Infrared radiation and therefore both Visible and NIR filters can be combined on the same mosaic. This is useful for low light night vision cameras which is a new challenge in digital imaging.We demonstrate the applicability of our algorithm on several state of the art cameras using these novel SFAs. In this thesis, we demonstrate that our method outperforms the state of art algorithms in image quality and computational efficiency. We propose a method to optimize filters and their arrangement such that it gives best results depending on metrics and application chosen.The method based on minimization of mean square error is linear in nature and therefore very fast and suitable for real time applications. Finally to challenge the linear nature of LMMSE we propose a demosaicing algorithm using Neural Networks training on a small database of images which is slightly better than the linear demosaicing however, it is computationally more expensive.

To reduce cost, many digital cameras use a single sensor array instead of using three arrays for the red, green and blue. Thus at each pixel location only the red, green or blue intensity value is available. And to generate a complete color image, the camera must estimate the missing two values at each pixel location .Color filter arrays are used to capture only one portion of the spectrum (Red, Green or Blue) at each location. Various arrangements of the Color Filter Array (CFA) are possible, but the Bayer array is the most commonly used arrangement and we will deal exclusively with the Bayer array in this thesis. Since each of the three colors channels are effectively downsampled, it leads to aliasing artifacts. This thesis will analyze the effects of aliasing in the frequency- domain and present a method to reduce the deterioration in image quality due to aliasing artifacts. Two reference algorithms, AH-POCS (Adams and Hamilton - Projection Onto Convex Sets) and Adaptive Homogeneity-Directed interpolation, are discussed in de- tail. Both algorithms use the assumption that there is high correlation in the high- frequency regions to reduce aliasing. AH-POCS uses alias cancellation technique to reduce aliasing in the red and blue images, while the Adaptive Homogeneity-Directed interpolation algorithm is an edge-directed algorithm. We present here an algorithm that combines these two techniques and provides a better result on average when compared to the reference algorithms.

This thesis is about the multicolored gradients seen when using certain dichroic color lters with artificial light. As of now, this phenomenon lacks a unambiguous descriptor, and “Goniochromatic Gradient” is proposed. With help of optical physics, the science of color vision and information about dichroic products, principles for the relationship between goniochromatic gradients and dichroic filters are formulated for anyone interested in exploring this visual phenomenon.