Image signal processors (ISPs) are historically grown legacy software systems for reconstructing color images from noisy raw sensor measurements. They are usually composited of many heuristic blocks for denoising, demosaicking, and color restoration. Color reproduction in this context is of particular importance, since the raw colors are often severely distorted, and each smart phone manufacturer has developed their own characteristic heuristics for improving the color rendition, for example of skin tones and other visually important colors. In recent years there has been strong interest in replacing the historically grown ISP systems with deep learned pipelines. Much progress has been made in approximating legacy ISPs with such learned models. However, so far the focus of these efforts has been on reproducing the structural features of the images, with less attention paid to color rendition. Here we present Color Rendition ISP (CRISPnet), the first learned ISP model to specifically target color rendition accuracy relative to a complex, legacy smart phone ISP. We achieve this by utilizing both image metadata (like a legacy ISP would), as well as by learning simple global semantics based on image classification – similar to what a legacy ISP does to determine the scene type. We also contribute a new ISP image dataset consisting of both high dynamic range monitor data, as well as real-world data, both captured with an actual cell phone ISP pipeline under a variety of lighting conditions, exposure times, and gain settings.

abstract: The last decade has witnessed a paradigm shift in computing platforms, from laptops and servers to mobile devices like smartphones and tablets. These devices host an immense variety of applications many of which are computationally expensive and thus are power hungry. As most of these mobile platforms are powered by batteries, energy efficiency has become one of the most critical aspects of such devices. Thus, the energy cost of the fundamental arithmetic operations executed in these applications has to be reduced. As voltage scaling has effectively ended, the energy efficiency of integrated circuits has ceased to improve within successive generations of transistors. This resulted in widespread use of Application Specific Integrated Circuits (ASIC), which provide incredible energy efficiency. However, these are not flexible and have high non-recurring engineering (NRE) cost. Alternatively, Field Programmable Gate Arrays (FPGA) offer flexibility to implement any application, but at the cost of higher area and energy compared to ASIC. In this work, a spatially programmable architecture customized for image processing applications is proposed. The intent is to bridge the efficiency gap between ASICs and FPGAs, by offering FPGA-like flexibility and ASIC-like energy efficiency. This architecture minimizes the energy overheads in FPGAs, which result from the use of fine-grained programming style and global interconnect. It is flexible compared to an ASIC and can accommodate multiple applications. The main contribution of the thesis is the feasibility analysis of the data path of this architecture, customized for image processing applications. The data path is implemented at the register transfer level (RTL), and the synthesis results are obtained in 45nm technology cell library from a leading foundry. The results of image-processing applications demonstrate that this architecture is within a factor of 10x of the energy and area efficiency of ASIC implementations. / Dissertation/Thesis / Masters Thesis Computer Science 2016

<p dir="ltr">A camera's image signal processor (ISP) is responsible for taking the mosaiced and noisy image signal from the image sensor and processing it such a way that an end-result image is produced that is informative and accurately captures the scene. Real-time video capture in photon-limited environments remains a challenge for many ISP's today. In these conditions, the image signal is dominated by the photon shot noise. Deep learning methods show promise in extracting the underlying image signal from the noise, but modern AI-based ISPs are too computationally complex to be realized as a fast and efficient hardware ISP. An ISP algorithm, BLADE2 has been designed, which leverages AI in a computationally conservative manner to demosaic and denoise low-light images. The original implementation of this algorihtm is in Python/PyTorch. This Thesis explores taking BLADE2 and implementing it on a general purpose GPU via a suite of Nvidia optimization toolkits, as well as a low-level implementation in C/C++, bringing the algorithm closer to FPGA realization. The GPU implementation demonstrated significant throughput gains and the C/C++ implementation demonstrated the feasibility of further hardware development.</p>

<p dir="ltr">In the ever-advancing landscape of technology, emerging technologies have emerged as<br>powerful catalysts for innovation across various domains. These technologies, situated at<br>the nexus of the physical and digital realms, hold tremendous potential for revolutionizing<br>industries, improving our quality of life, and addressing global challenges. Central to har-<br>nessing this potential is the efficient exchange of data and the processing of information, a<br>pivotal linchpin that underpins the success of emerging technologies.</p><p dir="ltr"><br>The thesis titled ”Empowering Emerging Technologies through Energy-Efficient Commu-<br>nication and In-Sensor Computing” delves into a critical facet of this technological revolution.<br>It explores the central role of energy-efficient communication and in-sensor computing in un-<br>locking the full potential of emerging technologies. This comprehensive exploration unfolds<br>across three distinct chapters, each addressing an essential aspect of the research undertaken.<br>The first two chapters are dedicated to the realm of wearable technology, where we delve<br>into the intricacies of Human Body Communication (HBC). Chapter 2 meticulously models<br>the human body, focusing on Galvanic excitation and termination, which are fundamental<br>to understanding communication within this unique domain. In Chapter 3, we introduce<br>a novel method employing resonance through the human body to enhance wearable device<br>functionality and efficiency, shedding light on its innovative potential.</p><p dir="ltr"><br>The fourth chapter takes us into the world of machine vision and computer vision, where<br>we unveil an ingenious solution—an ADC-less in-sensor image edge detection scheme. This<br>pioneering approach not only advances the field but also enables enhanced image processing<br>and analysis within sensors, thereby fostering the growth of machine vision applications.<br>This thesis represents a substantial contribution to the fields of HBC and in-sensor com-<br>puting. It models the Galvanic body channel, explores resonance-based power delivery and<br>communication, and demonstrates the importance of in-sensor image edge detection. Fur-<br>thermore, it presents a hardware-based CMOS image sensor capable of real-time edge image<br>extraction, enhancing computational efficiency while reducing latency.<br>As we embark on this intellectual journey, we invite the reader to delve deeper into the<br>realms of emerging technologies, energy-efficient communication, and in-sensor computing.</p><p dir="ltr">By the culmination of this thesis, it is our hope that the insights garnered from this re-<br>search will empower emerging technologies, inspire further innovation, and usher in a more<br>sustainable and technologically empowered future.</p>