This thesis aims to improve the performance of Deep Neural Networkss (DNNs) in Computer Vision tasks by optimizing the Image Signal Processor (ISP) parameters. The research investigates the use of simulated RAW images and the application of the DRL-ISP (Deep Reinforcement Learning for Image Signal Processor) method to enhance the accuracy and robustness of DNNs. The study begins by utilizing the Unpaired CycleR2R method to generate simulated RAW images from RGB images. The trained inverse ISP model successfully transforms the RGB images into simulated RAW images. The performance of DNNs in the Semantic Segmentation and Object Detection tasks is evaluated using both the simulated RAW and original RGB datasets. The results demonstrate the superiority of models trained on the original RGB dataset, highlighting the challenges and limitations of using simulated RAW images. Furthermore, the application of the DRL-ISP method for ISP parameter optimization improves Object Detection performance. This thesis provides valuable insights into the challenges and opportunities in utilizing simulated RAW data and optimizing ISP parameters for improved DNN performance in Computer Vision tasks. The findings contribute to the advancement of research in this field and lay the foundation for future investigations. / Syftet med denna uppsats är att förbättra Deep Neural Networkss (DNNs) prestanda i datorseendeuppgifter genom att optimera parametrarna för Image Signal Processing (ISP). I forskningen undersöks användningen av simulerade RAW-bilder och tillämpningen av DRL-ISP (Deep Reinforcement Learning for Image Signal Processing) för att förbättra DNN:s noggrannhet och robusthet. Undersökningen inleds med att använda metoden Unpaired CycleR2R för att generera simulerade RAW-bilder från RGB-bilder. Den tränade omvända ISP-modellen omvandlar framgångsrikt RGB-bilderna till simulerade RAW-bilder. DNN:s prestanda vid semantisk segmentering och objektdetektering utvärderas med hjälp av både simulerade RAW- och ursprungliga RGB-dataset. Resultaten visar att modeller som tränats på de ursprungliga RGB bilderna är överlägsna och belyser utmaningarna och begränsningarna med att använda simulerade RAW-bilder. Dessutom förbättrar tillämpningen av DRL-ISP-metoden för optimering av ISP-parametrar prestanda för objektdetektering. Den här uppsatsen ger värdefulla insikter i utmaningarna och möjligheterna med att använda simulerade RAW-data och optimera ISP-parametrar för förbättrad DNNprestanda i datorseendeuppgifter. Resultaten bidrar till att främja forskningen på detta område och lägger grunden för framtida undersökningar.

abstract: Generating real-world content for VR is challenging in terms of capturing and processing at high resolution and high frame-rates. The content needs to represent a truly immersive experience, where the user can look around in 360-degree view and perceive the depth of the scene. The existing solutions only capture and offload the compute load to the server. But offloading large amounts of raw camera feeds takes longer latencies and poses difficulties for real-time applications. By capturing and computing on the edge, we can closely integrate the systems and optimize for low latency. However, moving the traditional stitching algorithms to battery constrained device needs at least three orders of magnitude reduction in power. We believe that close integration of capture and compute stages will lead to reduced overall system power. We approach the problem by building a hardware prototype and characterize the end-to-end system bottlenecks of power and performance. The prototype has 6 IMX274 cameras and uses Nvidia Jetson TX2 development board for capture and computation. We found that capturing is bottlenecked by sensor power and data-rates across interfaces, whereas compute is limited by the total number of computations per frame. Our characterization shows that redundant capture and redundant computations lead to high power, huge memory footprint, and high latency. The existing systems lack hardware-software co-design aspects, leading to excessive data transfers across the interfaces and expensive computations within the individual subsystems. Finally, we propose mechanisms to optimize the system for low power and low latency. We emphasize the importance of co-design of different subsystems to reduce and reuse the data. For example, reusing the motion vectors of the ISP stage reduces the memory footprint of the stereo correspondence stage. Our estimates show that pipelining and parallelization on custom FPGA can achieve real time stitching. / Dissertation/Thesis / Prototype / Masters Thesis Electrical Engineering 2018

<p dir="ltr">A camera's image signal processor (ISP) is responsible for taking the mosaiced and noisy image signal from the image sensor and processing it such a way that an end-result image is produced that is informative and accurately captures the scene. Real-time video capture in photon-limited environments remains a challenge for many ISP's today. In these conditions, the image signal is dominated by the photon shot noise. Deep learning methods show promise in extracting the underlying image signal from the noise, but modern AI-based ISPs are too computationally complex to be realized as a fast and efficient hardware ISP. An ISP algorithm, BLADE2 has been designed, which leverages AI in a computationally conservative manner to demosaic and denoise low-light images. The original implementation of this algorihtm is in Python/PyTorch. This Thesis explores taking BLADE2 and implementing it on a general purpose GPU via a suite of Nvidia optimization toolkits, as well as a low-level implementation in C/C++, bringing the algorithm closer to FPGA realization. The GPU implementation demonstrated significant throughput gains and the C/C++ implementation demonstrated the feasibility of further hardware development.</p>