This thesis focuses on the design of a foveated stereo video compression algorithm for visual telepresence applications. In a typical telepresence application, a user at the local site views real-time stereo video recorded and transmitted from a robotic camera platform located at a remote site. The robotic camera platform tracks the user's head motion producing the sensation of being present at the remote site. The design of the stereo video compression algorithm revolved around a fast spatio-temporal block-based motion estimation algorithm, with a foveated SPIHT algorithm used to compress and foveate the independent frames and error residues. Also, the redundancy between the left and right video streams was exploited by disparity compensation. Finally, position feedback from the robotic camera platform was used to perform global motion compensation, increasing the compression performance without raising computation requirements. The algorithm was analysed by introducing the above mentioned components separately. It was found that each component increased the compression rate significantly, producing compressed video with similar compression and quality as MPEG2. The implementation of the algorithm did not meet the real-time requirements on the experiment computers. However, the algorithm does not contain any intrinsic delays. Therefore, given faster processors or optimized software implementation, the design should be able to run in real-time.

This thesis focuses on the design of a foveated stereo video compression algorithm for visual telepresence applications. In a typical telepresence application, a user at the local site views real-time stereo video recorded and transmitted from a robotic camera platform located at a remote site. The robotic camera platform tracks the user's head motion producing the sensation of being present at the remote site. The design of the stereo video compression algorithm revolved around a fast spatio-temporal block-based motion estimation algorithm, with a foveated SPIHT algorithm used to compress and foveate the independent frames and error residues. Also, the redundancy between the left and right video streams was exploited by disparity compensation. Finally, position feedback from the robotic camera platform was used to perform global motion compensation, increasing the compression performance without raising computation requirements. The algorithm was analysed by introducing the above mentioned components separately. It was found that each component increased the compression rate significantly, producing compressed video with similar compression and quality as MPEG2. The implementation of the algorithm did not meet the real-time requirements on the experiment computers. However, the algorithm does not contain any intrinsic delays. Therefore, given faster processors or optimized software implementation, the design should be able to run in real-time.

Persistics is an advanced framework for processing wide-area aerial surveillance video. This framework handles the tasks of data collection, stitching of multi-sensor imagery, image registration and stabilization, motion tracking, and compression. As the technology for image sensor sizes improves, significant improvements in compression techniques are necessary in order to make full use of the data. Because the information of interest in such video is naturally moving, point-like targets, the applicability of foveated coding to the compression problem is an interesting question. Foveated coding, a compression technique that was designed to be perceptually optimal for the human visual system, has several components that are appropriate to the persistics compression problem. Foveation is applied in several different scenarios and methods to persistics data. As foveation can make good use of the persistics tracker data, a problem affecting tracker performance is explored as well. The multi-sensor stitching component of persistics can generate artifacts that reduce the effectiveness of the tracker. A method for characterizing, detecting, and correcting such artifacts is desirable. These three concepts are explored, and a method for detection is developed. Components of these algorithms were absorbed into a more general framework for artifact correction. / text

There are inherent tradeoffs in size, weight, and adaptability for many military imaging systems. In some cases, active optical devices provide new alternatives external to the traditional trade-space. Applications of interest include remote wide-area surveillance, tactical use of high altitude and space-based sensors, remote navigation of unmanned ground and air vehicles, and night vision systems.My goal is to demonstrate that by augmenting or replacing static dioptric, catatropic, or catadioptric optical designs, mechanical complexity can be reduced while either maintaining or increasing performance in three areas:(1). Spectral Resolution(2). Spatial Resolution(3). MagnificationI present here three different imaging systems to showcase these capabilities.

Laparoscopic surgery or minimally invasive surgery has great advantages compared with the conventional open surgery, such as reduced pain, shorter recovery time and lower infection rate. It has become a standard clinical procedure for cholecystectomy, appendectomy and splenectomy. The state-of-the-art laparoscopic technologies suffer from several significant limitations, one of which is the tradeoff of the limited instantaneous field of view (FOV) for high spatial resolution versus the wide FOV for situational awareness but with diminished spatial resolution. Standard laparoscopes lack the ability to acquire both wide-angle and high-resolution images simultaneously through a single scope. During the surgery, a trained assistant is required to manipulate the laparoscope. The practice of frequently maneuvering the laparoscope by a trained assistant can lead to poor or awkward ergonomic scenarios. This type of ergonomic conflicts imposes inherent challenges to laparoscopic procedures, and it is further aggravated with the introduction of single port access (SPA) techniques to laparoscopic surgery. SPA uses one combined surgical port for all instruments instead of using multiple ports in the abdominal wall. The grouping of ports raises a number of challenges, including the tunnel vision due to the in-line arrangement of instruments, poor triangulation of instruments, and the instrument collision due to the close proximity to other surgical devices. A multi-resolution foveated laparoscope (MRFL) was proposed to address those limitations of the current laparoscopic surgery. The MRFL is able to simultaneously capture a wide-angle view for situational awareness and a high-resolution zoomed-in view for fine details. The high-resolution view can be scanned and registered anywhere within the wide-angle view, enabled by a 2D optical scanning mechanism. In addition, the high-resolution probe has optical zoom and autofocus capabilities, so that the field coverage can be dynamically varied while keep the same focus distance as the wide-angle probe. Moreover, the MRFL has a large working distance compared with the standard laparoscopes, the wide-angle probe has more than 8x field coverage than a standard laparoscope. On the other hand, the high-resolution probe has 3x spatial resolution than a standard one. These versatile capabilities are anticipated to have significant impacts on the diagnostic, clinical and technical aspects of minimally invasive surgery. In this dissertation, the development of the multi-resolution foveated laparoscope was discussed in detail. Starting from the refinement of the 1st order specifications, system configurations, and initial prototype demonstration, a customized dual-view MRFL system with fixed optical magnifications was developed and demonstrated. After the in-vivo test of the first generation prototype of the MRFL, further improvement was made on the high-resolution probe by adding an optical zoom and auto-focusing capability. The optical design, implementation and experimental validation of the MRFL prototypes were presented and discussed in detail.

The development of compact imaging systems capable of transmitting high-resolution images in real-time while covering a wide field-of-view (FOV) is critical in a variety of military and civilian applications: surveillance, threat detection, target acquisition, tracking, remote operation of unmanned vehicles, etc. Recently, optical foveated imaging using liquid crystal (LC) spatial light modulators (SLM) has received considerable attention as a potential approach to reducing size and complexity in fast wide-angle lenses. The fundamental concept behind optical foveated imaging is reducing the number of elements in a fast wide-angle lens by placing a phase SLM at the pupil stop to dynamically compensate aberrations left uncorrected by the optical design. In the recent years, considerable research and development has been conducted in the field of optical foveated imaging based on the LC SLM technology, and several foveated optical systems (FOS) prototypes have been built. However, most research has been focused so far on the experimental demonstration of the basic concept using off the shelf components, without much concern for the practicality or the optical performance of the systems. Published results quantify only the aberration correction capabilities of the FOS, often claiming diffraction limited performance at the region of interest (ROI). However, these results have continually overlooked diffraction effects on the zero-order efficiency and the image quality. The research work presented in this dissertation covers the methods and results of a detailed theoretical research study on the diffraction analysis, image quality, design, and optimization of fast wide-angle FOSs based on the current transmissive LC SLM technology. The amplitude and phase diffraction effects caused by the pixelated aperture of the SLM are explained and quantified, revealing fundamental limitations imposed by the current transmissive LC SLM technology. As a part of this study, five different fast wide-angle lens designs that can be used to build practical FOSs were developed, revealing additional challenges specific to the optical design of fast wide-angle systems, such as controlling the relative illumination, distortion, and distribution of aberrations across a wide FOV. One of the lens design examples was chosen as a study case to demonstrate the design, analysis, and optimization of a practical wide-angle FOS based on the current state-of-the-art transmissive LC SLM technology. The effects of fabrication and assembly tolerances on the image quality of fast wide-angle FOSs were also investigated, revealing the sensitivity of these fast well-corrected optical systems to manufacturing errors. The theoretical study presented in this dissertation sets fundamental analysis, design, and optimization guidelines for future developments in fast wide-angle FOSs based on transmissive SLM devices.

One way of increasing 3D rendering performance is the use of foveated rendering. In this thesis a novel foveated rendering technique called gaze contingent depth of field tessellation (GC DOF tessellation) is proposed. Tessellation is the process of subdividing geometry to increase detail. The technique works by applying tessellation to all objects within the focal plane, gradually decreasing tessellation levels as applied blur increases. As the user moves their gaze the focal plane shifts and objects go from blurry to sharp at the same time as the fidelity of the object increases. This can help hide the pops that occur as objects change shape. The technique was evaluated in a user study with 32 participants. For the evaluated scene the technique helped reduce the number of primitives rendered by around 70 % and frame time by around 9 % compared to using full adaptive tessellation. The user study showed that as the level of blur increased the detection rate for pops decreased, suggesting that the technique could be used to hide pops that occur due to tessellation. However, further research is needed to solidify these findings. / Ett sätt att öka renderingsprestanda i 3D applikationer är att använda foveated rendering. I denna uppsats presenteras en ny foveated rendering-teknik som kallas gaze contingent depth of field tessellering (GC DOF tessellering). Tessellering är när geometri delas i mindre delar för att öka detaljrikedom. Tekniken fungerar genom att applicera tessellering på alla objekt i fokalplanet och gradvis minska tesselleringsnivåer när oskärpan ökar. När användaren flyttar sin blick så flyttas fokalplanet och suddiga objekt blir skarpa samtidigt som detaljrikedomen i objektet ökar. Det kan hjälpa till att dölja de ’pops’ som uppstår när objekt ändrar form. Tekniken utvärderades i en användarstudie med 32 del- tagare. I den utvärderade scenen visade sig tekniken minska antalet renderade primitiver med ca 70 % och minska renderingstiden med ca 9 % jämfört med att använda full adaptiv tessellering. Användarstudien visade att när oskärpa ökade så minskade antalet som sa sig se ’pops’, vilket tyder på att tekniken kan användas för att dölja de ’pops’ som uppstår på grund av tessellering. Det behövs dock ytterligare forskning för att säkerställa dessa fynd.

Electronic imaging technologies are faced with the challenge of power consumption when transmitting large amounts of image data from the acquisition imager to the display or processing devices. This is especially a concern for portable applications, and becomes more prominent in increasingly high-resolution, high-frame rate imagers. Therefore, new sampling techniques are needed to minimize transmitted data, while maximizing the conveyed image information. <br /><br /> From this point of view, two approaches have been proposed and implemented in this thesis: <ol> <li> A system-level approach, in which the classical 1D row sampling CMOS imager is modified to a 2D ring sampling pyramidal architecture, using the same standard three transistor (3T) active pixel sensor (APS). </li> <li> A device-level approach, in which the classical orthogonal architecture has been preserved while altering the APS device structure, to design an expandable multiresolution image sensor. </li> </ol> A new scanning scheme has been suggested for the pyramidal image sensor, resulting in an intrascene foveated dynamic range (FDR) similar in profile to that of the human eye. In this scheme, the inner rings of the imager have a higher dynamic range than the outer rings. The pyramidal imager transmits the sampled image through 8 parallel output channels, allowing higher frame rates. The human eye is known to have less sensitivity to oblique contrast. Using this fact on the typical oblique distribution of fixed pattern noise, we demonstrate lower perception of this noise than the orthogonal FPN distribution of classical CMOS imagers. <br /><br /> The multiresolution image sensor principle is based on averaging regions of low interest from frame-sampled image kernels. One pixel is read from each kernel while keeping pixels in the region of interest at their high resolution. This significantly reduces the transferred data and increases the frame rate. Such architecture allows for programmability and expandability of multiresolution imaging applications.

Electronic imaging technologies are faced with the challenge of power consumption when transmitting large amounts of image data from the acquisition imager to the display or processing devices. This is especially a concern for portable applications, and becomes more prominent in increasingly high-resolution, high-frame rate imagers. Therefore, new sampling techniques are needed to minimize transmitted data, while maximizing the conveyed image information. <br /><br /> From this point of view, two approaches have been proposed and implemented in this thesis: <ol> <li> A system-level approach, in which the classical 1D row sampling CMOS imager is modified to a 2D ring sampling pyramidal architecture, using the same standard three transistor (3T) active pixel sensor (APS). </li> <li> A device-level approach, in which the classical orthogonal architecture has been preserved while altering the APS device structure, to design an expandable multiresolution image sensor. </li> </ol> A new scanning scheme has been suggested for the pyramidal image sensor, resulting in an intrascene foveated dynamic range (FDR) similar in profile to that of the human eye. In this scheme, the inner rings of the imager have a higher dynamic range than the outer rings. The pyramidal imager transmits the sampled image through 8 parallel output channels, allowing higher frame rates. The human eye is known to have less sensitivity to oblique contrast. Using this fact on the typical oblique distribution of fixed pattern noise, we demonstrate lower perception of this noise than the orthogonal FPN distribution of classical CMOS imagers. <br /><br /> The multiresolution image sensor principle is based on averaging regions of low interest from frame-sampled image kernels. One pixel is read from each kernel while keeping pixels in the region of interest at their high resolution. This significantly reduces the transferred data and increases the frame rate. Such architecture allows for programmability and expandability of multiresolution imaging applications.