Neural circuits that mediate complex behaviors contain several cell types, yet little is known about the role of each cell type within these circuits. An example problem in the auditory domain is how cortical circuits process complex natural sounds amidst competing stimuli from different spatial sources, also known as the "cocktail party effect". A pre-study recorded cortical responses in songbirds and found that neurons are broadly tuned to sound location when only one sound is present; when a competing stimulus is introduced, neurons sharpen their spatial tuning. These results were visualized by "spatial grids" that show preferred sound source locations in the presence of competing stimuli. These experiments motivated a computational model which proposed that lateral inhibition between spatially tuned channels within cortex is a key mechanism for spatial sound segregation. Cortical circuits are known to contain both excitatory cells and subpopulations of inhibitory interneurons, the roles of which can be probed in vivo with optogenetic techniques. Motivated by these past results and the optogenetic tools readily available in the mouse model, I present experimental and computational approaches in uncovering the cortical circuits that aid in solving the cocktail party problem in mouse auditory cortex (ACx). First, I probe the role of parvalbumin-expressing (PV) interneurons in solving the cocktail party problem using optogenetic and electrophysiological techniques. I found that mice exhibit similar cortical spatial grids as in songbirds, and optogenetic suppression of PV neurons reduces discriminability between dynamic sounds in both clean and masked presentations of spatially distributed stimuli. To mechanistically explain these results, I create a two-layer computational model of ACx with PV subpopulations that respond to distinct temporal stimulus features. I found that differentially weighing inhibition from these interneurons captures the range of neural discriminability performances found in cortex and the effects of optogenetically suppressing PV cells. Next, I analyze the population coding of neurons during the cocktail party problem. Here, I found that a relatively compact and diverse population of neurons within cortex is sufficient for encoding sounds from competing spatial locations. Finally, I determine how changes in behavioral states via tone extinction tasks affect activity in ACx and medial prefrontal cortex (mPFC). Results show that alpha and beta oscillations (8-18 Hz) in response to unrewarded tones exhibited immediate and robust increases in both regions prior to behavioral changes. When subjects learned to suppress behavioral responses, coherence at 8-18 Hz between ACx and mPFC was enhanced and spiking at ACx in response to the unrewarded tone was decreased. Taken together, this work advances the knowledge of both bottom-up and top-down circuit mechanisms underlying the cocktail party problem. / 2024-01-16T00:00:00Z

Previous research measuring visual short-term memory (VSTM) suggests that the capacity for representing the layout of objects is fairly high. In four experiments, we further explored the capacity of VSTM for layout of objects, using the change detection method. In Experiment 1, participants retained most of the elements in displays of 4 to 8 elements. In Experiments 2 and 3, with up to 20 elements, participants retained many of them, reaching a capacity of 13.4 stimulus elements. In Experiment 4, participants retained much of a complex naturalistic scene. In most cases, increasing display size caused only modest reductions in performance, consistent with the idea of configural, variable-resolution grouping. The results indicate that participants can retain a substantial amount of scene layout information (objects and locations) in short-term memory. We propose that this is a case of remote visual understanding, where observers’ ability to integrate information from a scene is paramount.

When analyzing complex scenes, humans often focus their attention on an object at a particular spatial location in the presence of background noises and irrelevant visual objects. The ability to decode the attended spatial location would facilitate brain computer interfaces (BCI) for complex scene analysis. Here, we tested two different neuroimaging technologies and investigated their capability to decode audio-visual spatial attention in the presence of competing stimuli from multiple locations. For functional near-infrared spectroscopy (fNIRS), we targeted dorsal frontoparietal network including frontal eye field (FEF) and intra-parietal sulcus (IPS) as well as superior temporal gyrus/planum temporal (STG/PT). They all were shown in previous functional magnetic resonance imaging (fMRI) studies to be activated by auditory, visual, or audio-visual spatial tasks. We found that fNIRS provides robust decoding of attended spatial locations for most participants and correlates with behavioral performance. Moreover, we found that FEF makes a large contribution to decoding performance. Surprisingly, the performance was significantly above chance level 1s after cue onset, which is well before the peak of the fNIRS response. For electroencephalography (EEG), while there are several successful EEG-based algorithms, to date, all of them focused exclusively on auditory modality where eye-related artifacts are minimized or controlled. Successful integration into a more ecological typical usage requires careful consideration for eye-related artifacts which are inevitable. We showed that fast and reliable decoding can be done with or without ocular-removal algorithm. Our results show that EEG and fNIRS are promising platforms for compact, wearable technologies that could be applied to decode attended spatial location and reveal contributions of specific brain regions during complex scene analysis.

This dissertation focuses on the many issues that arise from the visual rendering problem. Of primary consideration is light transport simulation, which is known to be computationally expensive. Monte Carlo methods represent a simple and general class of algorithms often used for light transport computation. Unfortunately, the images resulting from Monte Carlo approaches generally suffer from visually unacceptable noise artifacts. The result of any light transport simulation is, by its very nature, an image of high dynamic range (HDR). This leads to the issues of the display of such images on conventional low dynamic range devices and the development of data compression algorithms to store and recover the corresponding large amounts of detail found in HDR images. This dissertation presents our contributions relevant to these issues. Our contributions to high dynamic range image processing include tone mapping and data compression algorithms. This research proposes and shows the efficacy of a novel level set based tone mapping method that preserves visual details in the display of high dynamic range images on low dynamic range display devices. The level set method is used to extract the high frequency information from HDR images. The details are then added to the range compressed low frequency information to reconstruct a visually accurate low dynamic range version of the image. Additional challenges associated with high dynamic range images include the requirements to reduce excessively large amounts of storage and transmission time. To alleviate these problems, this research presents two methods for efficient high dynamic range image data compression. One is based on the classical JPEG compression. It first converts the raw image into RGBE representation, and then sends the color base and common exponent to classical discrete cosine transform based compression and lossless compression, respectively. The other is based on the wavelet transformation. It first transforms the raw image data into the logarithmic domain, then quantizes the logarithmic data into the integer domain, and finally applies the wavelet based JPEG2000 encoder for entropy compression and bit stream truncation to meet the desired bit rate requirement. We believe that these and similar such contributions will make a wide application of high dynamic range images possible. The contributions to light transport simulation include Monte Carlo noise reduction, dynamic object rendering and complex scene rendering. Monte Carlo noise is an inescapable artifact in synthetic images rendered using stochastic algorithm. This dissertation proposes two noise reduction algorithms to obtain high quality synthetic images. The first one models the distribution of noise in the wavelet domain using a Laplacian function, and then suppresses the noise using a Bayesian method. The other extends the bilateral filtering method to reduce all types of Monte Carlo noise in a unified way. All our methods reduce Monte Carlo noise effectively. Rendering of dynamic objects adds more dimension to the expensive light transport simulation issue. This dissertation presents a pre-computation based method. It pre-computes the surface radiance for each basis lighting and animation key frame, and then renders the objects by synthesizing the pre-computed data in real-time. Realistic rendering of complex scenes is computationally expensive. This research proposes a novel 3D space subdivision method, which leads to a new rendering framework. The light is first distributed to each local region to form local light fields, which are then used to illuminate the local scenes. The method allows us to render complex scenes at interactive frame rates. Rendering has important applications in mixed reality. Consistent lighting and shadows between real scenes and virtual scenes are important features of visual integration. The dissertation proposes to render the virtual objects by irradiance rendering using live captured environmental lighting. This research also introduces a virtual shadow generation method that computes shadows cast by virtual objects to the real background. We finally conclude the dissertation by discussing a number of future directions for rendering research, and presenting our proposed approaches.