The representation of incoming information, goals and the flexible processing of these are required for cognitive control. Efficient mechanisms are needed to decide when it is important that novel information enters working memory (WM) and when these WM ‘gates’ have to be closed. Compared to neural foundations of maintaining information in WM, considerably less is known about what neural mechanisms underlie the representational dynamics during WM gating. Using different EEG analysis methods, we trace the path of mental representations along the human cortex during WM gate opening and closing. We show temporally nested representational dynamics during WM gate opening and closing depending on multiple independent neural activity profiles. These activity profiles are attributable to a ventral stream-prefrontal cortex processing cascade. The representational dynamics start in the ventral stream during WM gate opening and WM gate closing before prefrontal cortical regions are modulated. A regional specific activity profile is shown within the prefrontal cortex depending on whether WM gates are opened or closed, matching overarching concepts of prefrontal cortex functions. The study closes an essential conceptual gap detailing the neural dynamics underlying how mental representations drive the WM gate to open or close to enable WM functions such as updating and maintenance.

Entgegen traditioneller Gesichtererkennungsmodelle konnte in einigen Studien gezeigt werden, dass die Erkennung des Emotionsausdrucks und der Bekanntheit interagieren. In dieser Dissertation wurde mit Hilfe von ereigniskorrelierten Potentialen untersucht, welche funktionalen Prozesse bei einer Interaktion moduliert werden. Teil I untersuchte, ob die Bekanntheit eines Gesichtes die Emotionsdiskrimination erleichtert. In mehreren Experimenten diskriminierten Versuchspersonen zwei Emotionen, die von bekannten und unbekannten Gesichtern praesentiert wurden . Dabei war die Entscheidung fuer persoenlich bekannte Gesichter mit froehlichem Ausdruck schneller und fehlerfreier. Dies zeigt sich in einer kuerzeren Latenz der P300 Komponente (Trend), welche die Dauer der Reizklassifikation auswies, sowie in einem verkuerzten Intervall zwischen Stimulus und Beginn des Lateralisierten Bereitschaftspotentials (S-LRP), welches die handspezifische Reaktionsauswahl anzeigt. Diese Befunde sprechen fuer eine Erleichterung der Emotionsdiskrimination auf spaeten perzeptuellen Verarbeitungsstufen bei persoenlich bekannten Gesichtern. In weiteren Experimenten mit oeffentlich bekannten, gelernten und unbekannten Gesichtern zeigte sich keine Erleichterung der Emotionsdiskrimination für bekannte Gesichter. Teil II untersuchte, ob es einen Einfluss des Emotionsausdrucks auf die Bekanntheitsentscheidung gibt. Eine Erleichterung zeigte sich fuer neutrale oder froehliche Emotionen nur bei persoenlich bekannten Gesichtern, nicht aber bei gelernten oder unbekannten Gesichtern. Sie spiegelt sich in einer Verkuerzung des S-LRP fuer persoenlich bekannte Gesichter wider, was eine Erleichterung der Reaktionsauswahl nahelegt. Zusammenfassend konnte gezeigt werden, dass eine Interaktion der Bekanntheit mit der Emotionserkennung unter bestimmten Bedingungen auftritt. In einer abschließenden Diskussion werden die experimentellen Ergebnisse in Beziehung gesetzt und in Hinblick auf bisherige Befunde diskutiert. / Contrasting traditional face recognition models previous research has revealed that the recognition of facial expressions and familiarity may not be independent. This dissertation attempts to localize this interaction within the information processing system by means of performance data and event-related potentials. Part I elucidated upon the question of whether there is an interaction between facial familiarity and the discrimination of facial expression. Participants had to discriminate two expressions which were displayed on familiar and unfamiliar faces. The discrimination was faster and less error prone for personally familiar faces displaying happiness. Results revealed a shorter peak latency for the P300 component (trend), reflecting stimulus categorization time, and for the onset of the lateralized readiness potential (S-LRP), reflecting the duration of pre-motor processes. A facilitation of perceptual stimulus categotization for personally familiar faces displaying happiness is suggested. The discrimination of expressions was not facilitated in further experiments using famous or experimentally familiarized, and unfamiliar faces. Part II raises the question of whether there is an interaction between facial expression and the discrimination of facial familiarity. In this task a facilitation was only observable for personally familiar faces displaying a neutral or happy expression, but not for experimentally familiarized, or unfamiliar faces. Event-related potentials reveal a shorter S-LRP interval for personally familiar faces, hence, suggesting a facilitated response selection stage. In summary, the results suggest that an interaction of facial familiarity and facial expression might be possible under some circumstances. Finally, the results are discussed in the context of possible interpretations, previous results, and face recognition models.

A natural auditory scene contains many sound sources each of which produces complex sounds. These sounds overlap and reach our ears at the same time, but they also change constantly. To still be able to follow the sound source of interest, the auditory system must decide where each individual tone belongs to and integrate this information over time. For well-controlled investigations on the mechanisms behind this challenging task, sound sources need to be simulated in the lab. This is mostly done with sine tones arranged in certain spectrotemporal patterns. The vast majority of studies simply interleave two sub-sequences of sine tones. Participants report how they perceive these sequences or they perform a task whose performance measure allows hints on how the scene was perceived. While many important insights have been gained with this procedure, the questions that can be addressed with it are limited and the commonly used response methods are partly susceptible to distortions or only indirect measures. The present thesis enlarged the complexity of the tone sequences and the diversity of perceptual measures used for investigations on auditory scene analysis. These changes are intended to open up new questions and give new perspectives on our knowledge about auditory scene analysis. In detail, the thesis established three-tone sequences as a tool for specific investigations on the perceptual foreground and background processing in complex auditory scenes. In addition, it modifies an already established approach for indirect measures of auditory perception in a way that enables detailed and univocal investigations on background processing. Finally, a new response method, namely a no-report method for auditory perception that might also serve as a method to validate subjective report measures, was developed. This new methodological approach uses eye movements as a measurement tool for auditory perception. With the aid of all these methodological improvements, the current thesis shows that auditory foreground formation is actually more complex than previously assumed since listeners hold more than one auditory source in the foreground without being forced to do so. In addition, it shows that the auditory system prefers a limited number of specific source configurations probably to avoid combinatorial explosion. Finally, the thesis indicates that the formation of the perceptual background is also quite complex since the auditory system holds perceptual organization alternatives in parallel that were basically assumed to be mutually exclusive. Thus, both the foreground and the background follow different rules than expected based on two-tone sequences. However, one finding seems to be true for both kinds of sequences: the impact of the tone pattern on the subjective perception is marginal, be it in two- or three-tone sequences. Regarding the no-report method for auditory perception, the thesis shows that eye movements and the reported auditory foreground formations were in good agreement and it seems like this approach indeed has the potential to become a first no-report measure for auditory perception.:Abstract 3 Acknowledgments 5 List of Figures 8 List of Tables 9 Collaborations 11 1 General Introduction 13 1.1 The auditory foreground 13 1.1.1 Attention and auditory scene analysis 13 1.1.2 Investigating auditory scene analysis with two-tone sequences 16 1.1.3 Multistability 18 1.2 The auditory background 21 1.2.1 Investigating auditory background processing 22 1.3 Measures of auditory perception 23 1.3.1 Report procedures 23 1.3.2 Performance-based measures 26 1.3.3 Psychophysiological measures 27 1.4 Summary and goals of the thesis 30 2 The auditory foreground 33 2.1 Study 1: Foreground formation in three-tone sequences 33 2.1.1 Abstract 33 2.1.2 Introduction 33 2.1.3 Methods 37 2.1.4 Results 43 2.1.5 Discussion 48 2.2 Study 2: Pattern effects in three-tone sequences 53 2.2.1 Abstract 53 2.2.2 Methods 53 2.2.3 Results 54 2.2.4 Discussion 58 2.3 Study 3: Pattern effects in two-tone sequences 59 2.3.1 Abstract 59 2.3.2 Introduction 59 2.3.3 General Methods 63 2.3.4 Experiment 1 – Methods and Results 65 2.3.5 Experiment 2 – Methods and Results 67 2.3.6 Experiment 3 – Methods and Results 70 2.3.7 Discussion 72 3 The auditory background 74 3.1 Study 4: Background formation in three-tone sequences 74 3.1.1 Abstract 74 3.1.2 Introduction 74 3.1.3 Methods 77 3.1.4 Results 82 3.1.5 Discussion 86 4 Audio-visual coupling for investigations on auditory perception 90 4.1 Study 5: Using Binocular Rivalry to tag auditory perception 90 4.1.1 Abstract 90 4.1.2 Introduction 90 4.1.3 Methods 92 4.1.4 Results 100 4.1.5 Discussion 108 5 General Discussion 113 5.1 Short review of the findings 113 5.2 The auditory foreground 114 5.2.1 Auditory foreground formation and attention theories 114 5.2.2 The role of tone pattern in foreground formation 116 5.2.3 Methodological considerations and continuation 117 5.3 The auditory background 118 5.3.1 Auditory object formation without attention 120 5.3.2 Multistability without attention 121 5.3.3 Methodological considerations and continuation 122 5.4 Auditory scene analysis by audio-visual coupling 124 5.4.1 Methodological considerations and continuation 124 5.5 Artificial listening situations and conclusions on natural hearing 126 6 Conclusions 128 References 130

Diese kumulative Dissertation umfasst zwei Studien zu den räumlichen und zeitlichen neuronalen Dynamiken des visuellen Arbeitsgedächtnisses (VA). Die erste Studie untersuchte, wie die oberflächlichen und tiefen Schichten des präfrontalen Kortex (PFC) zur Enkodierung, Aufrechterhaltung und zum Abruf von VA-Informationen bei unterschiedlichen Gedächtnisbelastungen beitragen. Die Ergebnisse zeigten, dass die oberflächlichen PFC-Schichten bei hoher Belastung während der Verzögerung und des Abrufs stärker aktiviert waren. Multivariate Decodierungstechniken zeigten eine dynamische neuronale Kodierung mit drei Clustern generalisierter Aktivitätsmuster in den Phasen der Enkodierung, Verzögerung und des Abrufs. Es gab jedoch keine Generalisierung zwischen diesen Phasen, was auf unterschiedliche neuronale Populationen für jede Phase hinweist. Die zweite Studie untersuchte die Mechanismen, die VA bei Ablenkungen aufrechterhalten. Dabei wurden Daten aus einer früheren Studie erneut analysiert, in der die Teilnehmer drei Arten von Aufgaben ausführten: eine mit einer leeren Verzögerungsphase, eine mit einem Orientierungsablenker und eine mit einem Rauschablenker. Die Studie analysierte die zeitliche Generalisierung der neuronalen Codes in Regionen des visuellen Kortex und prüfte, ob VA und Ablenker dieselben neuronalen Subräume nutzen. Die Ergebnisse zeigten eine dynamische Kodierung während der frühen und späten Verzögerungsphasen. Zudem wurden VA und der Orientierungsablenker in getrennten, nicht überlappenden Subräumen aufrechterhalten, was auf unterschiedliche neuronale Populationen für VA und Ablenker hindeutet. Zusammenfassend erweitert die Dissertation das Verständnis darüber, wie der PFC und visuelle Areale VA-Informationen aufrechterhalten, insbesondere unter unterschiedlichen Belastungen und Ablenkungen. Sie liefert zudem neue Ansätze zur Untersuchung der zeitlichen neuronalen Dynamik dieser Prozesse. / This cumulative thesis covers two scientific studies exploring the spatial and temporal neural dynamics of visual working memory (VWM) processes. The first study examined the contributions of the prefrontal cortex (PFC) layers—superficial and deep—during VWM encoding, maintenance, and retrieval under two memory load conditions. Results revealed heightened activation in the superficial layers of the PFC during high-load trials, particularly in the maintenance and retrieval phases. Using multivariate decoding techniques, the study assessed the temporal stability of neural codes distinguishing high- and low-load trials, identifying a dynamic code with three distinct clusters of generalization during encoding, delay, and retrieval phases. Notably, there was no generalization of neural patterns across these phases, suggesting distinct neural populations for each stage. The second study focused on the mechanisms enabling VWM maintenance in the presence of distractions. Reanalyzing prior data, the study examined VWM trials featuring either a blank delay, an orientation distractor, or a noise distractor. The study explored the temporal generalization of neural codes across visual cortex regions and whether VWM and distractors shared neural subspaces. Findings indicated dynamic neural coding during early and late memory delay periods. Additionally, VWM and orientation distractors were maintained in separate, non-overlapping subspaces, suggesting distinct neural populations for VWM and perceptual distractors. Collectively, this thesis enhances our understanding of how the PFC and visual areas support VWM maintenance and control, particularly under varying loads and distractions. It also introduces novel approaches for investigating the temporal neural dynamics underlying these processes.