Past decades of auditory research have identified several acoustic features that influence perceptual organization of sound, in particular, the frequency of tones and the rate of presentation. One class of stimuli that have been intensively studied are sequences of tones that alternate in frequency. They are typically presented in patterns of repeating doublets ABAB… or repeating triplets ABA-ABA-... where the symbol “-” stands for a gap of silence between triplets repeats. The duration of each tone or silence is typically tens to hundreds of milliseconds, and listeners hearing the sequence perceive either one auditory object ("stream integration") or two separate auditory objects (“stream segregation”). Animal studies have characterized single- and multi- unit neural activity and event-related local field potentials while systematically varying frequency separation between tones (ΔF) or the presentation rate (PR). They found that the B tone responses in doublets were differentially suppressed with increasing PR and that the B tones responses in triplets decreased with larger ΔF. However, the neural mechanisms underlying these animal data have yet to be explained. In this study, we built an integrate-and-fire network model of the primary auditory cortex (AC) that accurately reproduced the experimental results. Then, we extended the model to account for basic spectro-temporal features of electrocorticography (ECoG) recordings from the posteriomedial part of the Heschl's gyrus (HGPM; cortical area equivalent to the AC of monkeys), obtained from humans listening to sequences of triplets ABA-. Finally, we constructed a firing rate reduced model of the proposed integrate-and-fire network and analyzed its dynamics as function of parameters. A large network of voltage-dependent leaky integrate-and-fire neurons (3600 excitatory, 900 inhibitory) was constructed to simulate neural activity from layers 3/4 of AC during streaming of tone triplets. Parameters describing synaptic and membrane properties were based on experimental data from early studies of AC. Network structure assumed spatially-dependent probability of connections and tonotopic organization. Subpopulations of neurons were tuned to different frequencies along the tonotopic map. In-silico recordings were performed during the presentation of long sequences of triplets and/or doublets. The network’s output was derived with two types of measurements in mind: spiking activity of individual neurons and/or local populations of neurons, and local field potentials. The network spiking neural activity reproduced reliably data reports, including dependence of responses to the B tone in triplets ABA- on stimulus parameter ΔF. Approximations of average evoked potentials (AEPs) from ECoG signals recorded at four depth contacts placed over human HGPM during auditory streaming of triplets were also obtained.

There are multiple factors that influence auditory steaming. Some, like frequency separation or rate of presentation, have effects that are well understood while others remain contentious. Human behavioural studies and event-related potential (ERP) studies have shown dissociation between a pre-attentive sound segregation process and an attention-dependent process in forming perceptual objects and streams. This thesis first presents a model that synthetises the processes involved in auditory object creation. It includes sensory feature extraction based on research by Bregman (1990), sensory feature binding through an oscillatory neural network based on work by Wang (1995; 1996; 1999; 2005; 2008), work by Itti and Koch (2001a) for the saliency map, and finally, work by Wrigley and Brown (2004) for the architecture of single feature processing streams, the inhibition of return of the activation and the attentional leaky integrate and fire neuron. The model was tested using stimuli and an experimental paradigm used by Carlyon, Cusack, Foxton and Robertson (2001). Several modifications were then implemented to the initial model to bring it closer to psychological and cognitive validity. The second part of the thesis furthers the knowledge available concerning the influence of the time spent attending to a task on streaming. Two deviant detection experiments using triplet stimuli are presented. The first experiment is a follow-up of Thompson, Carlyon and Cusack (2011) and replicated their behavioural findings, showing that the time spent attending to a task enhances streaming, and that deviant detection is easier when one stream is perceived. The ERP results showed double decisions markers indicating that subjects may have made their deviant detection based on the absence of the time delayed deviant and confirmed their decision with its later presence. The second experiment investigated the effect of the time spent attending to the task in presence of a continuity illusion on streaming. It was found that the presence of this illusion prevented streaming in such a way that the pattern of the triplet was strengthened through time instead of separated into two streams, and that the deviant detection was easier the longer the subjects attended to the sound sequence.

S'ha investigat molt tant els mecanismes neuronals bàsics de l'audició com l'organització psicològica de la percepció de la parla. Tanmateix, en ambdós temes n'hi ha una relativa escassetat en quant a modelització. Aquí describim dos treballs de modelització. Un d'ells proposa un nou mecanisme de millora de selectivitat de freqüències que explica resultats de experiments neurofisiològics investigant manifestacions de forward masking y sobretot auditory streaming en l'escorça auditiva principal (A1). El mecanisme funciona en una xarxa feed-forward amb depressió sináptica entre el tàlem y l'escorça, però mostrem que és robust a l'introducció d'una organització realista del circuit de A1, que per la seva banda explica cantitat de dades neurofisiològics. L'altre treball descriu un mecanisme candidat d'explicar la trobada en estudis psicofísics de diferències en la percepció de paraules entre bilinguës primerencs y simultànis. Simulant tasques de decisió lèxica y discriminació de fonemes, fortifiquem l'hipòtesi de que persones sovint exposades a variacions dialectals de paraules poden guardar aquestes en el seu lèxic, sense alterar representacions fonemàtiques . / Though much experimental research exists on both basic neural mechanisms of hearing and the psychological organization of language perception, there is a relative paucity of modelling work on these subjects. Here we describe two modelling efforts. One proposes a novel mechanism of frequency selectivity improvement that accounts for results of neurophysiological experiments investigating manifestations of forward masking and above all auditory streaming in the primary auditory cortex (A1). The mechanism works in a feed-forward network with depressing thalamocortical synapses, but is further showed to be robust to a realistic organization of the neural circuitry in A1, which accounts for a wealth of neurophysiological data. The other effort describes a candidate mechanism for explaining differences in word/non-word perception between early and simultaneous bilinguals found in psychophysical studies. By simulating lexical decision and phoneme discrimination tasks in an attractor neural network model, we strengthen the hypothesis that people often exposed to dialectal word variations can store these in their lexicons, without altering their phoneme representations. / Se ha investigado mucho tanto los mecanismos neuronales básicos de la audición como la organización psicológica de la percepción del habla. Sin embargo, en ambos temas hay una relativa escasez en cuanto a modelización. Aquí describimos dos trabajos de modelización. Uno propone un nuevo mecanismo de mejora de selectividad de frecuencias que explica resultados de experimentos neurofisiológicos investigando manifestaciones de forward masking y sobre todo auditory streaming en la corteza auditiva principal (A1). El mecanismo funciona en una red feed-forward con depresión sináptica entre el tálamo y la corteza, pero mostramos que es robusto a la introducción de una organización realista del circuito de A1, que a su vez explica cantidad de datos neurofisiológicos. El otro trabajo describe un mecanismo candidato de explicar el hallazgo en estudios psicofísicos de diferencias en la percepción de palabras entre bilinguës tempranos y simultáneos. Simulando tareas de decisión léxica y discriminación de fonemas, fortalecemos la hipótesis de que personas expuestas a menudo a variaciones dialectales de palabras pueden guardar éstas en su léxico, sin alterar representaciones fonémicas.

Perceptual multistability is a phenomenon that is mostly studied in all modalities separately. The phenomenon reveals fundamental principles of the perceptual system in the formation of an emerging cognitive representation in the consciousness. The momentary perceptual organizations evoked during the stimulation with ambiguous stimuli switches between several perceptual organizations or percepts: The auditory streaming stimulus in audition and the moving plaids stimulus in vision, elicit different at least two percepts that dominate awareness exclusively for a random phase or dominance duration before an inevitable switch to another percept occurs. The similarity in the perceptual experience has led to propose a global mechanism contributing to the perceptual multistability phenomena crossmodally. Contrary, the difference in the perceptual experience has led to propose a distributed mechanism that is modality-specific. The development of a hybrid model has synergized both approaches. We accumulate empirical evidence for the contribution of a global mechanism, albeit distributed mechanisms play an indispensable role in this cross-modal interplay. The overt report of the perceptual experience in our experiments is accompanied by the recording of objective, cognitive markers of the consciousness: Reflexive movements of the eyes, namely the dilation of the pupil and the optokinetic nystagmus, correlate with the unobservable perceptual switches and perceptual states respectively and have their neuronal rooting in the brainstem. We complement earlier findings on the sensitivity of the pupil to visual multistability: It was shown in two independent experiments that the pupil dilates at the time of reported perceptual switches in auditory multistability. A control condition on confounding effects from the reporting process confines the results. Endogenous, evoked internally by the unchanged stimulus ambiguity, and exogenous, evoked externally by the changes in the physical properties of the stimulus, perceptual switches could be discriminated based on the maximal amplitude of the dilation. The effect of exogenous perceptual has on the pupil were captured in a report and no-report task to detect confounding perceptual effects. In two additional studies, the moment-by-moment coupling and coupling properties of percepts between concurrent multistable processes in audition, evoked by auditory streaming, and in vision, evoked by moving plaids, were found crossmodally. In the last study, the externally induced percept in the visual multistable process was not relayed to the simultaneous auditory multistable process: Still, the observed general coupling is fragile but existent. The requirement for the investigation of a moment-by-moment coupling of the multistable perceptual processes was the application of a no-report paradigm in vision: The visual stimulus evokes an optokinetic nystagmus that has machine learnable different properties when following either of the two percepts. In combination with the manually reported auditory percept, attentional bottlenecks due to a parallel report were circumvented. The two main findings, the dilation of the pupil along reported auditory perceptual switches and the crossmodal coupling of percepts in bimodal audiovisual multistability, speak in favor of a partly global mechanism being involved in control of perceptual multistability; the global mechanism is incarcerated by the, partly independent, distributed competition of percepts on modality level. Potentially, supramodal attention-related modulations consolidate the outcome of locally distributed perceptual competition in all modalities.:COVER 1 BIBLIOGRAPHISCHE BESCHREIBUNG 2 ACKNOWLEDGEMENTS 3 CONTENTS 4 CHAPTER 1: Introduction 6 C1.1: Stability and uncertainty in perception 6 C1.2: Auditory, visual and audio-visual multistability 14 C1.3: Capturing the subjective perceptual experience 25 C1.4: Limitations of preceding studies, objectives, and outline of the Thesis 33 CHAPTER 2: Study 1 “Pupillometry in auditory multistability” 36 C2.1.1 Experiment 1: Introduction 36 C2.1.2 Experiment 1: Material and Methods 38 C2.1.3 Experiment 1: Data analysis 44 C2.1.4 Experiment 1: Results 48 C2.1.5 Experiment 1: Discussion 52 C2.2.1 Experiment 2: Introduction 54 C2.2.2 Experiment 2: Material and Methods 54 C2.2.3 Experiment 2: Data analysis 56 C2.2.4 Experiment 2: Results 57 C2.3 Experiment 1 & 2: Discussion 61 C2.4 Supplement Study 1 65 CHAPTER 3: Study 2 “Multimodal moment-by-moment coupling in perceptual bistability” 71 C3.1.1 Experiment 1: Introduction 71 C3.1.2 Experiment 1: Results 74 C3.1.3 Experiment 1: Discussion 80 C3.1.4 Experiment 1: Material and Methods 84 C3.1.5 Experiment 1: Data analysis 87 C3.2 Supplement Study 2 92 CHAPTER 4: Study 3 “Boundaries of bimodal coupling in perceptual bistability” 93 C4.1.1 Experiment 1: Introduction 93 C4.1.2 Experiment 1: Material and Methods 98 C4.1.3 Experiment 1: Data analysis 102 C4.1.4 Experiment 1: Results 108 C4.1.5 Experiment 1: Discussion 114 C4.2.1 Experiment 2: Introduction 116 C4.2.2 Experiment 2: Material and Methods 119 C4.2.3 Experiment 2: Data analysis 125 C4.2.4 Experiment 2: Results 133 C4.3 Experiment 1 & 2: Discussion 144 C4.4 Supplement Study 3 151 CHAPTER 5: General Discussion 154 C5.1 Significance for models of multistability and implications for the perceptual architecture 162 C5.2 Recommendations for future research 166 C5.3 Conclusion 168 REFERENCES 170 APPENDIX 186 A1: List of Figures 186 A2: List of Tables 188 A3: List of Abbreviations and Symbols 189

Sensory inputs can be ambiguous. A physically constant stimulus that induces several perceptual alternatives is called multistable. Many factors can influence perception. In this thesis I investigate factors that affect visual multistability. All presented studies use a pattern-component rivalry stimulus consisting of two gratings drifting in opposite directions (called the plaid stimulus). This induces an “integrated” perception of a moving plaid (the pattern) or a “segregated” perception of overlaid gratings (the components). One study (chapter 2) investigates parameter dependence of a plaid stimulus on perception, with particular emphasis on the first percept. Specifically, it addresses how the enclosed angle (opening angle) affects the perception at stimulus onset and during prolonged viewing. The effects that are shown persist even if the stimulus is rotated. On a more abstract level it is shown that percepts can influence each other over time (chapter 3) which emphasizes the importance of instructions and report mode. In particular, it relates to the decision which percepts are instructed to be reported at all as well as which percepts can be reported as separate entities and which are pooled into the same response option. A further abstract level (predictability of a stimulus change, chapter 5) shows that transferring effects from one modality to another modality (specifically from audition to vision) requires careful choice of stimulus parameters. In this context, we give considerations to the proposal for a wider usage of sequential stopping rules (SSR, chapter 4), especially in studies where effect sizes are hard to estimate a priori. This thesis contributes to the field of visual multistability by providing novel experimental insights into pattern-component rivalry and by linking these findings to data on sequential dependencies, to the optimization of experimental designs, and to models and results from another sensory modality.:Bibliographische Beschreibung 3 Acknowledgments 4 CONTENTS 5 Collaborations 7 List of Figures 8 List of Tables 8 1. Introduction 9 1.1. Tristability 10 1.2. Two or more interpretations? 11 1.3. Multistability in different modalities 12 1.3.1. Auditory multistability 12 1.3.2. Haptic multistability 13 1.3.3. Olfactory multistability 13 1.4. multistability with several interpretations 13 1.5. Measuring multistability 14 1.5.1. The optokinetic nystagmus 14 1.5.2. Pupillometry 15 1.5.3. Measuring auditory multistability 15 1.5.4. Crossmodal multistability 16 1.6. Factors governing multistability 16 1.6.1. Manipulations that do not involve the stimulus 16 1.6.2. Manipulation of the stimulus 17 1.6.2.1. Factors affecting the plaid stimulus 17 1.6.2.2. Factors affecting the auditory streaming stimulus 18 1.7. Goals of this thesis 18 1.7.1. Overview of the thesis 18 2. Parameter dependence in visual pattern-component rivalry at onset and during prolonged viewing 21 2.1. Introduction 21 2.2. Methods 24 2.2.1. Participants 24 2.2.2. Setup 24 2.2.3. Stimuli 25 2.2.4. Procedure 26 2.2.5. Analysis 27 2.2.6. (Generalized) linear mixed-effects models 30 2.3. Results 30 2.3.1. Experiment 1 30 2.3.1.1. Relative number of integrated percepts 31 2.3.1.2. Generalized linear mixed-effects model 32 2.3.1.3. Dominance durations 33 2.3.1.4. Linear mixed-effects models 33 2.3.1.5. Control: Disambiguated trials 33 2.3.1.6. Time course of percept reports at onset 34 2.3.1.7. Eye movements 35 2.3.2. Experiment 2 36 2.3.2.1. Relative number of percepts 36 2.3.2.2. Generalized linear mixed-effects model 37 2.3.2.3. Dominance durations 38 2.3.2.4. Linear mixed-effects model 38 2.3.2.5. Control: Disambiguated trials 40 2.3.2.6. Time course of percept reports at onset 42 2.3.2.7. Eye movements 44 2.4. Discussion 45 2.5. Appendix 49 2.5.1. Appendix A 49 3. Perceptual history 51 3.1. Markov chains 52 3.1.1. Markov chains of order 1 and 2 52 3.2. Testing for Markov chains 55 3.2.1. The method of Naber and colleagues (2010) 56 3.2.1.1. The method 56 3.2.1.2. Advantages and disadvantages of the method 56 3.2.2. Further methods for testing Markov chains 57 3.3. Summary and discussion 58 4. Sequential stopping rules 60 4.1. The COAST rule 61 4.2. The CLAST rule 61 4.3. The variable criteria sequential stopping rule 61 4.4. Discussion 62 4.5. Using the vcSSR when transferring an effect from audition to vision 64 5. Predictability in visual multistability 66 5.1. Pretests 66 5.2. Predictability effects in visual pattern-component rivalry 69 5.2.1. Introduction 69 5.2.2. Methods 71 5.2.2.1. Participants 71 5.2.2.2. Setup 72 5.2.2.3. Stimuli 73 5.2.2.4. Conditions 73 5.2.2.5. Design and procedure 73 5.2.2.6. Analysis 74 5.2.3. Results 75 5.2.3.1. Valid reports 75 5.2.3.2. Verification of reports by eye movements 76 5.2.3.3. Onset latency 76 5.2.3.4. Dominance durations 78 5.2.3.5. Relative dominance of the segregated percept 78 5.2.4. Discussion 78 6. General discussion 83 6.1. Reporting percepts 83 6.1.1. Providing two versus three response options 83 6.1.2. Stimuli with more than three percepts 84 6.1.3. When to pool percepts together and when not 84 6.1.4. Leaving out percepts 87 6.1.5. Measuring (unreported) percepts 88 6.2. Comparing influencing factors on different levels 88 6.3. The use of the vcSSR 90 6.4. Valid reports 90 6.5. Conclusion 93 References 94