Studies that explore integration of visual, auditory or vestibular cues, are derived from stimulus detection and discrimination tasks in which stimuli are selective and controlled. Multisensory processing is not as well understood in more dynamic and realistic tasks such as driving. As visual information is the dominant source of information when controlling a vehicle, we were interested in the contribution of auditory and physical motion (vestibular and proprioceptive) information to vehicle control. The simulated environment consisted of a straight, two-lane road and the task was to drive in the center of the right lane and maintain a constant speed, slowing down for occasional speed bumps. We examined differences in driving performance under four sets of sensory cues: visual only, visual and auditory, visual and physical motion, and visual, auditory and physical motion. The quality of visual information was manipulated across two experiments. In Experiment 1, participants drove in daylight in sunny weather, providing excellent visual information. In Experiment 2, visual information was compromised by providing dark and stormy weather conditions. In both experiments we observed an advantage of multisensory information, an effect that was enhanced when visual information was compromised. Auditory cues were especially effective in improving driver control. / Thesis / Master of Science (MSc) / Multisensory processing (combining information from different sensory systems) is not well understood in realistic tasks such as driving. A simulated environment consisted of a straight, two-lane road was used for this study. The task was to drive in the center of the right lane and maintain a constant speed, slowing down for occasional speed bumps. We examined differences in driving performance under four sets of sensory cues: visual only, visual and auditory, visual and physical motion, and visual, auditory and physical motion. The visual information was manipulated across two experiments: first, participants drove in daylight in sunny weather, providing excellent visual information. Next, visual information was compromised by providing dark and stormy weather conditions. In both experiments we observed an advantage of multisensory information, an effect that was enhanced when visual information was compromised. Auditory cues were especially effective in improving driver control.

Multisensory processing in the brain underlies a wide variety of perceptual phenomena, but little is known about the underlying mechanisms of how multisensory neurons are generated and how the neurons integrate sensory information from environmental events. This lack of knowledge is due to the difficulty of biological experiments to manipulate and test the characteristics of multisensory processing. By using a computational model of multisensory processing this research seeks to provide insight into the mechanisms of multisensory processing. From a computational perspective, modeling of brain functions involves not only the computational model itself but also the conceptual definition of the brain functions, the analysis of correspondence between the model and the brain, and the generation of new biologically plausible insights and hypotheses. In this research, the multisensory processing is conceptually defined as the effect of multisensory convergence on the generation of multisensory neurons and their integrated response products, i.e., multisensory integration. Thus, the computational model is the implementation of the multisensory convergence and the simulation of the neural processing acting upon the convergence. Next, the most important step in the modeling is analysis of how well the model represents the target, i.e., brain function. It is also related to validation of the model. One of the intuitive and powerful ways of validating the model is to apply methods standard to neuroscience for analyzing the results obtained from the model. In addition, methods such as statistical and graph-theoretical analyses are used to confirm the similarity between the model and the brain. This research takes both approaches to provide analyses from many different perspectives. Finally, the model and its simulations provide insight into multisensory processing, generating plausible hypotheses, which will need to be confirmed by real experimentation.

In daily life, humans are bombarded with visual input. Yet, their attentional capacities for processing this input are severely limited. Several studies, including my own, have investigated factors that influence these attentional limitations and have identified methods to circumvent them. In the present thesis, I provide a review of my own and others' findings. I first review studies that have demonstrated limitations of visuospatial attention and investigated physiological correlates of these limitations. I then turn to studies in multisensory research that have explored whether limitations in visuospatial attention can be circumvented by distributing information processing across several sensory modalities. Finally, I discuss research from the field of joint action that has investigated how limitations of visuospatial attention can be circumvented by distributing task demands across people and providing them with multisensory input. Based on the reviewed studies, I conclude that limitations of visuospatial attention can be circumvented by distributing attentional processing across sensory modalities when tasks involve spatial as well as object-based attentional processing. However, if only spatial attentional processing is required, limitations of visuospatial attention cannot be circumvented by distributing attentional processing. These findings from multisensory research are applicable to visuospatial tasks that are performed jointly by two individuals. That is, in a joint visuospatial task that does require object-based as well as spatial attentional processing, joint performance is facilitated when task demands are distributed across sensory modalities. Future research could further investigate how applying findings from multisensory research to joint action research may potentially facilitate joint performance. Generally, these findings are applicable to real-world scenarios such as aviation or car-driving to circumvent limitations of visuospatial attention.

Children who struggle to learn math are often identified by their poor performance on common math learning activities, such as number line estimations. While such behavioral assessments are useful in the classroom, naturalistic neuroimaging of children engaged in real-world math learning activities has the potential to identify concurrent behavioral and neurological correlates to poor math performance. Such correlates may help pinpoint effective teaching strategies for atypical learners, and may highlight instructional methods that elicit typical neurological response patterns to such activities. For example, multisensory stimulation that contains information about number enhances infants' and preschool children's behavioral performance on many numerical tasks and has been shown to elicit neural activation in areas related to number processing and decision-making. Thus, when applied to math teaching tools, multisensory stimulation may provide a platform through which both behavioral and neural math-related processes may be enhanced. Common approaches to neuroimaging of math processing lack ecological validity and are often not analogous to real-world learning activities. However, because of its liberal tolerance of movement, near-infrared spectroscopy (NIRS) provides an ideal platform for such studies. Here, NIRS is used to provide the first concurrent examination of neurological and behavioral data from number line estimation performance within children and adults. Moreover, in an effort to observe the behavioral and neurological benefits to number line estimations that may arise from multisensory stimulation, differential feedback (i.e., visual, auditory, or audiovisual) about estimation performance is provided throughout a portion of the task. Results suggest behavioral and neural performance is enhanced by feedback. Moreover, significant effects of age suggest young children show greater neurological response to feedback, and increase in task difficulty resulted in decreased behavioral performance and increased neurological activation associated with mathematical processing. Thus, typical math learners effectively recruit areas of the brain known to process number when math activities become increasingly difficult. Data inform understanding typical behavioral and neural responses to real-world math learning tasks, and may prove useful in triangulating signatures of atypical math learning. Moreover, results demonstrate the utility of NIRS as a platform to provide simultaneous neurological and behavioral data during naturalistic math learning activities.

Les troubles du spectre autistique (TSA) sont actuellement caractérisés par une triade d'altérations, incluant un dysfonctionnement social, des déficits de communication et des comportements répétitifs. L'intégration simultanée de multiples sens est cruciale dans la vie quotidienne puisqu'elle permet la création d'un percept unifié. De façon similaire, l'allocation d'attention à de multiples stimuli simultanés est critique pour le traitement de l'information environnementale dynamique. Dans l'interaction quotidienne avec l'environnement, le traitement sensoriel et les fonctions attentionnelles sont des composantes de base dans le développement typique (DT). Bien qu'ils ne fassent pas partie des critères diagnostiques actuels, les difficultés dans les fonctions attentionnelles et le traitement sensoriel sont très courants parmi les personnes autistes. Pour cela, la présente thèse évalue ces fonctions dans deux études séparées. La première étude est fondée sur la prémisse que des altérations dans le traitement sensoriel de base pourraient être à l'origine des comportements sensoriels atypiques chez les TSA, tel que proposé par des théories actuelles des TSA. Nous avons conçu une tâche de discrimination de taille intermodale, afin d'investiguer l'intégrité et la trajectoire développementale de l'information visuo-tactile chez les enfants avec un TSA (N = 21, âgés de 6 à18 ans), en comparaison à des enfants à DT, appariés sur l’âge et le QI de performance. Dans une tâche à choix forcé à deux alternatives simultanées, les participants devaient émettre un jugement sur la taille de deux stimuli, basé sur des inputs unisensoriels (visuels ou tactiles) ou multisensoriels (visuo-tactiles). Des seuils différentiels ont évalué la plus petite différence à laquelle les participants ont été capables de faire la discrimination de taille. Les enfants avec un TSA ont montré une performance diminuée et pas d'effet de maturation aussi bien dans les conditions unisensorielles que multisensorielles, comparativement aux participants à DT. Notre première étude étend donc des résultats précédents d'altérations dans le traitement multisensoriel chez les TSA au domaine visuo-tactile. Dans notre deuxième étude, nous avions évalué les capacités de poursuite multiple d’objets dans l’espace (3D-Multiple Object Tracking (3D-MOT)) chez des adultes autistes (N = 15, âgés de 18 à 33 ans), comparés à des participants contrôles appariés sur l'âge et le QI, qui devaient suivre une ou trois cibles en mouvement parmi des distracteurs dans un environnement de réalité virtuelle. Les performances ont été mesurées par des seuils de vitesse, qui évaluent la plus grande vitesse à laquelle des observateurs sont capables de suivre des objets en mouvement. Les individus autistes ont montré des seuils de vitesse réduits dans l'ensemble, peu importe le nombre d'objets à suivre. Ces résultats étendent des résultats antérieurs d'altérations au niveau des mécanismes d'attention en autisme quant à l'allocation simultanée de l'attention envers des endroits multiples. Pris ensemble, les résultats de nos deux études révèlent donc des altérations chez les TSA quant au traitement simultané d'événements multiples, que ce soit dans une modalité ou à travers des modalités, ce qui peut avoir des implications importantes au niveau de la présentation clinique de cette condition. / Autism spectrum disorders (ASD) are currently characterized by a triad of impairments including social dysfunction, communication deficits and perseverative behaviours. The simultaneous integration of multiple senses is crucial in everyday life as it allows for the creation of a unified percept. Similarly, the allocation of attention to multiple events at the same time is critical in the processing of dynamic environmental information. In daily interactions with the environment, both sensory processing as well as attentional functions are building blocks to typical development (TD). Although not part of the current diagnostic criteria, difficulties with attention functions and sensory processing are very common among autistic persons. The present thesis therefore examined both these functions in two separate studies. The first study is based on the premise that alterations in basic sensory processing might underlie atypical sensory behaviours in ASD, as proposed by current theories of ASD. We conceived a cross-modal size discrimination task to assess the integrity and developmental course of visuo-tactile information in children with ASD (N = 21, aged 6-18 years), compared to age- and performance IQ-matched children with TD. In a simultaneous two-alternative forced-choice task, participants were asked to make a judgement on the size of two stimuli, based on unisensory (visual or tactile) or multisensory (visuo-tactile) inputs. Difference thresholds evaluated the smallest difference at which participants were capable to discriminate size. Children with ASD showed diminished performance and no maturational effects in both unisensory and multisensory conditions, compared to TD participants. Our first study therefore extends previous results of alterations in multisensory processing in ASD to the visuo-tactile domain. In our second study, we evaluated 3D-Multiple Object Tracking (3D-MOT) capacities in autistic adults (N = 15, aged 18-33 years), compared to age- and IQ-matched control participants, who were asked to track one or three moving targets amongst a set of distracters in a virtual reality environment. Performances were measured based on speed thresholds, which evaluates the greatest speed at which observers are capable of successfully tracking moving objects. Autistic individuals displayed overall reduced speed thresholds, whatever the number of spheres to track. These findings extend previous results of altered attention mechanisms in autism with regards to the simultaneous allocation of attention to multiple areas. Together, the findings of our two studies reveal alterations in ASD with regards to the processing of multiple events at the same time, be it within one modality or across modalities, which may have important implications for the clinical presentation of this condition.

Les troubles du spectre autistique (TSA) sont actuellement caractérisés par une triade d'altérations, incluant un dysfonctionnement social, des déficits de communication et des comportements répétitifs. L'intégration simultanée de multiples sens est cruciale dans la vie quotidienne puisqu'elle permet la création d'un percept unifié. De façon similaire, l'allocation d'attention à de multiples stimuli simultanés est critique pour le traitement de l'information environnementale dynamique. Dans l'interaction quotidienne avec l'environnement, le traitement sensoriel et les fonctions attentionnelles sont des composantes de base dans le développement typique (DT). Bien qu'ils ne fassent pas partie des critères diagnostiques actuels, les difficultés dans les fonctions attentionnelles et le traitement sensoriel sont très courants parmi les personnes autistes. Pour cela, la présente thèse évalue ces fonctions dans deux études séparées. La première étude est fondée sur la prémisse que des altérations dans le traitement sensoriel de base pourraient être à l'origine des comportements sensoriels atypiques chez les TSA, tel que proposé par des théories actuelles des TSA. Nous avons conçu une tâche de discrimination de taille intermodale, afin d'investiguer l'intégrité et la trajectoire développementale de l'information visuo-tactile chez les enfants avec un TSA (N = 21, âgés de 6 à18 ans), en comparaison à des enfants à DT, appariés sur l’âge et le QI de performance. Dans une tâche à choix forcé à deux alternatives simultanées, les participants devaient émettre un jugement sur la taille de deux stimuli, basé sur des inputs unisensoriels (visuels ou tactiles) ou multisensoriels (visuo-tactiles). Des seuils différentiels ont évalué la plus petite différence à laquelle les participants ont été capables de faire la discrimination de taille. Les enfants avec un TSA ont montré une performance diminuée et pas d'effet de maturation aussi bien dans les conditions unisensorielles que multisensorielles, comparativement aux participants à DT. Notre première étude étend donc des résultats précédents d'altérations dans le traitement multisensoriel chez les TSA au domaine visuo-tactile. Dans notre deuxième étude, nous avions évalué les capacités de poursuite multiple d’objets dans l’espace (3D-Multiple Object Tracking (3D-MOT)) chez des adultes autistes (N = 15, âgés de 18 à 33 ans), comparés à des participants contrôles appariés sur l'âge et le QI, qui devaient suivre une ou trois cibles en mouvement parmi des distracteurs dans un environnement de réalité virtuelle. Les performances ont été mesurées par des seuils de vitesse, qui évaluent la plus grande vitesse à laquelle des observateurs sont capables de suivre des objets en mouvement. Les individus autistes ont montré des seuils de vitesse réduits dans l'ensemble, peu importe le nombre d'objets à suivre. Ces résultats étendent des résultats antérieurs d'altérations au niveau des mécanismes d'attention en autisme quant à l'allocation simultanée de l'attention envers des endroits multiples. Pris ensemble, les résultats de nos deux études révèlent donc des altérations chez les TSA quant au traitement simultané d'événements multiples, que ce soit dans une modalité ou à travers des modalités, ce qui peut avoir des implications importantes au niveau de la présentation clinique de cette condition. / Autism spectrum disorders (ASD) are currently characterized by a triad of impairments including social dysfunction, communication deficits and perseverative behaviours. The simultaneous integration of multiple senses is crucial in everyday life as it allows for the creation of a unified percept. Similarly, the allocation of attention to multiple events at the same time is critical in the processing of dynamic environmental information. In daily interactions with the environment, both sensory processing as well as attentional functions are building blocks to typical development (TD). Although not part of the current diagnostic criteria, difficulties with attention functions and sensory processing are very common among autistic persons. The present thesis therefore examined both these functions in two separate studies. The first study is based on the premise that alterations in basic sensory processing might underlie atypical sensory behaviours in ASD, as proposed by current theories of ASD. We conceived a cross-modal size discrimination task to assess the integrity and developmental course of visuo-tactile information in children with ASD (N = 21, aged 6-18 years), compared to age- and performance IQ-matched children with TD. In a simultaneous two-alternative forced-choice task, participants were asked to make a judgement on the size of two stimuli, based on unisensory (visual or tactile) or multisensory (visuo-tactile) inputs. Difference thresholds evaluated the smallest difference at which participants were capable to discriminate size. Children with ASD showed diminished performance and no maturational effects in both unisensory and multisensory conditions, compared to TD participants. Our first study therefore extends previous results of alterations in multisensory processing in ASD to the visuo-tactile domain. In our second study, we evaluated 3D-Multiple Object Tracking (3D-MOT) capacities in autistic adults (N = 15, aged 18-33 years), compared to age- and IQ-matched control participants, who were asked to track one or three moving targets amongst a set of distracters in a virtual reality environment. Performances were measured based on speed thresholds, which evaluates the greatest speed at which observers are capable of successfully tracking moving objects. Autistic individuals displayed overall reduced speed thresholds, whatever the number of spheres to track. These findings extend previous results of altered attention mechanisms in autism with regards to the simultaneous allocation of attention to multiple areas. Together, the findings of our two studies reveal alterations in ASD with regards to the processing of multiple events at the same time, be it within one modality or across modalities, which may have important implications for the clinical presentation of this condition.

Background. In everyday life it is often required to integrate multisensory input to successfully conduct response inhibition (RI) and thus major executive control processes. Both RI and multisensory processes have been suggested to be altered in autism spectrum disorder (ASD). It is, however, unclear which neurophysiological processes relate to changes in RI in ASD and in how far these processes are affected by possible multisensory integration deficits in ASD. Method. Combining high-density EEG recordings with source localization analyses, we examined a group of adolescent ASD patients (n = 20) and healthy controls (n = 20) using a novel RI task. Results. Compared to controls, RI processes are generally compromised in adolescent ASD. This aggravation of RI processes is modulated by the content of multisensory information. The neurophysiological data suggest that deficits in ASD emerge in attentional selection and resource allocation processes related to occipito-parietal and middle frontal regions. Most importantly, conflict monitoring subprocesses during RI were specifically modulated by content of multisensory information in the superior frontal gyrus. Conclusions. RI processes are overstrained in adolescent ASD, especially when conflicting multisensory information has to be integrated to perform RI. It seems that the content of multisensory input is important to consider in ASD and its effects on cognitive control processes.

We can affirm to apprehend a system in its totality only when we know how it behaves under its natural operating conditions. However, in the face of the complexity of the world, science can only evolve by simplifications, which paradoxically hide a good deal of the very mechanisms we are interested in. On the other hand, scientific enterprise is very tightly related to the advances in technology and the latter inevitably influences the manner in which the scientific experiments are conducted. Due to this factor, experimental conditions which would have been impossible to bring into laboratory not more than 20 years ago, are today within our reach. This thesis investigates neuronal integrative processes by using a variety of theoretical and experimental techniques wherein the approximation of ecologically relevant conditions within the laboratory is the common denominator. The working hypothesis of this thesis is that neurons and neuronal systems, in the sensory and higher cortices, are specifically adapted, as a result of evolutionary processes, to the sensory signals most likely to be received under ecologically relevant conditions. In order to conduct the present study along this line, we first recorded movies with the help of two microcameras carried by cats exploring a natural environment. This resulted in a database of binocular natural movies that was used in our theoretical and experimental studies. In a theoretical study, we aimed to understand the principles of binocular disparity encoding in terms of spatio-temporal statistical properties of natural movies in conjunction with simple mathematical expressions governing the activity levels of simulated neurons. In an unsupervised learning scheme, we used the binocular movies as input to a neuronal network and obtained receptive fields that represent these movies optimally with respect to the temporal stability criterion. Many distinctive aspects of the binocular coding in complex cells, such as the phase and position encoding of disparity and the existence of unbalanced ocular contributions, were seen to emerge as the result of this optimization process. Therefore we conclude that the encoding of binocular disparity by complex cells can be understood in terms of an optimization process that regulates activities of neurons receiving ecologically relevant information. Next we aimed to physiologically characterize the responses of the visual cortex to ecologically relevant stimuli in its full complexity and compare these to the responses evoked by artificial, conventional laboratory stimuli. To achieve this, a state-of-the-art recording method, voltage-sensitive dye imaging was used. This method captures the spatio-temporal activity patterns within the millisecond range across large cortical portions spanning over many pinwheels and orientation columns. It is therefore very well suited to provide a faithful picture of the cortical state in its full complexity. Drifting bar stimuli evoked two major sets of components, one coding for the position and the other for the orientation of the grating. Responses to natural stimuli involved more complex dynamics, which were locked to the motion present in the natural movies. In response to drifting gratings, the cortical state was initially dominated by a strong excitatory wave. This initial spatially widespread hyper-excitatory state had a detrimental effect on feature selectivity. In contrast, natural movies only rarely induced such high activity levels and the onset of inhibition cut short a further increase in activation level. An increase of 30% of the movie contrast was estimated to be necessary in order to produce activity levels comparable to gratings. These results show that the operating regime within which the natural movies are processed differs remarkably. Moreover, it remains to be established to what extent the cortical state under artificial conditions represents a valid state to make inferences concerning operationally more relevant input. The primary visual cortex contains a dense web of neuronal connections linking distant neurons. However the flow of information within this local network is to a large extent unknown under natural stimulation conditions. To functionally characterize these long-range intra-areal interactions, we presented natural movies also locally through either one or two apertures and analyzed the effects of the distant visual stimulation on the local activity levels. The distant patch had a net facilitatory effect on the local activity levels. Furthermore, the degree of the facilitation was dependent on the congruency between the two simultaneously presented movie patches. Taken together, our results indicate that the ecologically relevant stimuli are processed within a distinct operating regime characterized by moderate levels of excitation and/or high levels of inhibition, where facilitatory cooperative interactions form the basis of integrative processes. To gather better insights into the motion locking phenomenon and test the generalizability of the local cooperative processes toward larger scale interactions, we resorted to the unequalized temporal resolution of EEG and conducted a multimodal study. Inspired from the temporal properties of our natural movies, we designed a dynamic multimodal stimulus that was either congruent or incongruent across visual and auditory modalities. In the visual areas, the dynamic stimulation unfolded neuronal oscillations with frequencies well above the frequency spectrum content of the stimuli and the strength of these oscillations was coupled to the stimuli's motion profile. Furthermore, the coupling was found to be stronger in the case where the auditory and visual streams were congruent. These results show that the motion locking, which was so far observed in cats, is a phenomenon that also exists in humans. Moreover, the presence of long-range multimodal interactions indicates that, in addition to local intra-areal mechanisms ensuring the integration of local information, the central nervous system embodies an architecture that enables also the integration of information on much larger scales spread across different modalities. Any characterization of integrative phenomena at the neuronal level needs to be supplemented by its effects at the behavioral level. We therefore tested whether we could find any evidence of integration of different sources of information at the behavioral level using natural stimuli. To this end, we presented to human subjects images of natural scenes and evaluated the effect of simultaneously played localized natural sounds on their eye movements. The behavior during multimodal conditions was well approximated by a linear combination of the behavior under unimodal conditions. This is a strong indication that both streams of information are integrated in a joint multimodal saliency map before the final motor command is produced. The results presented here validate the possibility and the utility of using natural stimuli in experimental settings. It is clear that the ecological relevance of the experimental conditions are crucial in order to elucidate complex neuronal mechanisms resulting from evolutionary processes. In the future, having better insights on the nervous system can only be possible when the complexity of our experiments will match to the complexity of the mechanisms we are interested in.