We can affirm to apprehend a system in its totality only when we know how it behaves under its natural operating conditions. However, in the face of the complexity of the world, science can only evolve by simplifications, which paradoxically hide a good deal of the very mechanisms we are interested in. On the other hand, scientific enterprise is very tightly related to the advances in technology and the latter inevitably influences the manner in which the scientific experiments are conducted. Due to this factor, experimental conditions which would have been impossible to bring into laboratory not more than 20 years ago, are today within our reach. This thesis investigates neuronal integrative processes by using a variety of theoretical and experimental techniques wherein the approximation of ecologically relevant conditions within the laboratory is the common denominator. The working hypothesis of this thesis is that neurons and neuronal systems, in the sensory and higher cortices, are specifically adapted, as a result of evolutionary processes, to the sensory signals most likely to be received under ecologically relevant conditions. In order to conduct the present study along this line, we first recorded movies with the help of two microcameras carried by cats exploring a natural environment. This resulted in a database of binocular natural movies that was used in our theoretical and experimental studies. In a theoretical study, we aimed to understand the principles of binocular disparity encoding in terms of spatio-temporal statistical properties of natural movies in conjunction with simple mathematical expressions governing the activity levels of simulated neurons. In an unsupervised learning scheme, we used the binocular movies as input to a neuronal network and obtained receptive fields that represent these movies optimally with respect to the temporal stability criterion. Many distinctive aspects of the binocular coding in complex cells, such as the phase and position encoding of disparity and the existence of unbalanced ocular contributions, were seen to emerge as the result of this optimization process. Therefore we conclude that the encoding of binocular disparity by complex cells can be understood in terms of an optimization process that regulates activities of neurons receiving ecologically relevant information. Next we aimed to physiologically characterize the responses of the visual cortex to ecologically relevant stimuli in its full complexity and compare these to the responses evoked by artificial, conventional laboratory stimuli. To achieve this, a state-of-the-art recording method, voltage-sensitive dye imaging was used. This method captures the spatio-temporal activity patterns within the millisecond range across large cortical portions spanning over many pinwheels and orientation columns. It is therefore very well suited to provide a faithful picture of the cortical state in its full complexity. Drifting bar stimuli evoked two major sets of components, one coding for the position and the other for the orientation of the grating. Responses to natural stimuli involved more complex dynamics, which were locked to the motion present in the natural movies. In response to drifting gratings, the cortical state was initially dominated by a strong excitatory wave. This initial spatially widespread hyper-excitatory state had a detrimental effect on feature selectivity. In contrast, natural movies only rarely induced such high activity levels and the onset of inhibition cut short a further increase in activation level. An increase of 30% of the movie contrast was estimated to be necessary in order to produce activity levels comparable to gratings. These results show that the operating regime within which the natural movies are processed differs remarkably. Moreover, it remains to be established to what extent the cortical state under artificial conditions represents a valid state to make inferences concerning operationally more relevant input. The primary visual cortex contains a dense web of neuronal connections linking distant neurons. However the flow of information within this local network is to a large extent unknown under natural stimulation conditions. To functionally characterize these long-range intra-areal interactions, we presented natural movies also locally through either one or two apertures and analyzed the effects of the distant visual stimulation on the local activity levels. The distant patch had a net facilitatory effect on the local activity levels. Furthermore, the degree of the facilitation was dependent on the congruency between the two simultaneously presented movie patches. Taken together, our results indicate that the ecologically relevant stimuli are processed within a distinct operating regime characterized by moderate levels of excitation and/or high levels of inhibition, where facilitatory cooperative interactions form the basis of integrative processes. To gather better insights into the motion locking phenomenon and test the generalizability of the local cooperative processes toward larger scale interactions, we resorted to the unequalized temporal resolution of EEG and conducted a multimodal study. Inspired from the temporal properties of our natural movies, we designed a dynamic multimodal stimulus that was either congruent or incongruent across visual and auditory modalities. In the visual areas, the dynamic stimulation unfolded neuronal oscillations with frequencies well above the frequency spectrum content of the stimuli and the strength of these oscillations was coupled to the stimuli's motion profile. Furthermore, the coupling was found to be stronger in the case where the auditory and visual streams were congruent. These results show that the motion locking, which was so far observed in cats, is a phenomenon that also exists in humans. Moreover, the presence of long-range multimodal interactions indicates that, in addition to local intra-areal mechanisms ensuring the integration of local information, the central nervous system embodies an architecture that enables also the integration of information on much larger scales spread across different modalities. Any characterization of integrative phenomena at the neuronal level needs to be supplemented by its effects at the behavioral level. We therefore tested whether we could find any evidence of integration of different sources of information at the behavioral level using natural stimuli. To this end, we presented to human subjects images of natural scenes and evaluated the effect of simultaneously played localized natural sounds on their eye movements. The behavior during multimodal conditions was well approximated by a linear combination of the behavior under unimodal conditions. This is a strong indication that both streams of information are integrated in a joint multimodal saliency map before the final motor command is produced. The results presented here validate the possibility and the utility of using natural stimuli in experimental settings. It is clear that the ecological relevance of the experimental conditions are crucial in order to elucidate complex neuronal mechanisms resulting from evolutionary processes. In the future, having better insights on the nervous system can only be possible when the complexity of our experiments will match to the complexity of the mechanisms we are interested in.

Le monde du divertissement numérique connaît depuis plusieurs années une évolution majeure avec la démocratisation des technologies vidéo 3D. Il est désormais commun de visualiser des vidéos stéréoscopiques sur différents supports : au cinéma, à la télévision, dans les jeux vidéos, etc. L'image 3D a considérablement évolué mais qu'en est-il des technologies de restitution sonore associées ? La plupart du temps, le son qui accompagne la vidéo 3D est basé sur des effets de latéralisation, plus au moins étendus (stéréophonie, systèmes 5.1). Il est pourtant naturel de s'interroger sur le besoin d'introduire des événements sonores en lien avec l'ajout de cette nouvelle dimension visuelle : la profondeur. Plusieurs technologies semblent pouvoir offrir une description sonore 3D de l'espace (technologies binaurales, Ambisonics, Wave Field Synthesis). Le recours à ces technologies pourrait potentiellement améliorer la qualité d'expérience de l'utilisateur, en termes de réalisme tout d'abord grâce à l'amélioration de la cohérence spatiale audio-visuelle, mais aussi en termes de sensation d'immersion. Afin de vérifier cette hypothèse, nous avons mis en place un système de restitution audio-visuelle 3D proposant une présentation visuelle stéréoscopique associée à un rendu sonore spatialisé par Wave Field Synthesis. Trois axes de recherche ont alors été étudiés : 1 / Perception de la distance en présentation unimodale ou bimodale. Dans quelle mesure le système audio-visuel est-il capable de restituer des informations spatiales relatives à la distance, dans le cas d'objets sonores, visuels, ou audio-visuels ? Les expériences menées montrent que la Wave Field Synthesis permet de restituer la distance de sources sonores virtuelles. D'autre part, les objets visuels et audio-visuels sont localisés avec plus de précisions que les objets uniquement sonores. 2 / Intégration multimodale suivant la distance. Comment garantir une perception spatiale audio-visuelle cohérente de stimuli simples ? Nous avons mesuré l'évolution de la fenêtre d'intégration spatiale audio-visuelle suivant la distance, c'est-à-dire les positions des stimuli audio et visuels pour lesquelles la fusion des percepts a lieu. 3 / Qualité d'expérience audio-visuelle 3D. Quel est l'apport du rendu de la profondeur sonore sur la qualité d'expérience audio-visuelle 3D ? Nous avons tout d'abord évalué la qualité d'expérience actuelle, lorsque la présentation de contenus vidéo 3D est associée à une bande son 5.1, diffusée par des systèmes grand public (système 5.1, casque, et barre de son). Nous avons ensuite étudié l'apport du rendu de la profondeur sonore grâce au système audio-visuel proposé (vidéo 3D associée à la Wave Field Synthesis). / The digital entertainment industry is undergoing a major evolution due to the recent spread of stereoscopic-3D videos. It is now possible to experience 3D by watching movies, playing video games, and so on. In this context, video catches most of the attention but what about the accompanying audio rendering? Today, the most often used sound reproduction technologies are based on lateralization effects (stereophony, 5.1 surround systems). Nevertheless, it is quite natural to wonder about the need of introducing a new audio technology adapted to this new visual dimension: the depth. Many alternative technologies seem to be able to render 3D sound environments (binaural technologies, ambisonics, Wave Field Synthesis). Using these technologies could potentially improve users' quality of experience. It could impact the feeling of realism by adding audio-visual spatial congruence, but also the immersion sensation. In order to validate this hypothesis, a 3D audio-visual rendering system is set-up. The visual rendering provides stereoscopic-3D images and is coupled with a Wave Field Synthesis sound rendering. Three research axes are then studied: 1/ Depth perception using unimodal or bimodal presentations. How the audio-visual system is able to render the depth of visual, sound, and audio-visual objects? The conducted experiments show that Wave Field Synthesis can render virtual sound sources perceived at different distances. Moreover, visual and audio-visual objects can be localized with a higher accuracy in comparison to sound objects. 2/ Crossmodal integration in the depth dimension. How to guarantee the perception of congruence when audio-visual stimuli are spatially misaligned? The extent of the integration window was studied at different visual object distances. In other words, according to the visual stimulus position, we studied where sound objects should be placed to provide the perception of a single unified audio-visual stimulus. 3/ 3D audio-visual quality of experience. What is the contribution of sound depth rendering on the 3D audio-visual quality of experience? We first assessed today's quality of experience using sound systems dedicated to the playback of 5.1 soundtracks (5.1 surround system, headphones, soundbar) in combination with 3D videos. Then, we studied the impact of sound depth rendering using the set-up audio-visual system (3D videos and Wave Field Synthesis).

Traffic is a complex environment in which many actors take part; several new technologies bring promises of reducing this complexity. However, cyclists—a particularly vulnerable road user group—have so far been somewhat put aside in these new developments, among them being Cooperative Intelligent Traffic Systems (C-ITS) and their aspects of human–computer interaction. This master’s thesis of industrial design engineering presents five multimodal collision warning signals for cyclists—future ones in these supposed C-ITS—using a novel application of bone conduction headphones (BCH) via sensations of both sound and touch. The thesis project was conducted as a complementary subset of the larger research project ‘V2Cyclist’ orchestrated by RISE Interactive. V2Cyclist set out to adapt the wireless V2X-protocol for cyclists by developing a physical prototype in the form of a bicycle helmet and corresponding human–computer interface. A significant part of the theoretical framework for this thesis was multiple resource theory: tasks in a different modality can be performed more effectively than in one already taxed attentively. Literature on human factors was also applied, particularly with regards to the perception of sound; evidence suggests that humans evolved a perceptual bias for threatening and ‘looming’ sounds that appear to encroach our peripersonal space; ethological findings point toward the association with low-frequency sounds to largeness. Sound design techniques usually applied to more artistic ends, such as synthesis and mixing, were repurposed for the novel, audiotactile context of this thesis. The thesis process was rooted in design thinking and consisted of four stages: context immersion, ideation, concept development, and lastly evaluation; converging and diverging the novel design space of using BCH in an audiotactile, i.e. bimodal way. The divergent approach generated a wide range of ideas. The later convergent approach did not result in one, definite design as further evaluation is required but also due to unknowns in terms of future hardware and network constraints. Given the plurality and diversity of cyclists, it may well follow that there is no optimal collision warning design in the singular. Hence, a range of five different solutions is presented. Each of the five multimodal collision warnings presents a different approach to conveying a sense of danger and urgency. Some warning signals are static in type, while others are more dynamic. Given the presumed rarity of collision warnings, multiple design techniques and rationales were applied separately, as well as in combination, to create different warning stimuli that signaled high urgency in an intuitive way. Namely, the use of: conventions in design and culture; explicitness in the form of speech; visceral appeal via threatening and animalistic timbres; dynamic and procedurally generated feedback; multimodal salience; crossmodal evocation of ‘roughness;’ size-sound symbolism to imply largeness; and innately activating characteristics of looming sounds. / Trafiken är en komplex miljö med många deltagare; diverse ny teknik gör anspråk på att underlätta denna komplexitet. Men, cyklister—en särskilt utsatt grupp av trafikanter—har hittills hamnat i skymundan för sådana utvecklingar. Vidare, aspekten av användbara gränssnitt för cyklister inom sådana uppkopplade och samverkande trafiksystem (C-ITS) har utforskats desto mindre. Det här examensarbetet inom Teknisk design presenterar fem multimodala kollisionsvarningar avsedda för cyklister—framtida sådana i dessa C-ITS—genom en ny och bimodal användning av benledande hörlurar via både ljud och vibrationer. Examensarbetet genomfördes i koppling till forskningsprojektet V2Cyclist, orkestrerat av RISE Interactive, vars projektmål var att anpassa det trådlösa kommunikationsprotokollet V2X för cyklister via en fysisk prototyp i form av en cykelhjälm och parallellt utveckla ett tillhörande användargränssnitt. En viktig del av det teoretiska ramverket för det här examensarbetet grundar sig på multiple resource theory: uppgifter kan utföras mer effektivt i en annan modalitet än i en som redan är belastad med uppmärksamhet. Mänskliga faktorer och teori om vår uppfattning användes; bevis pekar på att människor har evolutionärt utvecklat en bias för hotande ljud som upplevs inkräkta på vårt närmsta personliga revir; etologiska rön visar på en koppling mellan lågfrekventa ljud och ‘storhet.’ Tekniker inom ljuddesign vanligtvis använda till mer artistiska ändamål, såsom syntes och mixning, användes här till godo för att utforska den nya och bimodala designrymden. Processen för arbetet grundade sig i design thinking och bestod av fyra faser: kontextfördjupning, idégenerering, konceptutveckling, och utvärdering. En ny och tidigare outforskad designrymd beståendes av en bimodal, ljudtaktil användning av benledande hörlurar divergerades och konvergerades. Ett initialt utforskande angreppssätt gav upphov till en bred mängd av idéer. Ett senare renodlande angreppssätt gick, dock, inte hela vägen till endast en optimal lösning, då vidare utvärdering krävs men också på grund av okända teknologiska begränsningar. Dessutom, givet cyklisters stora mångfald, kan det möjligtvis följa att det inte finns någon enskild design av den optimala kollisionsvarningen. Ett spann på fem olika lösningar presenteras därmed. Fem koncept för multimodala kollisionsvarningar presenteras där varje variant uttrycker fara och kritiskhet på olika sätt. Vissa är statiska i typ, medan andra verkar mer kontinuerligt och dynamiskt. Det antogs att kollisionsvarningar sker sällan. Olika designtekniker och motiveringar har använts, ibland i kombination med varandra, för att skapa kollisionsvarningar vars avsikter omedelbart förstås: normer inom design och kultur gällande ljud; uttalad kommunikation i form av tal; anspråk på människors biologiska intuition via hotfulla och djurliknande klangfärger; dynamisk och procedurellt genererad feedback; multimodal effektfullhet; korsmodal känsla av grova texturer; size-sound symbolism för att antyda ‘storhet;’ samt de naturligt aktiverande egenskaperna hos looming sounds.