Current multimodal interfaces for complex systems, such as those designed using the Ecological Interface Design (EID) methodology, have largely focused on effective design of interfaces that treat each sensory modality as either an independent channel of information or as a way to provide redundant information. However, there are many times when operationally related information is presented in different sensory modalities. There is very little research that has examined how this information in different modalities can be linked at a perceptual level. When related information is presented through multiple sensory modalities, interface designers will require perceptual methods for linking relevant information together across modalities. This thesis examines one possible crossmodal perceptual relationship, temporal synchrony, and evaluates whether the relationship is useful in the design of multimodal interfaces for complex systems. Two possible metrics for the evaluation of crossmodal perceptual relationships were proposed: resistance to changes in workload, and stream monitoring awareness. Two experiments were used to evaluate these metrics. The results of the first experiment showed that temporal rate synchrony was not resistant to changes in workload, manipulated through a secondary visual task. The results of the second experiment showed that participants who used crossmodal temporal rate synchrony to link information in a multimodal interface did not achieve better performance in the monitoring of the two streams of information being presented over equivalent unimodal interfaces. Taken together, these findings suggest that temporal rate synchrony may not be an effective method for linking information across modalities. Crossmodal perceptual relationships may be very different from intra-modal perceptual relationships. However, methods for linking information across sensory modalities are still an important goal for interface designers, and a key feature of future multimodal interface design for complex systems.

Made available in DSpace on 2014-12-17T15:36:57Z (GMT). No. of bitstreams: 1 FabioVC.pdf: 1566053 bytes, checksum: ff1176922095055a7b4aa58b50429d3f (MD5) Previous issue date: 2010-02-26 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior / The currently accepted model of sensory processing states that different senses are processed in parallel, and that the activity of specific cortical regions define the sensorial modality perceived by the subject. In this work we used chronic multielectrode extracellular recordings to investigate to which extent neurons in the visual and tactile primary cortices (V1 and S1) of anesthetized rats would respond to sensory modalities not traditionaly associated with these cortices. Visual stimulation yielded 87% of responsive neurons in V1, while 82% of S1 neurons responded to tactile stimulation. In the same stimulation sessions, we found 23% of V1 neurons responding to tactile stimuli and 22% of S1 neurons responding to visual stimuli. Our data supports an increasing body of evidence that indicates the existence multimodal processing in primary sensory cortices. Our data challenge the unimodal sensory processing paradigm, and suggest the need of a reinterpretation of the currently accepted model of cortical hierarchy. / O modelo de processamento sensorial mais aceito atualmente afirma que os sentidos s?o processados em paralelo, e que a atividade de c?rtices sensoriais espec?ficos define a modalidade sens?ria percebida subjetivamente. Neste trabalho utilizamos registros eletrofisiol?gicos cr?nicos de m?ltiplos neur?nios para investigar se neur?nios nos c?rtices prim?rios visual (V1) e t?til (S1) de ratos anestesiados podem responder a est?mulos das modalidades sensoriais n?o associadas tradicionalmente a estes c?rtices. Durante a estimula??o visual, 87% dos neur?nios de V1 foram responsivos, enquanto 82% dos neur?nios de S1 responderam ? estimula??o t?til. Nos mesmos registros, encontramos 23% dos neur?nios de V1 responsivos a est?mulos t?teis e 22% dos neur?nios de S1 responsivos a est?mulos visuais. Nossos dados corroboram uma crescente s?rie de evid?ncias que indica a presen?a de processamento multimodal nos c?rtices sensoriais prim?rios, o que desafia o paradigma do processamento sensorial unimodal e sugere a necessidade de uma reinterpreta??o do modelo de hierarquia cortical atualmente aceito.

From the start of their lives, infants and young children are surrounded by a tremendous amount of multimodal social information. One intriguing question in the study of early social cognition is how vital social information is detected and processed and how and when young infants begin to make sense of what they see and hear and learn to understand other people’s behavior. The overall aim of this thesis was to provide new insights to this exciting field. Investigating behavior and/or neural mechanisms in early life, the three different studies included in this thesis therefore strive to increase our understanding on perception and processing of social information. Study I used eye-tracking to examine infants´ observations of gaze in a third-party context. The results showed that 9-, 16- and 24-month-old infants differentiate between the body orientations of two individuals on the basis of static visual information. More particularly, they shift their gaze more often between them when the social partners face each other than when they are turned away from each other. Using ERP technique, Study II demonstrated that infants at the age of 4 to 5 months show signs of integrating visual and auditory information at a neural level. Further, direct gaze in combination with backwards-spoken words leads to earlier or enhanced neural processing in comparison to other gaze-word combinations. Study III, also an EEG investigation, found that children between 18 and 30 months of age show a desynchronization of the mu rhythm during both the observation and execution of object-directed actions. Also, the results suggest motor system activation when young children observe others’ mimed actions. To summarize, the findings reported in this thesis strengthen the idea that infants are sensitive to others´ gaze and that this may extend to third-party contexts. Also, gaze is processed together with other information, for instance words, even before infants are able to understand others’ vocabulary. Furthermore, the motor system in young children is active during both the observation and imitation of another person’s goal-directed actions. This is in line with findings in infants, children and adults, indicating that these processes are linked at neural level.

The experiments reported within this thesis use psychophysical techniques to examine the factors which determine perceived multisensory timing in humans. Chapters 1 and 2 describe anatomical and psychophysical features of temporal processing, respectively, whilst Chapter 3 introduces the reader to psychophysical methods. Chapter 4 examines the relationship between two measures of sensory latency, reaction time (RT) and crossmodal temporal order judgment (TOJ). Despite task and attentional manipulations the two measures do not correlate, suggesting that they measure some fundamentally different aspect(s) of temporal perception. Chapter 5 examines the effects of adaptation to asynchronous stimulus pairs on perceived audiovisual (AV), audiotactile (AT) and visuotactile (VT) temporal order. Significant temporal shifts are recorded in all three conditions. Evidence is also presented showing that crossmodal TOJs are intransitive. Chapter 6 shows that concurrent adaptation to two sets of asynchronous AV stimulus pairs causes perceived AV temporal order to recalibrate at two locations simultaneously, and that AV asynchrony adaptation effects are significantly affected by observers' attention during adaptation. Finally, Chapter 7 shows that when observers are accustomed to a physical delay between motor actions and sensory events, an event presented at a reduced delay appears to precede the causative motor action. The data are well-described by a simple model based on a strong prior assumption of physical synchrony between motor actions and their sensory consequences.

Thèse numérisée par la Division de la gestion de documents et des archives de l'Université de Montréal.

Thèse réalisée en collaboration avec le Département de neurosciences et pharmacologie de l'Université de Copenhague, Danemark. / La vision est cruciale dans la recherche et l’identification de nourriture. Non seulement elle déclenche le réflexe céphalique de la digestion mais, combinée à l’expérience alimentaire, elle aide à raffiner nos prévisions par rapport aux aliments. En un simple clin d’œil, la vue renseigne sur la disponibilité, l’identité, la comestibilité, les saveurs, les textures et les contenus nutritionnel, calorique et toxique des aliments qui nous entourent. Étant donnée l’importance de la vue dans l’expérience gustative, il est judicieux de se poser la question suivante : Qu’arrive-t-il au goût en absence de vision? Cette thèse répond à cette question par l’étude de cette modalité chez l’aveugle de naissance grâce aux techniques de psychophysique et d’imagerie cérébrale. De plus, les conséquences gustatives de la cécité sont comparées à celles suivant la perte d’un autre sens important dans l’appréciation des aliments, soit l’odorat (anosmie). Les résultats comportementaux démontrent premièrement que l’absence de vision depuis la naissance abaisse la sensibilité gustative, reflétée par des seuils élevés de détection et d’identification des cinq goûts de base (sucré, salé, acide, amer, umami). Deuxièmement, bien que les aveugles congénitaux aient plus de facilité à identifier les odeurs comestibles par leurs narines (voie olfactive orthonasale), ceux-ci perdent leur avantage par rapport aux voyants quand ils doivent identifier ces stimuli placés sur la langue (voie olfactive rétronasale). Les résultats d’imagerie indiquent en outre que les aveugles congénitaux activent moins leur cortex gustatif primaire (insula/opercule) et leur hypothalamus par rapport aux voyants durant une tâche gustative. De plus, l’absence d’activation dans le cortex (« visuel ») occipital chez l’aveugle pointe vers le manque de plasticité intermodale en gustation. Chez les anosmiques congénitaux d’autre part, non seulement l’absence d’odorat diminue l’habileté à reconnaître les goûts mais elle abaisse également la force du signal dans les aires olfactives (ex : cortex orbitofrontal médial) durant une tâche gustative. Les résultats chez l’aveugle contrastent grandement avec les études antérieures soulignant l’amélioration de leurs sens extéroceptifs tels que l’audition, l’olfaction (orthonasale) et le toucher qui font tous intervenir la plasticité intermodale. Par ailleurs, les données chez l’anosmique concordent avec ceux de la littérature indiquant une diminution similaire de la chémosensation trigéminale, laquelle est également associée à un affaiblissement du circuit neural des saveurs. Ceci suggère que le sens du goût ne soit pas utile aux handicapés visuels pour percevoir l’environnement extérieur et ainsi compenser leur perte de vision. De plus, bien que l’odorat participe à l’appréciation de la nourriture en bouche, sa perte n’entraîne pas de compensation sensorielle chez l’anosmique. Prises ensemble, ces données indiquent différents mécanismes d’adaptation suivant la cécité et l’anosmie. Elles soutiennent également le point de vue selon lequel la perception unifiée de goûts et de saveurs inclut non seulement les sens chimiques et le toucher mais également la vision. Considérant l’importance du goût et de l’alimentation dans la qualité de vie, ces résultats encouragent la société tout comme les professionnels de la réadaptation à faciliter l’accès à la nourriture ainsi qu’à l’enseignement culinaire chez les handicapés sensoriels. / Vision is crucial for seeking and identifying food. Not only does it trigger the cephalic digestion reflex but, when combined with the experience of eating, it helps to refine expectations about foods. In a single eye blink, sight informs us about the availability, identity, palatability, flavours, textures as well as nutritional, caloric and toxic contents of foods surrounding us. Given the importance of sight in the gustatory experience, one may therefore ask the following question: What happens to gustation without vision? This thesis answers this question by studying this modality in congenitally blind subjects using psychophysical and brain imaging techniques. Additionally, the gustatory consequences of blindness are compared to those following the loss of another important modality involved in the appreciation of food, i.e. the sense of smell (anosmia). Behavioural results first show that the absence of vision from birth reduces the gustatory sensitivity, as reflected by higher detection and identification thresholds of the five basic tastes (sweet, salty, acid, bitter, umami). Second, although congenitally blind subjects are better at identifying palatable odorant stimuli through their nostrils (orthonasal olfactory route), they lose this advantage over sighted people when identifying these stimuli placed on their tongue (retronasal olfactory route). Neuroimaging results also reveal that congenitally blind subjects activate the primary gustatory cortex (insula/operculum) and the hypothalamus less compared to blindfolded sighted participants. Moreover, the absence of occipital (“visual”) cortex activity in the blind points towards the lack of crossmodal plasticity in gustation. In congenitally anosmics, on the other hand, not only does the absence of smell lower the ability to recognize tastes but it also lowers the strength of the signal in olfactory areas (e.g. medial orbitofrontal cortex) during a gustatory task. The results in the blind greatly contrast with previous studies highlighting the enhancement of their exteroceptive senses such as audition, (orthonasal) olfaction and touch, all of which involve crossmodal plasticity. Moreover, data in the anosmic group are consistent with previous literature describing similar decrease of trigeminal chemosensation that is also associated with a weakening of the flavour neural network. This suggests that the sense of taste is not useful to the visually impaired to perceive their exterior environment and compensate for their lack of vision. Furthermore, although olfaction contributes to the appreciation of foods in the mouth, the lack of this modality does not drive sensory compensation in anosmic subjects. Taken together, these data indicate different adaptation mechanisms following blindness and anosmia. They also support the view according to which the unified perception of tastes and flavours includes not only the chemical senses (taste, smell and trigeminal chemosensation) and touch but also vision. Given the importance of taste and eating experience in quality of life, these results encourage society as well as rehabilitation professionals to facilitate access to foods and culinary lessons in sensory deprived subjects.

Thèse numérisée par la Division de la gestion de documents et des archives de l'Université de Montréal

The main focus for this thesis was cross-modal attention capture by sudden and unexpected sounds and vibrations, known as deviants, presented in a stream the same to-be-ignored stimulus. More specifically, the thesis takes a multisensory perspective and examines the possible similarities and differences in how deviant vibrations and sounds affect visual task performance (Study I), and whether the deviant and standard stimuli have to be presented within the same modality to capture attention away from visual tasks (Study II). Furthermore, by presenting spatial deviants (changing the source of the stimuli from one side of the body to the other) in audiotactile (bimodal), tactile, and auditory to-be-ignored, it explores whether bimodal stimuli are more salient compared to unimodal (Study III). In addition, Study III tested the claims that short-term memory is domain-specific. In line with previous research, Study I found that both auditory and tactile deviants captured attention away from the visual task. However, the temporal dynamics between the two modalities seem to differ. That is, it seems like practice causes the effect of vibratory deviants to reduce, whereas this is not the case for auditory deviants. This suggests that there are central mechanisms (detection of the change) and sensory-specific mechanisms. Study II found that the deviant and standard stimuli must be presented within the same modality. If attention capture by deviants is produced by a mismatch within a neural model predicting upcoming stimuli, the neural model is likely built on stimuli within each modality separately. The results of Study III revealed that spatial and verbal short-term memory are negatively affected by a spatial change in to-be-ignored sequences, but only when the change is within a bimodal sequence. These results can be taken as evidence for a unitary account of short-term memory (verbal and spatial information stored in the same storage) and that bimodal stimuli may be integrated into a unitary percept that make any change in the stream more salient.

The experiments reported within this thesis use psychophysical techniques to examine the factors which determine perceived multisensory timing in humans. Chapters 1 and 2 describe anatomical and psychophysical features of temporal processing, respectively, whilst Chapter 3 introduces the reader to psychophysical methods. Chapter 4 examines the relationship between two measures of sensory latency, reaction time (RT) and crossmodal temporal order judgment (TOJ). Despite task and attentional manipulations the two measures do not correlate, suggesting that they measure some fundamentally different aspect(s) of temporal perception. Chapter 5 examines the effects of adaptation to asynchronous stimulus pairs on perceived audiovisual (AV), audiotactile (AT) and visuotactile (VT) temporal order. Significant temporal shifts are recorded in all three conditions. Evidence is also presented showing that crossmodal TOJs are intransitive. Chapter 6 shows that concurrent adaptation to two sets of asynchronous AV stimulus pairs causes perceived AV temporal order to recalibrate at two locations simultaneously, and that AV asynchrony adaptation effects are significantly affected by observers¿ attention during adaptation. Finally, Chapter 7 shows that when observers are accustomed to a physical delay between motor actions and sensory events, an event presented at a reduced delay appears to precede the causative motor action. The data are well-described by a simple model based on a strong prior assumption of physical synchrony between motor actions and their sensory consequences.

This dissertation is about sound in context. Since sensory processing is inherently multimodal, research in sound is necessarily multidisciplinary. The present work has been guided by principles of systematicity, ecological validity, complementarity of  methods, and integration of science and art. The main tools to investigate the mediating relationship of people and environment through sound have been empiricism and psychophysics. Four of the seven included papers focus on perception. In paper A, urban soundscapes were reproduced in a 3D installation. Analysis of results from an experiment revealed correlations between acoustic features and physiological indicators of stress and relaxation. Paper B evaluated soundscapes of different type. Perceived quality was predicted not only by psychoacoustic descriptors but also personality traits. Sound reproduction quality was manipulated in paper D, causing two effects on source localisation which were explained by spatial and semantic crossmodal correspondences. Crossmodal correspondence was central in paper C, a study of colour association with music. A response interface employing CIE Lab colour space, a novelty in music emotion research, was developed. A mixed method approach supported an emotion mediation hypothesis, evidenced in regression models and participant interviews. Three papers focus on design. Field surveys and acoustic measurements were carried out in restaurants. Paper E charted relations between acoustic, physical, and perceptual features, focussing on designable elements and materials. This investigation was pursued in Paper F where a taxonomy of sound sources was developed. Analysis of questionnaire data revealed perceptual and crossmodal effects. Lastly, paper G discussed how crossmodal correspondences facilitated creation of meaning in music by infusing ecologically founded sonification parameters with visual and spatial metaphors. The seven papers constitute an investigation into how sound affects us, and what sound means to us. / Denna doktorsavhandling handlar om ljud i sammanhang. Eftersom informationsbehandling genom sinnena alltid är multimodal så kräver ljudforskning en tvärvetenskaplig forskningsansats. Arbetet i denna avhandling har vägletts av principer såsom systematik, ekologisk validitet, samspel mellan metoder, och integration av vetenskap och konst. De viktigaste redskapen för att undersöka den ömsesidiga påverkan mellan människa och miljö genom ljud har varit empiri och psykofysik.Fyra artiklar handlar om perception. I artikel A återskapades urbana ljudlandskap i en 3D-ljudinstallation. Analys av experimentresultat avslöjade samband mellan akustiska mått och fysiologiska markörer av stress och avslappning.  Artikel B utvärderade olika typer av ljudlandskap. Upplevd kvalitet kunde prediceras inte bara av psykoakustiska mått utan även av personlighetsdrag. Ljudåtergivningskvalitet manipulerades i artikel D och orsakade två effekter på lokalisering av en ljudkälla vilka förklarades av rumslig och semantisk korsmodala kopplingar. Korsmodalitet var huvudpunkten i artikel C, en studie av färgassociation till musik. Ett användargränssnitt utvecklades som använder färgrymden CIE Lab, en nyhet i forskningfältet musik och känslor. En abduktiv metod stödde hypotesen att känslouttryck medierar korsmodala kopplingar, vilket framgick av regressionsmodeller och intervjuer med försökspersonerna.Tre artiklar handlar om design. Fältundersökningar och ljudmätningar utfördes i restauranger. Artikel E kartlade samband mellan akustiska, fysiska och perceptuella särdrag, med fokus på formbara element och material. Detta arbete fortsattes i artikel F varigenom en taxonomi av ljudkällor utvecklades. Analys av enkätdata avslöjade perceptuella och korsmodala effekter. Slutligen, artikel G diskuterade hur korsmodala kopplingar främjade meningsskapande i musik genom att ekologiskt motiverade sonifikationsparametrar samverkade med visuella och spatiala uttryck. De sju artiklarna utgör landmärken i avhandlingens utforskande av hur ljud påverkar oss, och vad ljud betyder för oss. / <p>QC 20151118</p>