- / -

During multisensory integration, information from distinct sensory systems that refers to the same physical event is combined. For example, the sound and image that an individual generates as s/he interacts with the world, will provide the nervous system with multiple cues which can be integrated to estimate the individual’s position in the environment. However, the information that is perceived through different sensory pathways/systems can be qualitatively different. The information can be redundant and describe the same property of an event in a common reference frame (i.e., the image and sound referring to the individual’s location), or it can be complementary. Combining complementary information can be advantageous in that it extends the range and richness of the information available to the nervous system, but can also be superfluous and unnecessary to the task at hand – i.e. olfactory cues about the individuals perfume can increase the richness of the representation but not necessarily aid in localisation. Over the last century or so, a large body of research has focused on different aspects of multisensory interactions at both the behavioural and neural levels. It is currently unclear whether the mechanisms underlying multisensory interactions for both type of cue are similar or not. Moreover, the evidence for differences in behavioural outcome, dependent on the nature of the cue, is growing. Such cue property effects possibly reflect a processing heuristic for more efficient parsing of the vast amount of sensory information available to the nervous system at any one time. The present thesis assesses the effects of cue properties (i.e., redundant or complementary) on multisensory processing and reports a series of experiments demonstrating that the nature of the cue, defined by the task of the observer, influences whether the cues compete for representation as a result of interacting, or whether instead multisensory information produces an optimal increase in reliability of the event estimate. Moreover, a bridging series of experiments demonstrate the key role of redundancy in inferring that two signals have a common physical cause and should be integrated, despite conflict in the cues. The experiments provide insights into the different strategies adopted by the nervous system and some tentative evidence for possible, distinct underlying mechanisms.

Les simulateurs de conduite permettent d’explorer certains domaines de recherche difficiles à appréhender en conditions réelles, comme l'intégration de différents signaux sensoriels (ex. visuel, vestibulaire, somesthésique) pour la perception du mouvement. Malgré leur complexité, les simulateurs de conduite ne produisent pas toujours une sensation de conduite réelle, spécialement dans les situations comportant des freinages ou des virages. Leurs limites mécaniques en sont la cause. En conséquence, les lois de mouvement des simulateurs sont basées sur la technique de la « tilt-coordination ». Cette technique consiste à incliner un véhicule de telle sorte que la force gravitationnelle soit équivalente à l’accélération gravito-inertielle (GIA) résultant d’une accélération linéaire. La « tilt-coordination » se base sur l'ambigüité perçue par le système vestibulaire entre un basculement et une translation. Sur simulateur de conduite, l'algorithme « washout » combine la « tilt-coordination » à des translations pour produire une sensation d'accélération linéaire. L'objectif de ces travaux de recherche est d'atteindre une meilleure compréhension de l'intégration multisensorielle pour la perception des accélérations linéaires en simulateur de conduite. Les expériences présentées ci-dessous montrent que la perception des décélérations linéaires dépend de la manière dont le basculement et la translation sont combinés pour produire une perception cohérente. Par ailleurs, nos résultats montrent qu'il y a une différence importante dans la perception des accélérations et des décélérations. Pour le freinage, le rapport basculement/translation le plus réaliste dépend du niveau de décélération. Pour l'accélération, le mouvement est généralement surestimé et dépend du niveau d'accélération. Dans ce cas, la perception ne dépend pas du rapport basculement/translation. Ces résultats suggèrent que les signaux visuels, vestibulaires et somesthésiques sont intégrés de façon Bayésienne. En conclusion, il n'est pas conseillé d'utiliser l'algorithme « washout » sans prendre en compte la non-linéarité de la perception humaine. Nous proposons un modèle qui décrit la relation entre le basculement, la translation et le niveau d'accélération ou décélération souhaité. Ce modèle peut être utilisé pour améliorer la loi du mouvement afin de produire des simulations de conduite plus réalistes. / Driving simulators allow the exploration of certain areas of research that are difficult to reach in normal conditions, like the integration of different sensory inputs (visual, vestibular and somesthesic) for perception of self-motion. In spite of their complexity, driving simulators do not produce a realistic sensation of driving, especially for braking and turnings. This is due to their mechanical limitations. As a consequence, driving simulators' motion algorithm is based on tilt-coordination technique, which assumes the tilt of the car so that the driver's force of gravity is oriented in the same way as the gravito-inertial acceleration (GIA) during a linear acceleration. This technique is based on the tilt-translation ambiguity of the vestibular system and is used on dynamic driving simulators in combination with linear translations in so-called washout algorithm, to produce a sensation of linear acceleration. The aim of the present research is to understand how humans use multiple sensory signals (vestibular, visual and somatosensory) during the perception of linear acceleration on a driving simulator. The conducted experiments show that the perception of motion depends on the manner tilt and translation are used together to provide a unified percept of linear acceleration. Further, our results show that there is an important difference on how humans perceive accelerations and decelerations. For braking, the most realistic tilt/translation ratio depends on the level of deceleration. For acceleration, the motion is generally overestimated and depends on the level of acceleration, but not on the variation of tilt/translation ratio. The results suggest that visual, vestibular and proprioceptive cues are integrated in an optimal Bayesian fashion. In conclusion, it is not advisable to use a washout algorithm without taking into account the non-linearity of human perception. We propose an empirically found data-driven fitting model that describes the relationship between tilt, translation and the desired level of acceleration or deceleration. This model is intended to be a supplement to motion cueing algorithms that should improve the realism of driving simulations.

Au quotidien, notre système nerveux central utilise et intègre de multiples informations sensorielles pour percevoir de façon cohérente et efficace les mouvements de notre corps et de nos membres. Une littérature abondante a décrit chacune des contributions des entrées sensorielles visuelle, proprioceptive musculaire ou encore tactile dans l’émergence de cette perception, appelée kinesthésie. Mais des questions demeurent quant aux principes régissant l’intégration de ces différents sens à des fins kinesthésiques. Cette thèse apporte des éléments de réponse en soulignant la contribution majeure de la proprioception musculaire dans la construction des percepts multisensoriels kinesthésiques chez l’adulte jeune. Compte tenu du déclin de toutes les modalités sensorielles au cours du vieillissement, nous avons étudié l’évolution possible de ces règles d’intégration dans la dernière partie de ce travail.Revisitant un phénomène illusoire classiquement considéré comme d’origine purement visuelle, le paradigme miroir, nous montrons tout d’abord l’influence des afférences proprioceptives musculaires controlatérales dans l’estimation visuo-proprioceptive des mouvements d’un segment corporel, le bras. En effet, nos résultats montrent que cette illusion de mouvement d’un bras caché derrière un miroir, crée par la réflexion du bras controlatéral en mouvement dans ce miroir, émerge de la prise en compte des informations visuelles mais également des afférences proprioceptives bilatérales. Dans un deuxième temps, pour estimer plus précisément les contributions relatives et interactions entre différentes modalités, nous appliquons des stimulations sensorielles spécifiques des entrées visuelles, tactiles et/ou proprioceptives musculaires, induisant des sensations illusoires de mouvement de la main chez des sujets parfaitement immobiles. En combinant méthode psychophysique et modélisation bayésienne, nous démontrons que l’intégration des informations visuelles et tactiles permet d’optimiser la capacité à discriminer la vitesse des mouvements de la main ; Toutefois, la perception kinesthésique demeure biaisée en faveur des informations proprioceptives musculaires chez des sujets adultes jeunes. Enfin, dans une perspective développementale, nous mettons en évidence un changement de pondération relative entre ces trois différentes entrées au cours du vieillissement en faveur des afférences d’origine tactile et visuelle, probablement dû à une altération plus importante de la sensibilité proprioceptive musculaire. / In our daily live, our central nervous system uses and integrates multiples sensory information to efficiently and accurately perceive self-body and self-limb movements. An important part of the studies on this perception, called kinesthesia, address the question of the multisensory integration principles in this particular perceptive domain. This thesis brings up some answers to this question, enlightening muscle proprioception major contribution to the elaboration of multisensory kinesthetic percepts in young adults and changes occurring with age in the rules governing the integration of vision touch and muscle proprioception in kinesthesia.Revisiting a classical illusory phenomenon implicitly supposed of visual origin, the mirror paradigm, we investigate how the contralateral muscle proprioceptive afferences contribute to the visuo-proprioceptive estimation of self-arm movements. Indeed, the movement illusion of an arm, hidden behind a mirror, created by the reflection of the contralateral arm moving in this mirror seems to emerge from the integration of visual and bilateral proprioceptive feedbacks, as attested by our results. Then, in order to estimate the relative contribution and interaction of vision, touch, and muscle proprioception, we applied specific sensory stimulations on muscle proprioception, touch and/or vision. These stimulations elicit in perfectly still participants self-hand movement illusions. Combining psychophysics and Bayesian approach, we demonstrate that visual and tactile cue integration leads to an optimization of our ability to discriminate self-hand rotation velocity. Nevertheless, kinesthesia remains biased in favor of proprioceptive cues. Finally, knowing that all sensory systems decline across life span, we study the potential evolution of multisensory integration rules in kinesthesia with age. We show a profound reshaping in the weighting of the three sensory entries in older individuals, in favor of touch and vision, probably due to a relative greater impairment of muscle proprioception.

Based on research on the “McGurk Effect” (McGurk & McDonald, 1976) in speech perception, some researchers (e.g. Liberman & Mattingly, 1985) have argued that humans uniquely interpret auditory and visual (motor) speech signals as a single intended audiovisual articulatory gesture, and that such multisensory integration is innate and specific to language. Our goal for the present study was to determine if a McGurk-like Effect holds true for music perception as well, as a domain for which innateness and experience can be disentangled more easily than in language. We sought to investigate the effects of visual musical information on auditory music perception and judgment, the impact of music experience on such audiovisual integration, and the possible role of eye gaze patterns as a potential mediator for music experience and the extent of visual influence on auditory judgments. 108 participants (ages 18-40) completed a questionnaire and melody/rhythm perception tasks to determine music experience and abilities, and then completed speech and musical McGurk tasks. Stimuli were recorded from five sounds produced by a speaker or musician (cellist and trombonist) that ranged incrementally along a continuum from one type to another (e.g. non-vibrato to strong vibrato). In the audiovisual condition, these sounds were paired with videos of the speaker/performer producing one type of sound or another (representing either end of the continuum) such that the audio and video matched or mismatched to varying degrees. Participants indicated, on a 100-point scale, the extent to which the auditory presentation represents one end of the continuum or the other. Auditory judgments for each sound were then compared based on their visual pairings to determine the impact of visual cues on auditory judgments. Additionally, several types of music experience were evaluated as potential predictors of the degree of influence visual stimuli had on auditory judgments. Finally, eye gaze patterns were measured in a different sample of 15 participants to assess relationships between music experience and eye gaze patterns, and eye gaze patterns and extent of visual on auditory judgments. Results indicated a reliable “musical McGurk Effect” in the context of cello vibrato sounds, but weaker overall effects for trombone vibrato sounds and cello pluck and bow sounds. Limited evidence was found to suggest that music experience impacts the extent to which individuals are influenced by visual stimuli when making auditory judgments. The support that was obtained, however, indicated the possibility for diminished visual influence on auditory judgments based on variables associated with music “production” experience. Potential relationships between music experience and eye-gaze patterns were identified. Implications for audiovisual integration in the context of speech and music perception are discussed, and future directions advised.

Os profissionais em design de exposição, em sua responsabilidade enquanto projetistas devem pensar sobre como desenvolver sistemas e/ou recursos para fruição artística que atendam a pessoas com deficiência visual, incluindo a forma como uma pessoa totalmente habilitada utiliza este objeto/ambiente. Assim, apresenta-­‐se a pesquisa intitulada Design para experiência multissensorial em museus: fruição de objetos culturais por pessoas com deficiência visual. Seu objetivo é propor os requisitos para o projeto e desenvolvimento de recursos multimodais táteis e sonoros para fruição de objetos culturais pelo usuário com deficiência visual em museus, segundo uma abordagem integrada com base no design para a experiência. Para tanto busca-­‐se referências em áreas como o Design para Experiência e Design Emocional, Museologia, Fabricação Digital, Comunicação Acessível, Semiótica Social e Multimodalidade, entre outros. O embasamento metodológico ampara-­‐se nas abordagens de pesquisa qualitativa e pesquisa aplicada para coleta e análise de dados referente a aferição da experiência do usuário. Assim, são empregadas as técnicas de observação direta, entrevistas e Engenharia Kansei para caracterizar e compreender as experiências dos sujeitos da pesquisa, bem como para compreender os processos de desenvolvimento e os materiais para produção de recursos para acessibilidade em museus e sua consequente avaliação. A verificação do trabalho ocorreu no Museu de Porto Alegre – Joaquim Felizardo. Os requisitos de projeto para promoção da experiência multissensorial por pessoas com deficiência visual validaram a hipótese elencada em resposta ao problema de pesquisa. Além dos requisitos de projeto, chegou-­‐se a novos instrumentos para o auxílio no projeto de recursos de acessibilidade, tal como: metodologia proposta para o desenvolvimento de recursos multimodais táteis e sonoros segundo uma abordagem integrada; diretrizes para o desenvolvimento de recursos táteis e sonoros para pessoas com deficiência visual em museus; instrumento para o relacionamento de elementos táteis e sonoros segundo uma abordagem integrada; e, instrumento para avaliação da experiência pelo usuário por meio de escalas de diferencial semântico. / Professionals in exhibition design, in their responsibility as designers should think about how to develop systems and/or resources for artistic fruition that meets the visually impaired, including how a fully qualified person uses this object/environment. Thus, presents this research entitled Design for multisensory experience in museums: fruition of cultural objects by visually impaired people. This aims to propose the requirements for the design and development of tactile and audible multimodal resources for the fruition of cultural objects by the users with visual impairment in museums, in an integrated approach based on the design for the experience. Therefore look for references in areas as the Design for Experience and Emotional Design, Museology, Digital Manufacturing, Affordable Communication, Social Semiotics and Multimodality, among others. The methodological basis employs the qualitative research approaches and applied research for data collection and analysis concerning user experience’s. Thus, use of the techniques of direct observation, interviews and Kansei Engineering to characterize and understand the experiences of research subjects, as well as understand the development of processes and materials for the production of resources for accessibility in museums and their consequent evaluation. The work of verification took place at the Museum of Porto Alegre -­‐ Joaquim Felizardo. The project requirements elected for the promotion of multisensory experience for the visually impaired validated the hypothesis in response to the research problem. In addition to the project requirements, came to new tools to aid in the project of accessibility resources, such as: proposed methodology for the development of tactile and audible multimodal resources through an integrated approach; guidelines for the development of tactile and sound resources for visually impaired people in museums; instrument for the relationship of tactile and sound elements through an integrated approach; and, tool for evaluation of the experience by the user by semantic differential scales.

The 21st century automobile has become more than just a simple tool for transportation and more of a brand image or a way for drivers to express their personal taste. This has made it increasingly important for automotive manufacturers to design the driver experience and driver feeling so as to tailor their preferences and interests. Currently there is not enough information on how to design or brand the communication of meaningful feedback from the automobile to the driver. With the development of new advanced technologies such as electric steer-by-wire systems or electric automobiles, the need to provide meaningful feedback to the driver plays a central role in the experience of using the new driving technology. Thus it is important to understand how to assess the emotional response to the stimuli reaching the driver so to be able to optimise at later stage the perceived experience. Steering wheel vibration feedback plays an important role for the driver’s control input when driving. There is currently a lack of research on the formal assessment criteria of driver emotional response used to define automotive steering wheel vibration feedback, therefore this thesis proposes a newly Driver Emotional Semantic (DES) Scale to answer the research question: “How can the emotional response to steering wheel vibration be assessed?”. This study starts with a comparison of a questionnaire survey (Exp.1) and a laboratory test (Exp.2) to identify if a correlation exists between the emotional ratings measured from the expected driver’s perception of the vibration and the experienced emotional feeling of steering wheel vibration. The work then defines a semantic scale to capture the vibrational vocabulary used by the driver to express their feeling of perceived vibration during real-road driving scenarios. Experiment 3 was therefore carried out to gather the underlying semantic descriptors used by drivers during driving scenarios. To test the reliability of the descriptive pairs of the DES rating scale developed, two evaluations of the assessment criteria were carried out: in real road scenarios (Exp.4) and laboratory test setting (Exp.5). Current research findings of this thesis suggest that the consistency of the scale dimensions found in the field study has captured with greater accuracy the driver semantic experience of automotive steering wheel vibration character as compared to the laboratory experiment dimensionality. Results suggest that the main vibrotactile semantic descriptors to assess the human emotional response to automotive steering wheel vibration were found to be four: pleasant, smooth, sharp and powerful. The final proposed DES scale could help automotive research and industry determine and customise the aspects of the automobile towards drivers’ preferences of felt experience.

abstract: Sound localization can be difficult in a reverberant environment. Fortunately listeners can utilize various perceptual compensatory mechanisms to increase the reliability of sound localization when provided with ambiguous physical evidence. For example, the directional information of echoes can be perceptually suppressed by the direct sound to achieve a single, fused auditory event in a process called the precedence effect (Litovsky et al., 1999). Visual cues also influence sound localization through a phenomenon known as the ventriloquist effect. It is classically demonstrated by a puppeteer who speaks without visible lip movements while moving the mouth of a puppet synchronously with his/her speech (Gelder and Bertelson, 2003). If the ventriloquist is successful, sound will be “captured” by vision and be perceived to be originating at the location of the puppet. This thesis investigates the influence of vision on the spatial localization of audio-visual stimuli. Participants seated in a sound-attenuated room indicated their perceived locations of either ISI or level-difference stimuli in free field conditions. Two types of stereophonic phantom sound sources, created by modulating the inter-stimulus time interval (ISI) or level difference between two loudspeakers, were used as auditory stimuli. The results showed that the light cues influenced auditory spatial perception to a greater extent for the ISI stimuli than the level difference stimuli. A binaural signal analysis further revealed that the greater visual bias for the ISI phantom sound sources was correlated with the increasingly ambiguous binaural cues of the ISI signals. This finding suggests that when sound localization cues are unreliable, perceptual decisions become increasingly biased towards vision for finding a sound source. These results support the cue saliency theory underlying cross-modal bias and extend this theory to include stereophonic phantom sound sources. / Dissertation/Thesis / Masters Thesis Bioengineering 2015

abstract: Children with dyslexia have difficulty learning to read. The purpose of this study was to investigate whether the use of simultaneous multisensory structured language (multisensory) instruction promoted better letter name and sound production, word reading, and word spelling for second grade children with typical development (TD; N=6) or with dyslexia (DYS; N=5) than structured language instruction alone. The use of non-English graphemes (letters) to represent two pretend languages were used to control for children’s lexical knowledge. A multiple baseline, multiple probe across subjects single-case design, paired with an alternating treatments design, was used to compare the efficacy of multisensory and structure language interventions. Participant’s graphed data was visually analyzed and individual Tau-U and weighted Tau-U effect sizes were calculated for the outcome variables: letter name production, letter sound production, word reading, and word spelling. Both interventions had an overall effect for participants with TD and DYS, though for individual participants intervention effects varied across outcome variables. However, the multisensory intervention did not provide a clear advantage over the structured intervention for participants with TD or DYS. / Dissertation/Thesis / Doctoral Dissertation Speech and Hearing Science 2016

Os profissionais em design de exposição, em sua responsabilidade enquanto projetistas devem pensar sobre como desenvolver sistemas e/ou recursos para fruição artística que atendam a pessoas com deficiência visual, incluindo a forma como uma pessoa totalmente habilitada utiliza este objeto/ambiente. Assim, apresenta-­‐se a pesquisa intitulada Design para experiência multissensorial em museus: fruição de objetos culturais por pessoas com deficiência visual. Seu objetivo é propor os requisitos para o projeto e desenvolvimento de recursos multimodais táteis e sonoros para fruição de objetos culturais pelo usuário com deficiência visual em museus, segundo uma abordagem integrada com base no design para a experiência. Para tanto busca-­‐se referências em áreas como o Design para Experiência e Design Emocional, Museologia, Fabricação Digital, Comunicação Acessível, Semiótica Social e Multimodalidade, entre outros. O embasamento metodológico ampara-­‐se nas abordagens de pesquisa qualitativa e pesquisa aplicada para coleta e análise de dados referente a aferição da experiência do usuário. Assim, são empregadas as técnicas de observação direta, entrevistas e Engenharia Kansei para caracterizar e compreender as experiências dos sujeitos da pesquisa, bem como para compreender os processos de desenvolvimento e os materiais para produção de recursos para acessibilidade em museus e sua consequente avaliação. A verificação do trabalho ocorreu no Museu de Porto Alegre – Joaquim Felizardo. Os requisitos de projeto para promoção da experiência multissensorial por pessoas com deficiência visual validaram a hipótese elencada em resposta ao problema de pesquisa. Além dos requisitos de projeto, chegou-­‐se a novos instrumentos para o auxílio no projeto de recursos de acessibilidade, tal como: metodologia proposta para o desenvolvimento de recursos multimodais táteis e sonoros segundo uma abordagem integrada; diretrizes para o desenvolvimento de recursos táteis e sonoros para pessoas com deficiência visual em museus; instrumento para o relacionamento de elementos táteis e sonoros segundo uma abordagem integrada; e, instrumento para avaliação da experiência pelo usuário por meio de escalas de diferencial semântico. / Professionals in exhibition design, in their responsibility as designers should think about how to develop systems and/or resources for artistic fruition that meets the visually impaired, including how a fully qualified person uses this object/environment. Thus, presents this research entitled Design for multisensory experience in museums: fruition of cultural objects by visually impaired people. This aims to propose the requirements for the design and development of tactile and audible multimodal resources for the fruition of cultural objects by the users with visual impairment in museums, in an integrated approach based on the design for the experience. Therefore look for references in areas as the Design for Experience and Emotional Design, Museology, Digital Manufacturing, Affordable Communication, Social Semiotics and Multimodality, among others. The methodological basis employs the qualitative research approaches and applied research for data collection and analysis concerning user experience’s. Thus, use of the techniques of direct observation, interviews and Kansei Engineering to characterize and understand the experiences of research subjects, as well as understand the development of processes and materials for the production of resources for accessibility in museums and their consequent evaluation. The work of verification took place at the Museum of Porto Alegre -­‐ Joaquim Felizardo. The project requirements elected for the promotion of multisensory experience for the visually impaired validated the hypothesis in response to the research problem. In addition to the project requirements, came to new tools to aid in the project of accessibility resources, such as: proposed methodology for the development of tactile and audible multimodal resources through an integrated approach; guidelines for the development of tactile and sound resources for visually impaired people in museums; instrument for the relationship of tactile and sound elements through an integrated approach; and, tool for evaluation of the experience by the user by semantic differential scales.