The world is filled with richly diverse sounds which we are able to perceptually distinguish using a variety of properties. One of these properties is the amplitude envelope, or the intensity of a sound over time. While it is common in the real world for sounds to have time-varying amplitude envelopes, the majority of sounds used in perceptual research have time-invariant or unspecified amplitude envelopes. The aim of the present thesis is twofold. Because many of the studies using time-invariant or undefined envelopes make use of very short sounds (below 100 msec), the first experiment aimed to determine the duration required for discriminating among three different envelopes: flat (invariant), ramped (increasing in intensity over time), and damped (decreasing in intensity over time). In Experiment 1, participants took part in a 2-alternative forced choice, psychophysical staircase paradigm in which they indicated which of two envelopes they thought they were listening to. Results showed that, when telling ramped tones apart from either flat or damped tones, participants showed discrimination thresholds below 50 msec, while they had thresholds of approximately 75-80 msec when differentiating flat from damped tones. Because amplitude envelope has been shown to impact audiovisual integration and the perceptual system is sensitive to interaural envelope differences when localizing sounds, the second experiment aimed to determine whether amplitude envelope could modulate the visual bias present in spatial ventriloquism, an audiovisual illusion where the perceived location of a sound is influenced by the location of a visual stimulus. In Experiment 2, participants performed a psychophysical staircase task which measured their accuracy in localizing sounds with flat and damped envelopes, with or without a simultaneous flash on the screen in front of them. Results showed that, at durations above the envelope discrimination thresholds found in Experiment 1 (83 msec), there was no visual bias on perceived location of the sound, while the bias was present at a duration below this threshold (16 msec). Together, these results add to the mounting evidence suggesting that amplitude envelope has profound and varied effects on our perception of sounds, and is an important property to consider when designing experiments. / Thesis / Master of Science (MSc)

abstract: Sound localization can be difficult in a reverberant environment. Fortunately listeners can utilize various perceptual compensatory mechanisms to increase the reliability of sound localization when provided with ambiguous physical evidence. For example, the directional information of echoes can be perceptually suppressed by the direct sound to achieve a single, fused auditory event in a process called the precedence effect (Litovsky et al., 1999). Visual cues also influence sound localization through a phenomenon known as the ventriloquist effect. It is classically demonstrated by a puppeteer who speaks without visible lip movements while moving the mouth of a puppet synchronously with his/her speech (Gelder and Bertelson, 2003). If the ventriloquist is successful, sound will be “captured” by vision and be perceived to be originating at the location of the puppet. This thesis investigates the influence of vision on the spatial localization of audio-visual stimuli. Participants seated in a sound-attenuated room indicated their perceived locations of either ISI or level-difference stimuli in free field conditions. Two types of stereophonic phantom sound sources, created by modulating the inter-stimulus time interval (ISI) or level difference between two loudspeakers, were used as auditory stimuli. The results showed that the light cues influenced auditory spatial perception to a greater extent for the ISI stimuli than the level difference stimuli. A binaural signal analysis further revealed that the greater visual bias for the ISI phantom sound sources was correlated with the increasingly ambiguous binaural cues of the ISI signals. This finding suggests that when sound localization cues are unreliable, perceptual decisions become increasingly biased towards vision for finding a sound source. These results support the cue saliency theory underlying cross-modal bias and extend this theory to include stereophonic phantom sound sources. / Dissertation/Thesis / Masters Thesis Bioengineering 2015

<p>Audio-visual temporal processing, and integration of auditory and visual information, is necessary for perceiving the world around us. Although previous research has indicated a slowing of temporal processing in older adults for unisensory stimuli, little work has examined the role of aging on multisensory temporal processing. The goal of this thesis is to use temporal-order judgment (TOJ) tasks to investigate age-related changes in audio-visual temporal processing. Overall, our results indicate that older adults do not demonstrate impairments on simple audio-visual TOJs, but they do exhibit deficits on more complex TOJ tasks. We also found no influence of spatial cues on TOJs for younger or older adults. Finally, we found age differences in complex TOJ tasks could not be explained by changes in the ability of older adults to detect a gap between sequential visual stimuli. The work in the thesis suggests that although there may be slowing in audio-visual temporal processing in complex situations, there are circumstances where audio-visual temporal processing is spared. By categorizing multisensory processing deficits in the elderly, we can aim to improve quality of life by preventing fails and perpetuating social interactions.</p> / Master of Science (MSc)

在人類溝通及語言使用行為中，口語和場景是構成人類跨感官理解歷程的兩項重要成分。究竟兩類資訊如何共同改變理解歷程仍待檢驗。本論文旨在探問四項研究問題：一，過去文獻對理解期間的視覺注意力運作提出兩類觀點。階層取徑主張口語優先並決定視覺表徵的處理，互動取徑則認為口語和視覺表徵可獨立影響視覺注意力。二，口語可促進指涉物體的凝視行為，然口語指涉效果是否受作業目標影響的本質仍不清楚。三，以複雜場景作為視覺情境，探討視覺複雜性和語義一致性表徵如何影響理解歷程。四，檢驗視覺刺激的預覽時間如何改變口語和場景表徵因素對理解歷程的影響。 　　本論文透過一系列視覺情境典範實驗探討以上研究問題。在每ㄧ項嘗試次中，參與者在聆聽中文語句期間同時觀看包含包含兩項物體的圖片：一為鑲嵌在一致（例如：原野）、不一致（例如：天空）和空白背景的口語指涉目標物體（例如：老虎），另一項則為口語未指涉且與背景一致的非目標物體（例如：禿鷹）。其次，四項實驗直交地操弄「作業目標」（「口語理解作業」或「場景理解作業」）和「預覽時間」（「一秒預覽」或「無預覽」）因素。 　　實驗結果發現：一，無論作業目標為何，所有實驗皆出現穩定的口語指涉效果。二，場景的視覺複雜性和語義一致性表徵不僅可獨立引導物體凝視行為，也可和口語共同決定理解期間的視覺注意力運作。三，作業目標對口語指涉效果及場景一致性效果產生差異化的調節作用。四，預覽時間有效促進口語理解作業的口語指涉效果，場景理解作業則不受影響。 　　整體而言，本論文的實驗證據支持互動取徑觀點。換言之，在跨感官理解的過程中，人類認知運作可透過協調語言、視覺和記憶等次系統，快速整合口語和場景所提供的物理和語義表徵，並依據當下情境動態地改變人類對外在世界的感官經驗。 / In human communication and language use, both speech and scene constitute the　cross-modal comprehension process. However, how these two elements combine to　affect human comprehension process has not yet been fully resolved. Four　research　questions will be examined. First, two approaches can account for the comprehension process: the hierarchical approach asserts speech plays the main part whereas the visual feature has only a supporting role, while the interactive approach states that both speech and visual feature combine to determine the comprehension process. Second, despite the speech can cause the spoken reference effect on having more fixations on its visual referent, the nature of this effect is still unclear. Third, most past studies adopted simple object array as visual context, little is known about the impact of real world scenes on the comprehension process. Fourth, whether the preview time could alter the influence of speech and scene on comprehension will be tested. A series of visual world paradigm experiments were conducted. Factors of task demand (speech comprehension vs. scene comprehension) and preview time (1-second vs. none) were orthogonally manipulated in four experiments. In each trial, participants listened to a spoken sentence in Chinese while viewing a picture with two critical objects: one is the mentioned target object (e.g., tiger), which was embedded in either a consistent, inconsistent or blank background; the other is an unmentioned non-target object (e.g., eagle) that was always consistent with its background. Several findings were found. First, the reliable spoken reference effect were shown regardless of the task demand was given. Second, the visual complexity and scene consistency not only can individually guide fixations on objects, but can work together with the speech to determine the visual attention during comprehension. Third, task demand could differently modulate the spoken reference and scene consistency effect, respectively. Fourth, preview time significantly enhances the spoken reference effect in the speech comprehension task, whereas no impact was observed in the scene comprehension task. These evidence supported the view of interactive approach. In conclusion, human’s different cognitive systems, including language, vision and memory, can interact with each other and cause the moment to moment experience of how we understand the complex world around us.

Le terme stéréopsie renvoie à la sensation de profondeur qui est perçue lorsqu’une scène est vue de manière binoculaire. Le système visuel s’appuie sur les disparités horizontales entre les images projetées sur les yeux gauche et droit pour calculer une carte des différentes profondeurs présentes dans la scène visuelle. Il est communément admis que le système stéréoscopique est encapsulé et fortement contraint par les connexions neuronales qui s’étendent des aires visuelles primaires (V1/V2) aux aires intégratives des voies dorsales et ventrales (V3, cortex temporal inférieur, MT). A travers quatre projets expérimentaux, nous avons étudié comment le système visuel utilise la disparité binoculaire pour calculer la profondeur des objets. Nous avons montré que le traitement de la disparité binoculaire peut être fortement influencé par d’autres sources d’information telles que l’occlusion binoculaire ou le son. Plus précisément, nos résultats expérimentaux suggèrent que : (1) La stéréo de da Vinci est résolue par un mécanisme qui intègre des processus de stéréo classiques (double fusion), des contraintes géométriques (les objets monoculaires sont nécessairement cachés à un œil, par conséquent ils sont situés derrière le plan de l’objet caché) et des connaissances à priori (une préférence pour les faibles disparités). (2) Le traitement du mouvement en profondeur peut être influencé par une information auditive : un son temporellement corrélé avec une cible définie par le mouvement stéréo peut améliorer significativement la recherche visuelle. Les détecteurs de mouvement stéréo sont optimalement adaptés pour détecter le mouvement 3D mais peu adaptés pour traiter le mouvement 2D. (3) Grouper la disparité binoculaire avec un signal auditif dans une dimension orthogonale (hauteur tonale) peut améliorer l’acuité stéréo d’approximativement 30% / Stereopsis refers the perception of depth that arises when a scene is viewed binocularly. The visual system relies on the horizontal disparities between the images from the left and right eyes to compute a map of the different depth values present in the scene. It is usually thought that the stereoscopic system is encapsulated and highly constrained by the wiring of neurons from the primary visual areas (V1/V2) to higher integrative areas in the ventral and dorsal streams (V3, inferior temporal cortex, MT). Throughout four distinct experimental projects, we investigated how the visual system makes use of binocular disparity to compute the depth of objects. In summary, we show that the processing of binocular disparity can be substantially influenced by other types of information such as binocular occlusion or sound. In more details, our experimental results suggest that: (1) da Vinci stereopsis is solved by a mechanism that integrates classic stereoscopic processes (double fusion), geometrical constraints (monocular objects are necessarily hidden to one eye, therefore they are located behind the plane of the occluder) and prior information (a preference for small disparities). (2) The processing of motion-in-depth can be influenced by auditory information: a sound that is temporally correlated with a stereomotion defined target can substantially improve visual search. Stereomotion detectors are optimally suited to track 3D motion but poorly suited to process 2D motion. (3) Grouping binocular disparity with an orthogonal auditory signal (pitch) can increase stereoacuity by approximately 30%