Human listeners possess good speaker recognition abilities, and are capable of discriminating and identifying speakers from a range of spoken utterances. However, voice recognition can be enhanced when a listener is capable of understanding the speech produced by a talker. A well-established demonstration of this is known as the “Language-Familiarity” Effect (LFE) for voice recognition. This effect manifests as an impairment for voice recognition in foreign language speech conditions, as contrasted with recognition of talkers who are speaking in a listener’s mother tongue, and has been repeatedly demonstrated across a range of different tasks and languages. The LFE has previously been conceptualized as an analogue to the even better-known “Other-Race” Effect (ORE) for face recognition, where own-race faces are better remembered than other-race faces. An influential theoretical model of the ORE posits that faces are represented in a multidimensional “face-space”, whose dimensions are shaped by perceptual experience and code for features which are diagnostic for face individuation (Valentine, 1991). Over the course of an individual’s perceptual experience, these dimensions might become attuned for own-race face recognition; as a consequence, the dimensions will be sub-optimal for other-race recognition, leading to the illusion of increased similarity among different other-race faces, relative to own-race faces – what has been termed the “they-all-look-alike” effect. The idea of a complementary “voice-space” has already been posited in the auditory domain, and might serve as a useful model for the LFE. Speakers might be individuated on the basis of diagnostic dimensions which might code for important voice-acoustical attributes. However, these dimensions might also be shaped according to linguistic experience, and voice individuation (and recognition) might be optimised when listeners can take advantage of both general voice acoustics and stored representations of their native language to tell speakers apart. The face-space hypothesis represents a plausible model for the ORE, and evidence for it has accrued through computational modelling and neuroimaging work. Conversely, however, at present it merely serves as a descriptive model for the LFE. In this thesis, I combine behavioural testing, and neuroimaging studies using functional Magnetic Resonance Imaging (fMRI) to probe the nature of the representations of native and foreign speakers. Chapter 1 provides a general overview of voice processing with an emphasis on voice recognition. Subsequently, I provide a review of relevant literature pertaining to the LFE, and introduce a brief comparison to the ORE for faces in the context of the Valentine (1991) similarity model, ending with a description of the aims of the thesis. In Chapter 2, I present the results of a behavioural experiment where native English and Mandarin speaking listeners rated all pairwise combinations of a series of English- and Mandarin-speaking voices. Crucially, the LFE does not appear to be dependent on full comprehension of the linguistic message, as young infants can better tell apart speakers in their native language than in a foreign language before their speech comprehension abilities are fully mature. This suggests that exposure to the sound-structure characteristic of infants’ nascent mother tongue might be sufficient to enhance native language speaker discrimination, in the absence of full comprehension. Therefore, to examine a counterpart in adults, speech stimuli were subjected to time-reversal, a process which precludes lexical and semantic access but which leaves intact certain phonemic properties of the original speech signal. Both the English and Mandarin listeners rated pairs of native-language voices as sounding more dissimilar than foreign voices, suggesting that the language-specific sound-structure elements remaining in the reversed speech enabled an enhanced individuation of native voices. Next, in Chapter 3, I aimed to probe the neural basis of this enhanced individuation in an fMRI experiment which was intended to capture dissimilarities among paired cerebral responses to unintelligible native and foreign speakers. Here, I did not find a direct correlate of the behavioural effect, but did find that local patterns of response estimates in the bilateral superior temporal cortex (STC) appear to “discriminate” the different language categories in both English and Mandarin listeners. Specifically, when the pairwise dissimilarity in brain responses to different speakers was collected, relatively high dissimilarity was observed for pairs consisting of a response to an English speaker and a Mandarin speaker, whereas relatively low dissimilarity was observed for pairs consisting of two English or two Mandarin speakers. In Chapter 4, I report what is, to my knowledge, the first explicit examination of the neural basis for the LFE in intelligible speech. A monolingual sample of English speakers participated in an fMRI experiment where they listened to the voices of English and Mandarin speakers. Importantly, speech stimuli in both language conditions were matched in inter-speaker acoustical variability. Combined response patterns from bilateral voice-sensitive temporal lobe regions enabled a learning algorithm to decode the identities of the voices who elicited the responses, but, crucially, only in the native speech (English) condition. Interestingly, native-language speaker decoding was also achieved from a left-hemisphere voice-sensitive region alone, but not a right-hemisphere region. This putative leftward bias might reflect a higher discriminability of native-language talkers in the brain, via an enhanced ability to individuate voices on the basis of indexical variation around stored speech-sound representations. Finally, in Chapter 5, I conclude with a general discussion of the foregoing results, their implications for an analogous conception of the LFE and ORE, and some strands of thought for future investigation.

Our world is multisensory! To function and survive in our environment we utilise all of the cross-modal sensory information available to us. Some of it may be redundant but that simply makes our decisions more reliable (e.g., Clark and Yuille, 1990; Landy et al., 1995; Ernst and Banks, 2002). Knowledge of multisensory processes, which make use of this multitude of rich cross-modal information, is growing rapidly (for a review see, Calvert et al., 2004) and changing the way both philosophers (e.g., Macpherson, 2011) and scientists (e.g., Driver and Spence, 2000) think about perception and the senses. One of the main challenges faced by this research field is how to decide which types of evidence are sufficient to prove that multisensory integration has occurred (e.g., Stein et al., 2009). To make things more complicated, the solution will be different dependent on whether integration is investigated at the neuronal, cerebral or behavioural level. Chapter 2 of this thesis provides new evidence about the integration criteria generally used in functional magnetic resonance imaging (fMRI). The choice of criteria had an obvious influence on which regions were found to integrate audiovisual information from the face and voice. Raising the question, which of these criteria should be used? Our conclusion was that it would be prudent to investigate the results of all of these criteria and; moreover, that a more cogent method of investigation would be to combine these criteria with manipulations of stimulus signal-to-noise ratio and/or congruency. At the level of the neuron we know that the relative temporal synchrony of cross-modal cues can be an important factor in determining whether the neuron displays a multisensory response or not (e.g., Stein and Meredith, 1993). Manipulating temporal synchrony is actually one of the main techniques that researchers have used to explore multisensory processes at the neuronal, cerebral and behavioural levels (see relevant chapters in, Calvert et al., 2004). The experiments presented in Chapters 3, 4 and 5 of the current thesis focus on this temporal factor of audiovisual integration: the synchrony perception process in particular. Chapters 3 and 4 present new evidence that two of the main experimental tasks used to investigate the synchrony perception process should not be used interchangeably. The experiments in those chapters asked participants to make both synchrony judgments (SJs) and temporal order judgments (TOJs) to identical audiovisual stimuli with various levels of cue asynchrony. First, Chapter 3 provided further evidence (Fujisaki and Nishida, 2009; van Eijk et al., 2008; Vatakis et al., 2008b; Vroomen and Stekelenburg, 2011) that these two tasks differ at the behavioural level. Second, Chapter 4 gave further credence to these behavioural results by using fMRI to localise, for the first time, the different neural correlates of these two tasks. Moving beyond localisation of function, Chapter 5 outlines a fMRI experiment, which explored the role that different brain regions play in the synchrony perception process. Participants performed a SJ task to a large set of asynchronous audiovisual speech stimuli. Networks of regions responding preferentially to either synchronous or asynchronous audiovisual speech stimuli were found. However, which regions comprised these networks and whether they were both localised was dependent on which asynchronous conditions were included in contrasts of interest. This indicates that differences in stimulus and contrast choice in previous research could have helped produce inconsistencies in results (Stevenson et al., 2010).

Synaesthesia is an unusual phenomenon, in which additional sensory perceptions are triggered by apparently unrelated sensory or conceptual stimuli. The main foci of this thesis lie in speech sound - colour and voice-induced synaesthesia. While grapheme-colour synaesthesia has been intensively researched, few studies have approached types of synaesthesia based on vocal inducers with detailed acoustic-phonetic and colorimetric analyses. This approach is taken here. First, a thorough examination of speech-sound - colour synaesthesia was conducted. An experiment is reported that tested to what extent vowel acoustics influence colour associations for synaesthetes and non-synaesthetes. Systematic association patterns between vowel formants and colour measures could be found in general, but most strongly in synaesthetes. Synaesthetes also showed a more consistent pattern of vowel-colour associations. The issue of whether or not speech-sound - colour synaesthesia is a discrete type of synaesthesia independent of grapheme-colour synaesthesia is discussed, and how these might influence each other. Then, two experiments are introduced to explore voice-induced synaesthesia. First, a comprehensive voice description task was conducted with voice synaesthetes, phoneticians and controls to investigate their verbal voice quality descriptions and the colour and texture associations that they have with voices. Qualitative analyses provided data about the nature of associations by the participant groups, while quantitative analyses revealed that for all groups, acoustic parameters such as pitch, pitch range, vowel formants and other spectral properties influenced colour and texture associations in a systematic way. Above all, a strong connection was found between these measures and luminance. Finally, voice-induced synaesthetes, other synaesthetes and controls participated in a voice line-up, of the kind used in forensic phonetic case work. This experiment, motivated by previous findings of memory advantages in synaesthetes in certain areas, tested whether synaesthetes’ voice memory is influenced by their condition. While no difference in performance was found between groups when using normal speech, voice-induced synaesthetes outperformed others in identifying a whispering speaker. These are the first group studies on the otherwise under-researched type of voice-induced synaesthesia, with a focus on acoustic rather than semantic analysis. This adds knowledge to the growing field of synaesthesia research from a largely neglected phonetic angle. The debate around (re)defining synaesthesia is picked up. The voice description experiment, in particular, leads to a discussion of a synaesthesia spectrum in the population, as many common mechanisms and associations were found. It was also revealed that less common types of synaesthesia are often difficult to define in a rigid way using traditional criteria. Finally, the interplay of different types of synaesthesia is discussed and findings are evaluated against the background of the existing theories of synaesthesia.

No description available.

The amount of pain we feel is not always directly related to the amount of damage our body suffers. In fact, research over the last decade has shown that the experience of pain is strongly linked with how we feel about our body, including its shape and how much we like it. Chronic pain is associated with a distorted mental representation of the body, which can have a significant impact on everyday live. Evidence suggests that multisensory illusions can modulate pain and can lead to changes in body perception. However, the factors that may contribute to previously observed analgesic effects remain unclear. This thesis aimed to systematically asses the effects of these illusions on pain and body representation in both healthy individuals and individuals with chronic pain. First, this thesis aimed to investigate the effects of multisensory illusions on body representation, sense of body ownership, and pain in healthy individuals. Chapter 2 found that multisensory illusions can alter perceived body shape, body satisfaction, without losing ownership. Chapter 3 examined whether distorting the size and shape of the virtual hand could modulate pain in healthy participants. No evidence was found for this, which could indicate that pain is not necessarily affected by virtual body ownership over a distorted hand. Having established that multisensory illusions can alter body perception and the affective experience of the body, the second aim of this thesis was to investigate the effect of multisensory illusions on body representation, body ownership, and pain in individuals with hand osteoarthritis (HOA). Chapter 4 investigated whether people with HOA performed differently on a hand laterality motor task to investigate whether people with HOA present with a disruption of the working body schema. Though no evidence was found for a general impairment on this task, the findings suggested that performance on the task was mediated by the presence of pain. Chapter 5 investigated effects of multisensory illusions on pain and pain sensitivity in individuals with HOA. The results demonstrated that viewing the body could modulate pain; and affected body representation, body ownership, and agency. Chapter 6 examined the effects of multisensory illusions on subjective and objective aspects of body perception in people with HOA of the hands, compared to healthy controls. Though no evidence was found for an analgesic effect of the illusion, results showed that participants with HOA have a disturbed experience of the size of their hand compared to healthy controls. Furthermore, the results suggest that individuals with HOA may have an abnormally high body dissatisfaction that cannot readily be altered by multisensory illusions. This thesis found mixed support for the analgesic effects of multisensory illusions on pain in HOA and concludes that the specific context in which pain occurs is important. It also highlights the many perceptual and cognitive factors that may contribute to the modulation of pain. The findings imply that future work should focus on interventions that are more portable and accessible for home use, and focus on developing research around the effects of repeated and prolonged exposure to multisensory illusions on pain, body image disturbances and body dissatisfaction.

In synaesthetic personification, personality traits and other human characteristics are attributed to linguistic sequences and objects. Such non-perceptual concurrents are different from those found in most frequently studied types of synaesthesia, in which the eliciting stimuli induce sensory experiences. Here, subjective reports from synaesthetes were analysed and the cognitive and neural mechanisms underlying personification were investigated. Specifically, the neural bases of personification were examined using functional MRI in order to establish whether brain regions implicated in social cognition are involved in implementing personification. Additional behavioural tests were used to determine whether personification of inanimate objects is automatic in synaesthesia. Subjective reports describing general characteristics of synaesthetic personification were collected using a semi-structured questionnaire. A Stroop-like paradigm was developed in order to examine the automaticity of object personification, similarly to the previous investigations. Synaesthetes were significantly slower in responding to incongruent than to congruent stimuli. This difference was not found in the control group. The functional neuroimaging investigations demonstrated that brain regions involved in synaesthetic personification of graphemes and objects partially overlap with brain areas activated in normal social cognition, including the temporo-parietal junction, precuneus and posterior cingulate cortex. Activations were observed in areas known to be correlated with mentalising, reflecting the social and affective character of concurrents described in subjective reports. Psychological factors linked with personification in previous studies were also assessed in personifiers, using empathy, mentalising and loneliness scales. Neither heightened empathy nor mentalising were found to be necessary for personification, but personifying synaesthetes in the study felt lonelier than the general population, and this was more pronounced in those who personified more. These results demonstrate that personification shares many defining characteristics with classical forms of synaesthesia. Ascribing humanlike characteristics to graphemes and objects is a spontaneous and automatic process, inducer-concurrent pairings are consistent over time and the phenomenological character of concurrents is reflected in functional neuroanatomy. Furthermore, the neuroimaging findings are consistent with the suggestions that synaesthetes have a lower threshold for activation brain regions implicated in self-projection and mentalising, which may facilitate the personification processes in synaesthesia.

Perception has traditionally been studied as a modular function where different sensory systems operate as separate and independent modules. However, multisensory integration is essential for the perception of a coherent and unified representation of the external world that we experience phenomenologically. Mounting evidence suggests that the senses do not operate in isolation but that the brain processes and integrates information across modalities. A standing debate is at what level in the processing hierarchy the sensory streams converge, for example, if multisensory speech information converges first in higher-order polysensory areas such as STS and is then fed back to sensory areas, or if information is already integrated in primary and secondary sensory areas at the early stages of sensory processing. The studies in this thesis aim to investigate this question by focussing on the spatio-temporal aspects of multisensory processing, as well as investigating phonetic and non-phonetic integration in the human brain during auditory-visual speech perception.

The physical properties of the distal stimuli activating our senses are often correlated in nature; it would therefore be advantageous to exploit such correlations to better process sensory information. Stimulus correlations can be contingent and readily available to the senses (like the temporal correlation between mouth movements and vocal sounds in speech), or can be the results of the statistical co-occurrence of certain stimulus properties that can be learnt over time (like the relation between the frequency of acoustic resonance and the size of the resonator). Over the last century, a large body of research on multisensory processing has demonstrated the existence of compatibility effects between individual features of stimuli from different sensory modalities. Such compatibility effects, termed crossmodal correspondences, possibly reflect the internalization of the natural correlation between stimulus properties. The present dissertation assesses the effects of crossmodal correspondences on multisensory processing and reports a series of experiments demonstrating that crossmodal correspondences influence the processing rate of sensory information, distort perceptual experiences and lead to stronger multisensory integration. Moreover, a final experiment investigating the effects of contingent signals’ correlation on multisensory processing demonstrates the key role of temporal correlation in inferring whether two signals have a common physical cause or not (i.e., the correspondence problem). A Bayesian framework is proposed to interpret the present results whereby stimulus correlations, represented on the prior distribution of expected crossmodal co-occurrence, operate as cues to solve the correspondence problem.

Ce doctorat a pour objectif de contribuer à une meilleure compréhension du lien, plusieurs fois démontré, entre odeurs et couleurs. Il s’organise en 4 parties correspondant à 4 grandes études réalisées dans différents pays, la France, le Liban et Taiwan, ainsi que dans différentes régions au sein d'un même pays (Liban urbain, Liban rural). La première étude effectuée dans les trois pays avait pour objectif de démontrer l’impact de la culture sur la construction du lien odeurs – couleurs. Les résultats obtenus ont mis en évidence un effet significatif du lieu de résidence des participants sur la construction du lien mais aussi du rôle prépondérant de la « fonction de l'odeur » (alimentaire, cosmétique, industrielle, …). La seconde étude interculturelle entre la France et le Liban s’est intéressée plus spécifiquement à la méthodologie employée pour réaliser les tests odeurs – couleurs. En effet, dans la littérature, deux types de procédures sont généralement employées soit la présentation de couleurs physiques que le participant peut regarder soit l’absence de présentation de couleurs physiques et l’utilisation de noms de teintes (bleu, vert, rouge, …). La comparaison de ces deux méthodes a mis en évidence un effet de la procédure tout aussi important que celui de la culture et de la fonction de l'odeur sur l'association odeur-couleur. Les données ont aussi mis en évidence un réel pouvoir d’évocation olfactive des couleurs. Pour mieux, comprendre ce phénomène, nous avons réalisé une étude en IRMf. Cette troisième étude utilisant l'IRMf a montré que le pouvoir d’évocation olfactive d’arrangements abstraits de couleurs était différent de celui obtenu à partir de représentations visuelles figuratives colorées. Les processus cognitifs mis en œuvre lors d’une évocation olfactive à partir de couleurs sont plus complexes et multimodaux impliquant des processus olfactifs, émotionnels, visio-spatial, du langage ainsi que la mémoire. La dernière partie de ce doctorat s’est intéressé à mieux comprendre comment les processus émotionnels mis en évidence par IRMf pouvaient interférer avec la construction du lien odeurs – couleurs. L’étude a été réalisée en France ainsi que dans deux régions, rurale et urbaine, du Liban. Les résultats ont montré que l'affect associé à une odeur influençait le lien odeurs-couleurs mais surtout, que le choix préférentiel de certains affects était lié à la situation socio-économique et culturel du participant. Cette thèse est donc une contribution à une meilleure compréhension du lien qui unit odeurs et couleurs. Elle en a démontré la complexité avec un effet avéré de la fonction de l’odeur dans le pays, de la méthodologie employée ainsi que du lieu de résidence et du niveau socio-culturel du participant. Elle souligne cependant la nécessité d’une approche multidisciplinaire pour en comprendre encore plus précisément l’ensemble des rouages / This doctorate aims to contribute to a better understanding of the link, several times demonstrated, between odors and colors. It is organized in 4 parts corresponding to 4 major studies carried out in different countries, France, Lebanon and Taiwan, as well as in different regions within the same country (urban Lebanon, rural Lebanon). The first study carried out in the three countries aimed at demonstrating the impact of culture on the construction of the odor-color link. The results obtained revealed a significant effect of the participants' place of residence on the construction of the link but also the predominant role of the "function of the odor" (food, cosmetics, industrial ...). The second intercultural study between France and Lebanon focused more specifically on the methodology used to carry out odor - color tests. Indeed, in the literature, two types of procedures are generally used: presentation of physical colors that the participant can see or absence of presentation of physical colors and the use of color names (blue, green, red, ...). The comparison of these two methods revealed an equally important effect between the procedure, the culture and the function of the odor on the odor-color association. The data also revealed that colors have the capacity of olfactory evocation. To better understand this phenomenon, we performed an fMRI study. This third study using fMRI showed that the olfactory evocation power of abstract color arrangements was different from that obtained from colored figurative visual representations. The cognitive processes involved in an olfactory evocation from colors arrangement, is more complex and multimodal involving olfactory, emotional, visio-spatial, language and memory processes. The final part of this Ph.D. examined how emotional evidenced by fMRI could interfere with the construction of the odor - color link. The study was carried out in France as well as in two rural and urban areas of Lebanon. The results showed that the affect associated with an odor influenced the odor-color link, but above all, that the preferential choice of certain affects was linked to the socioeconomic and cultural situation of the participant. This thesis is therefore a contribution to a better understanding of the link between odors and colors. It has demonstrated its complexity with a proven effect of the odor function in the country, the methodology used and the place of residence and socio-cultural level of the participant. However, it stresses the need for a multidisciplinary approach to understand even more precisely the whole workings

William Eric Leifur Grimson. / Based on the author's thesis (Ph.D.--Massachusetts Institute of Technology) Includes indexes. / Bibliography: p. [247]-267.