Human survival critically relies on communicating a broad set of social messages including physical states and mental states. The prerequisite for any successful social communication is the shared knowledge between the sender and the receiver about what and how a specific social signal is used. To communicate the broad set of social messages in daily life, human beings have developed complex facial movement patterns as one of the most important and powerful social signals. With increasing globalization, cross-cultural interactions are fast becoming integral to modern life, which presents increasing pressure for cross-cultural communication. Specifically, a broad set of facial expressions including conversational facial expressions is critical for clear communication because they guide the flow of social exchanges. Yet, our knowledge of such facial movement patterns is relatively limited in terms of their functions in different cultural context – for example, whether these important everyday facial expressions are understood across cultures or cause cross-cultural confusions. In this thesis, I explored how facial movement patterns are used in Western and East Asian culture to communicate a broad set of social messages including physical states and mental states. Specifically, I objectively characterized the structure of dynamic facial expression patterns using 4D computer graphics and a data-driven social psychophysics method. I then examined the role of culture in signaling and receiving facial expressions using the signal detection theory and Mutual Information analysis. Together, my results reveal for the first time how specific facial movement patterns are used to communicate a broad set of social messages in Western and East Asian culture and how culture shapes the signalling and perception of such facial expressions in cross-cultural communication. Finally, I discussed the implication of my results in the field of psychology, computer science and social robotics, with links to my future work on developing a mathematical model of face social signalling and transfer this knowledge to socially and culturally sensitive conversational agents.

Modern cognitive science is characterized by a number of different proposals that aim to provide a means of conceptualizing cognitive systems, and stating the kinds of information that are needed to study the phenomenon from a number of different perspectives. Recently, there has been a growing trend to think about cognitive systems in terms of embodiment and embeddedness. These two factors emphasize, respectively, the role the body and the environment in shaping the nature of cognitive processes. However, what these theses currently lack is a well-articulated conceptual framework for guiding their work. Specifically, it lacks an explicit account of the types of levels of analysis that are needed to study cognition from these perspectives. This explicitness is a feature of alternative conceptions of the mind, the classical approaches and connectionist approaches that despite their differences, both draw upon the notions of computation and representation in explaining cognitive life. Where these proposals fall short is their lack of focus of the kinds of factors that motivate the embedded and embodied view. This thesis aims to fill this gap in the literature by beginning to develop an explicit conceptual framework for embodied and embedded cognition. It is proposed that such a framework can be based upon the principles of Systemic Functional Linguistics (SFL), a theory of language that considers communication in its eco-social context. Specifically, it proposes a number of levels of analysis that are useful for such a framework. Motivation for such a proposal is that: (1) both language and general cognition are semiotic systems, (2) they can both be described as dynamic, open systems, and (3) all the types of levels of analysis previously identified in cognitive science are also found in the SFL literature. This thesis, therefore, investigates whether such a cross-disciplinary application has the potential to be successful. In order to achieve the above goals, this thesis examines the following possible applications of notions from SFL to conceptual frameworks in cognitive science. Firstly, this thesis examines the theoretical notions of stratification, instantiation, delicacy and rank in SFL and then continues to outline their application to this neighbouring field. These are the levels of analysis indicated above. Stratification allows cognitive systems to be considered simultaneously at a number of different levels of interpretation. Specifically, it considers cognitive systems to consist of a neurophysiological system and a psychological system, i.e. a system of possible neurological states, and a system of possible cognitive acts. Delicacy describes these systems in more or less specific terms. Rank identifies the structural unit at which meaningful behaviour is taking place, and instantiation describes the interplay between observed instances and the underlying system. This last factor enables a dynamic view of cognition to be proposed. Furthermore, this thesis examines whether ideas from SFL concerning text-context interactions can prove useful for considering organism-environment interactions. The main proposal here is that the environment need not be considered as a vast, heterogeneous entity, but instead is itself a system involved in the creation of meaning. In this way, it can also be described along the dimensions discussed above, although at this stage only the notions of stratification and instantiation can be discussed in any depth. Also, it proposes that the environment can be described according to three environmental parameters: (1) the objective, that describes the objects and relations between them, (2) the relational, which describes relations between the individual and said objects, and (3) the goal, which provides the situation the individual is attempting to bring about. In this way structure can be brought to studies of cognitive embeddedness. Finally, this work considers the notion of abstraction in SFL as it relates to the dimensions of stratification, delicacy and rank. All of these have been described as operating over this relation, but this thesis argues that its meaning is different in each case. It argues this by considering the links between abstraction and the related notions of omission, generalization and decontextualization. In line with the overall goal of this work, it is proposed that a more explicit understanding of these terms can help the process of beginning to build the framework under discussion, by linking data to the overall framework. In sum, this thesis begins to develop a novel framework for cognitive science that is based upon the principles of Systemic Functional Linguistics. It is hoped that this work can bring some extra clarity to the notions of embeddedness and embodiment in cognitive science and show how cognitive systems may be studied from this perspective.

Traditionally, psychologists and linguists have assumed that phonological speech errors result from the substitution of well-formed segments. However, there is growing evidence from acoustic and articulatory analyses of these errors which suggests ac- tivation from competing phonological representations can cascade to articulation. This thesis assumes a cascading model, and investigates further constraints for psy- cholinguistic models of speech production. Two major questions are addressed: whether such a cascading model should include feedback; and whether phonologi- cal representations are still required if articulation is not well-formed. In order to investigate these questions a new method is introduced for the analysis of artic- ulatory data, and its application for analysing EPG and ultrasound recordings is demonstrated. A speech error elicitation experiment is presented in which acoustic and elec- tropalatography (EPG) signals were recorded. A transcription analysis of both data sets tentatively supports a feedback account for the lexical bias effect. Cru- cially, however, the EPG data in conjunction with a perceptual experiment highlight that categorising speech errors is problematic for a cascaded view of production. Therefore, the new analysis technique is used for a reanalysis of the EPG data. This allows us to abandon a view in which each utterance is an error or not. We demon- strate that articulation is more similar to a competing phonological representation when the competitor yields a real word. This pattern firmly establishes evidence for feedback in speech production. Two additional experiments investigate whether phonological representations, in addition to lower-level representations (e.g., features), are required to account for ill-formed speech. In two tongue-twister experiments we demonstrate with both EPG and ultrasound, that articulation is most variable when there is one compet- ing feature, but not when there are two competing features. This pattern is best accounted for in a feedback framework in which feature representations feedback to reinforce phonological representations. Analysing articulation using a technique which does not require the categorisation of responses allows us to investigate the consequences of cascading. It demonstrates that a cascading model of speech production requires feedback between levels of representation and that phonemes should still be represented even if articulation is malformed.

edited by Uli H. Frauenfelder and Lorraine Komisarjevsky Tyler. / "Reprinted from Cognition ... volume 25 (1987)"--T.p. verso. "A Bradford book." / Includes bibliographies and index.

The current thesis investigates the link between gestures and metaphor. In Chapter 3, we investigated whether left-hand gestures improve metaphor explanation compared to right-hand gestures or not gesturing at all. Additionally, we collected individual measurements for hemispheric involvement during speech production using the mouth asymmetry technique. We found a left-over-right hand gesturing advantage, which was higher for those with stronger right hemispheric involvement during speech production. This finding suggested that gestures’ self-oriented functions are hemisphere specific. In Chapter 4, we investigated whether left-hand gestures rather than taps trigger metaphorical language use. We found no such evidence, but we found that gestures compared to taps increased the number of words uttered, which in turn led to the use of more metaphors. This points towards gestures’ facilitative effect on speech production, but further research is needed to pin-point exactly what process is facilitated. In Chapter 5, we investigated whether gestures with a particular hand, when produced without speech, prime semantic categorisation of sentences (metaphorical and literal). We found no evidence for priming effects, and further research is needed to examine the effects that gestures, when produced alone might have on semantic processing. Finally, in Chapter 6 we found that producing content compared to function words, makes metaphor processing right hemisphere specific. This indicated that semantic processing is the key to the lateralisation of metaphor processing. The results validated the use of the mouth asymmetry technique as an indirect measurement of hemispheric involvement during speech production tasks.

Linguistics has developed elaborate accounts of the <u>social</u> aspects of language use - 'how to do things with words' - but the <u>emotional-dynamic</u> aspects have hitherto received less attention. Such discussions of emotive or affective meaning as there have been have tended to concentrate on the linguistic resources that are coded into the language system, rather than the dynamics of emotional interaction enacted through language use. The clinical discipline of Kleinian psychoanalysis, by contrast, has made emotional dynamics its central concern. Furthermore the main tool of the psychoanalyst's trade is the verbal interpretation of the patient's material, much of which is itself verbal. These factors have led to the development in Kleinian psychodynamic theory of a particularly rich vocabulary for understanding emotional-dynamic interaction, and specifically those aspects which are verbally enacted. The goal of this thesis is to outline a linguistic theory of emotional dynamics based on insights derived from Kleinian psychoanalysis. It aims to extrapolate from a clinical context Kleinian ideas that can be integrated with those of the school of Linguistic thought that has emphasised the dynamic aspects of locally-managed discourse meaning.

Theories of embodiment propose that our bodily states can influence affective processing. This thesis investigated the possibility that facial feedback (i.e., afferent signals from facial muscles) can influence the interpretation of affective stimuli. One study tested the effect of overt smiling and frowning on the interpretation of short descriptions of everyday events. Smiling, as compared to frowning, led to more positive evaluations, but only for participants who were aware of the emotional relevance of their expressions. A second study tested whether subtle changes in facial activation (elicited by unconsciously presented happy/angry facial expressions) led to changes in evaluations of ambiguous target symbols. While angry prime faces, as compared to happy prime faces, induced more frowning (as measured via electromyography), this change in facial activation did not translate into a behavioural effect on subsequent evaluations. A third study investigated the relation between naturally occurring facial reactions and interpretations of both clearly valenced and ambiguous facial expressions. Results indicate that facial reactivity predicts participants' self-reports of their own emotional reactions towards others' expressions (Experiment 1). A relation between facial reactions and interpretations of the expression senders' emotional states was only found in cases in which participants with high sensitivity towards their own bodily states (as measured with a test of interoceptive accuracy) tried to interpret ambiguous expressions (Experiment 2). In a last experiment, prolonged presentation of emotional prime faces led to expression-congruent facial reactions, but resulted in expression-incongruent behavioural reactions in both classification speed and interpretative tendency of emotional target faces. Overall, this thesis suggests that facial feedback is not generally involved in the interpretation of affective stimuli, but that it might contribute to evaluative processes only under special circumstances.

Les interactions sociales sont souvent vues comme des échanges d’informations au cours desquels des états mentaux individuels se succèdent. Comment pouvons-nous dès lors être « ensemble » et partager un moment ? Dans cette thèse, nous optons pour une perspective centrée sur la coordination temporelle des comportements des personnes en interaction, en nous intéressant aux phénomènes de dynamiques collectives incarnées. Ces dynamiques coordonnent communément les comportements et émergent directement du processus d’interaction mutuel lui-même. Pour évaluer cette proposition, nous avons mené trois expériences. Notre méthodologie générale distinguait trois situations types : une situation de coordination individuelle, une situation pseudo-sociale dans laquelle le comportement doit être coordonné à celui d’une autre personne sans que la réciproque soit vraie, et enfin une situation dans laquelle l’interaction est mutuelle. Une première expérience nous a permis de montrer que la mutualité de l’interaction suffisait à induire une coordination entre les participants, dont les comportements étaient pris dans une dynamique collective qui leur échappait totalement. Ensuite, nous avons montré que la mutualité de l’interaction augmentait la stabilité des interactions rythmiques et provoquait un appariement de la complexité des comportements. Enfin, nous avons montré qu’en dépit d’un environnement commun très structuré, la mutualité de l’interaction induisait une organisation légèrement plus coordonnée du temps. Nous discutons ces résultats en regard de notre objet de recherche et de notre arrière-plan théorique / Social interactions are mostly seen as information exchanges during which individual mental states follow each other. How could we, accordingly, be « toghether » and share a moment ? In this thesis, we took a perspective focused on the temporal coordinations of interpersonal behaviors as they occur in the course of interactions, by studying the phenomenon of embodied collective dynamics. Such dynamics manifest coordinates behaviors mutually and emerge from the interaction process itself. To evaluate this hypothesis, we devised three experimental paradigms. We used a general methodology in which three typical situations are distinguished : the individual situation in which participants possess all the capacities to pursue the goal of the task, a pseudo-social situation in which behavior can be coordinated to the mouvement of an other, and a situation in which the interaction is mutual. In the first experiment, we showed the mutuality of interaction was sufficient to induce participants, whose behaviors were caught in collective dynamics unbeknownst to them. Then, we showed that mutuality of interaction was enhancing the stability of rhythmic interactions as well as the complexity matching between their behaviors. Finally, we showed that, in despite of a temporary structured environment, mutuality of interaction induced a slight more coordinated temporal organization of behaviors. We discuss these results in respect of our object of research and our theoretical background

In this thesis we present novel methods for constructing and fitting 2d models of shape and appearance which are used for analysing human faces. The first contribution builds on previous work on discriminative fitting strategies for active appearance models (AAMs) in which regression models are trained to predict the location of shapes based on texture samples. In particular, we investigate non-parametric regression methods including random forests and Gaussian processes which are used together with gradient-like features for shape model fitting. We then develop two training algorithms which combine such models into sequences, and systematically compare their performance to existing linear generative AAM algorithms. Inspired by the performance of the Gaussian process-based regression methods, we investigate a group of non-linear latent variable models known as Gaussian process latent variable models (GPLVM). We discuss how such models may be used to develop a generative active appearance model algorithm whose texture model component is non-linear, and show how this leads to lower-dimensional models which are capable of generating more natural-looking images of faces when compared to equivalent linear models. We conclude by describing a novel supervised non-linear latent variable model based on Gaussian processes which we apply to the problem of recognising emotions from facial expressions.

Voices are used as a vehicle for language, and variation in the acoustic properties of voices also contains information about the speaker. Listeners use measurable qualities, such as pitch and formant traits, as cues to a speaker’s physical stature and attractiveness. Emotional states and personality characteristics are also judged from vocal stimuli. The research contained in this thesis examines vocal masculinity, aesthetics and personality, with an emphasis on the perception of prosocial traits including trustworthiness and cooperativeness. I will also explore themes which are more cognitive in nature, testing aspects of vocal stimuli which may affect trait attribution, memory and the ascription of identity. Chapters 2 and 3 explore systematic differences across vocal utterances, both in types of utterance using different classes of stimuli and across the time course of perception of the auditory signal. These chapters examine variation in acoustic measurements in addition to variation in listener attributions of commonly-judged speaker traits. The most important result from this work was that evaluations of attractiveness made using spontaneous speech correlated with those made using scripted speech recordings, but did not correlate with those made of the same persons using vowel stimuli. This calls into question the use of sustained vowel sounds for the attainment of ratings of subjective characteristics. Vowel and single-word stimuli are also quite short – while I found that attributions of masculinity were reliable at very short exposure times, more subjective traits like attractiveness and trustworthiness require a longer exposure time to elicit reliable attributions. I conclude with recommending an exposure time of at least 5 seconds in duration for such traits to be reliably assessed. Chapter 4 examines what vocal traits affect perceptions of pro-social qualities using both natural and manipulated variation in voices. While feminine pitch traits (F0 and F0-SD) were linked to cooperativeness ratings, masculine formant traits (Df and Pf) were also associated with cooperativeness. The relative importance of these traits as social signals is discussed. Chapter 5 questions what makes a voice memorable, and helps to differentiate between memory for individual voice identities and for the content which was spoken by administering recognition tests both within and across sensory modalities. While the data suggest that experimental manipulation of voice pitch did not influence memory for vocalised stimuli, attractive male voices were better remembered than unattractive voices, independent of pitch manipulation. Memory for cross-modal (textual) content was enhanced by raising the voice pitch of both male and female speakers. I link this pattern of results to the perceived dominance of voices which have been raised and lowered in pitch, and how this might impact how memories are formed and retained. Chapter 6 examines masculinity across visual and auditory sensory modalities using a cross-modal matching task. While participants were able to match voices to muted videos of both male and female speakers at rates above chance, and to static face images of men (but not women), differences in masculinity did not influence observers in their judgements, and voice and face masculinity were not correlated. These results are discussed in terms of the generally-accepted theory that masculinity and femininity in faces and voices communicate the same underlying genetic quality. The biological mechanisms by which vocal and facial masculinity could develop independently are speculated.