This dissertation investigates vocal behavior, measured using standard acoustic and commercial vocal analysis software, as it occurs naturally while lying, experiencing cognitive dissonance, or receiving a security interview conducted by an Embodied Conversational Agent (ECA).In study one, vocal analysis software used for credibility assessment was investigated experimentally. Using a repeated measures design, 96 participants lied and told the truth during a multiple question interview. The vocal analysis software's built-in deception classifier performed at the chance level. When the vocal measurements were analyzed independent of the software's interface, the variables FMain (Stress), AVJ (Cognitive Effort), and SOS (Fear) significantly differentiated between truth and deception. Using these measurements, a logistic regression and machine learning algorithms predicted deception with accuracy up to 62.8%. Using standard acoustic measures, vocal pitch and voice quality was predicted by deception and stress.In study two, deceptive vocal and linguistic behaviors were investigated using a direct manipulation of arousal, affect, and cognitive difficulty by inducing cognitive dissonance. Participants (N=52) made verbal counter-attitudinal arguments out loud that were subjected to vocal and linguistic analysis. Participants experiencing cognitive dissonance spoke with higher vocal pitch, response latency, linguistic Quantity, and Certainty and lower Specificity. Linguistic Specificity mediated the dissonance and attitude change. Commercial vocal analysis software revealed that cognitive dissonance induced participants exhibited higher initial levels of Say or Stop (SOS), a measurement of fear.Study three investigated the use of the voice to predict trust. Participants (N=88) received a screening interview from an Embodied Conversational Agent (ECA) and reported their perceptions of the ECA. A growth model was developed that predicted trust during the interaction using the voice, time, and demographics.In study four, border guards participants were randomly assigned into either the Bomb Maker (N = 16) or Control (N = 13) condition. Participants either did or did not assemble a realistic, but non-operational, improvised explosive device (IED) to smuggle past an ECA security interviewer. Participants in the Bomb Maker condition had 25.34% more variation in their vocal pitch than the control condition participants.This research provides support that the voice is potentially a reliable and valid measurement of emotion and deception suitable for integration into future technologies such as automated security screenings and advanced human-computer interactions.

Les travaux de recherche de la thèse portent sur l'étude et la formalisation des interactions émotionnelles Humain-Machine. Au delà d’une détection d'informations paralinguistiques (émotions, disfluences,...) ponctuelles, il s'agit de fournir au système un profil interactionnel et émotionnel de l'utilisateur dynamique, enrichi pendant l’interaction. Ce profil permet d’adapter les stratégies de réponses de la machine au locuteur, et il peut également servir pour mieux gérer des relations à long terme. Le profil est fondé sur une représentation multi-niveau du traitement des indices émotionnels et interactionnels extraits à partir de l'audio via les outils de détection des émotions du LIMSI. Ainsi, des indices bas niveau (variations de la F0, d'énergie, etc.), fournissent des informations sur le type d'émotion exprimée, la force de l'émotion, le degré de loquacité, etc. Ces éléments à moyen niveau sont exploités dans le système afin de déterminer, au fil des interactions, le profil émotionnel et interactionnel de l'utilisateur. Ce profil est composé de six dimensions : optimisme, extraversion, stabilité émotionnelle, confiance en soi, affinité et domination (basé sur le modèle de personnalité OCEAN et les théories de l’interpersonal circumplex). Le comportement social du système est adapté en fonction de ce profil, de l'état de la tâche en cours, et du comportement courant du robot. Les règles de création et de mise à jour du profil émotionnel et interactionnel, ainsi que de sélection automatique du comportement du robot, ont été implémentées en logique floue à l'aide du moteur de décision développé par un partenaire du projet ROMEO. L’implémentation du système a été réalisée sur le robot NAO. Afin d’étudier les différents éléments de la boucle d’interaction émotionnelle entre l’utilisateur et le système, nous avons participé à la conception de plusieurs systèmes : système en Magicien d’Oz pré-scripté, système semi-automatisé, et système d’interaction émotionnelle autonome. Ces systèmes ont permis de recueillir des données en contrôlant plusieurs paramètres d’élicitation des émotions au sein d’une interaction ; nous présentons les résultats de ces expérimentations, et des protocoles d’évaluation de l’Interaction Humain-Robot via l’utilisation de systèmes à différents degrés d’autonomie. / Analysing and formalising the emotional aspect of the Human-Machine Interaction is the key to a successful relation. Beyond and isolated paralinguistic detection (emotion, disfluences…), our aim consists in providing the system with a dynamic emotional and interactional profile of the user, which can evolve throughout the interaction. This profile allows for an adaptation of the machine’s response strategy, and can deal with long term relationships. A multi-level processing of the emotional and interactional cues extracted from speech (LIMSI emotion detection tools) leads to the constitution of the profile. Low level cues ( F0, energy, etc.), are then interpreted in terms of expressed emotion, strength, or talkativeness of the speaker. These mid-level cues are processed in the system so as to determine, over the interaction sessions, the emotional and interactional profile of the user. The profile is made up of six dimensions: optimism, extroversion, emotional stability, self-confidence, affinity and dominance (based on the OCEAN personality model and the interpersonal circumplex theories). The information derived from this profile could allow for a measurement of the engagement of the speaker. The social behaviour of the system is adapted according to the profile, and the current task state and robot behaviour. Fuzzy logic rules drive the constitution of the profile and the automatic selection of the robotic behaviour. These determinist rules are implemented on a decision engine designed by a partner in the project ROMEO. We implemented the system on the humanoid robot NAO. The overriding issue dealt with in this thesis is the viable interpretation of the paralinguistic cues extracted from speech into a relevant emotional representation of the user. We deem it noteworthy to point out that multimodal cues could reinforce the profile’s robustness. So as to analyse the different parts of the emotional interaction loop between the user and the system, we collaborated in the design of several systems with different autonomy degrees: a pre-scripted Wizard-of-Oz system, a semi-automated system, and a fully autonomous system. Using these systems allowed us to collect emotional data in robotic interaction contexts, by controlling several emotion elicitation parameters. This thesis presents the results of these data collections, and offers an evaluation protocol for Human-Robot Interaction through systems with various degrees of autonomy.

A utilização de jogos sérios em muitas atividades, incluindo casos de saúde, como o processo de reabilitação motora, tem demonstrado resultados satisfatórios que encorajam o desenvolvimento de novas aplicações neste cenário. Jogos podem tornar tais atividades mais interessantes e divertidas para os pacientes, como também auxiliar as etapas do processo de reabilitação. Nestas aplicações, estratégias que visam a manutenção do nível de motivação do usuário durante a utilização são muito importantes. Assim, esta pesquisa investiga a adaptação de contexto em jogos sérios com a utilização de técnicas de Computação Afetiva. A proposta consiste em um framework que torna mais baixo ao programador o custo de implementação da adaptação afetiva em jogos e permite que o fisioterapeuta configure as adaptações que serão executadas no jogo conforme o perfil dos pacientes. Com o intuito de verificar a viabilidade da proposta, dois jogos para reabilitação motora e uma versão do framework foram implementados, permitindo a realização de experimentos com programadores, fisioterapeutas e pacientes. Os resultados obtidos permitem concluir que a abordagem proposta tende a proporcionar grande impacto social e tecnológico / The use of serious games in many activities, including health cases, like the motor rehabilitation process, has demonstrated results that encourage the development of new applications in this scenario. These activities can be more interesting and funnier by using games, as well as help the patients to execute the steps of the rehabilitation process. In these applications, strategies to maintain the user\'s motivation level during the game are very important. Thus, in this research, we investigated the context adaptation on serious games using techniques of Affective Computing. The proposal consists of a framework that makes the cost of implementing affective adaptation in games lower to programmers and allows the physiotherapists to configure the adaptations that will be executed in the game, according to the profile of the patients. In order to verify the feasibility of the proposal, two games for motor rehabilitation and a version of the framework were implemented, allowing the realization of experiments with programmers, physiotherapists, and patients. The results obtained allow us to conclude that the proposed approach tends to provide great social and technological impact

Uma das principais motivações da Inteligência Artificial no contexto dos sistemas de entretenimento digital é criar personagens adaptáveis a novas situações, pouco previsíveis, com aprendizado rápido, memória de situações passadas e uma grande diversidade de comportamentos consistente e convincente ao longo do tempo. De acordo com recentes estudos desenvolvidos nos campos da Neurociência e da Psicologia, a capacidade de resolução de problemas não está unicamente atrelada à facilidade na manipulação de símbolos, mas também à exploração das características do ambiente e à interação social, que pode ser expressa na forma de fenômenos emocionais. Os resultados desses estudos confirmam o papel fundamental que cumprem a personalidade e as emoções nas atividades de percepção, planejamento, raciocínio, criatividade, aprendizagem, memória e tomada de decisão. Quando módulos para a manipulação de personalidade e emoções são incorporados à teoria de agentes, é possível a construção de Agentes com Comportamento Convincente (Believable Agents). O objetivo principal deste trabalho é desenvolver e implementar uma arquitetura de agentes inteligentes para construir personagens sintéticos cujos estados afetivos influenciam em suas atividades cognitivas. Para o desenvolvimento de tal arquitetura utilizou-se o modelo BDI (Beliefs, Desires e Intentions) como base e aos módulos existentes em uma implementação desse modelo foi incluído um Módulo Afetivo. Esse Módulo Afetivo é constituído por três submódulos (Personalidade, Humor e Emoção) e deve impactar nas atividades cognitivas de percepção, memória e tomada de decisão do agente. Duas provas de conceito (experimentos) foram construídas : a simulação do problema do ``Dilema do Prisioneiro Iterado\'\' e a versão computadorizada do ``Jogo da Memória\'\'. A construção desses experimentos permitiu avaliar empiricamente a influência da personalidade, humor e emoção nas atividades cognitivas dos agentes, e consequentemente no seu comportamento. Os resultados evidenciam que a utilização da nova arquitetura permite a construção de agentes com comportamentos mais coerentes, adaptativos e cooperativos quando comparados aos de agentes construídos com arquiteturas cujas atividades cognitivas não consideram o estado afetivo, e também produz um comportamento mais próximo de um agente humano que de um comportamento ótimo ou aleatório. Essa evidência de sucesso, apresentada nos resultados, mostra que os agentes construídos com a arquitetura proposta nessa dissertação indicam um avanço na direção do desenvolvimento dos Agentes com Comportamento Convincente. / One of the main motivations of Artificial Intelligence in the context of the digital entertainment systems is to create characters that are adaptable to new situations, unpredictable, fast learners, enable with memory of past situations and a variety of consistent and convincing behavior over time. According to recent studies conducted in the fields of Neuroscience and Psychology, the ability to solve problems is not only related to the capacity to manipulate symbols, but also to the ability to explore the environment and to engage into social interaction, which can be expressed as emotional phenomena. The results of these studies confirm the key role the personality and emotions play in the activities of perception, attention, planning, reasoning, creativity, learning, memory and decision making. When modules for handling personality and emotion, are incorporated in a theory of agents, it is possible to build Believable Agents. The main objective of this work is to develop and implement an intelligent agent architecture to build synthetic characters whose affective states influence their cognitive activities. To develop such architecture the BDI model (Beliefs, Desires and Intentions) was used as a basis, to which an Affective Module was included. The Affective Module consists of three sub-modules (Personality, Mood and Emotion), which influence the cognitive activities of perception, memory and decision making. Finally, two proofs of concept were built: the simulation of the problem of ``Iterated Prisoner\'s Dilemma\'\' and the computerized version of the ``Memory Game.\'\' The construction of these experiments allowed to evaluate empirically the influence of personality, mood and emotion in cognitive activities of agents and consequently in their behavior. The results show that using the proposed architecture one can build agents with more consistent, adaptive and cooperative behaviors when compared to agents built with architectures whose affective states do not influence their cognitive activities. It also produces a behavior that is closer to a human user than that of optimal or random behavior. This evidence of success, presented in the obtained results, show that agents built with the proposed architecture indicate an advance towards the development of Believable Agents.

The automotive industry is facing significant technological and sociological shifts, calling for an improved understanding of driver and passenger behaviours, emotions and needs, and a transformation of the traditional automotive design process. This research takes a human-centred approach to automotive research, investigating the users' emotional states during automobile driving, with the goal to develop a framework for automotive emotion research, thus enabling the integration of technological advances into the driving environment. A literature review of human emotion and emotion in an automotive context was conducted, followed by three driving studies investigating emotion through Facial-Expression Analysis (FEA): An exploratory study investigated whether emotion elicitation can be applied in driving simulators, and if FEA can detect the emotions triggered. The results allowed confidence in the applicability of emotion elicitation to a lab-based environment to trigger emotional responses, and FEA to detect those. An on-road driving study was conducted in a natural setting to investigate whether natures and frequencies of emotion events could be automatically measured. The possibility of assigning triggers to those was investigated. Overall, 730 emotion events were detected during a total driving time of 440 minutes, and event triggers were assigned to 92% of the emotion events. A similar second on-road study was conducted in a partially controlled setting on a planned road circuit. In 840 minutes, 1947 emotion events were measured, and triggers were successfully assigned to 94% of those. The differences in natures, frequencies and causes of emotions on different road types were investigated. Comparison of emotion events for different roads demonstrated substantial variances of natures, frequencies and triggers of emotions on different road types. The results showed that emotions play a significant role during automobile driving. The possibility of assigning triggers can be used to create a better understanding of causes of emotions in the automotive habitat. Both on-road studies were compared through statistical analysis to investigate influences of the different study settings. Certain conditions (e.g. driving setting, social interaction) showed significant influence on emotions during driving. This research establishes and validates a methodology for the study of emotions and their causes in the driving environment through which systems and factors causing positive and negative emotional effects can be identified. The methodology and results can be applied to design and research processes, allowing the identification of issues and opportunities in current automotive design to address challenges of future automotive design. Suggested future research includes the investigation of a wider variety of road types and situations, testing with different automobiles and the combination of multiple measurement techniques.

A utilização de jogos sérios em muitas atividades, incluindo casos de saúde, como o processo de reabilitação motora, tem demonstrado resultados satisfatórios que encorajam o desenvolvimento de novas aplicações neste cenário. Jogos podem tornar tais atividades mais interessantes e divertidas para os pacientes, como também auxiliar as etapas do processo de reabilitação. Nestas aplicações, estratégias que visam a manutenção do nível de motivação do usuário durante a utilização são muito importantes. Assim, esta pesquisa investiga a adaptação de contexto em jogos sérios com a utilização de técnicas de Computação Afetiva. A proposta consiste em um framework que torna mais baixo ao programador o custo de implementação da adaptação afetiva em jogos e permite que o fisioterapeuta configure as adaptações que serão executadas no jogo conforme o perfil dos pacientes. Com o intuito de verificar a viabilidade da proposta, dois jogos para reabilitação motora e uma versão do framework foram implementados, permitindo a realização de experimentos com programadores, fisioterapeutas e pacientes. Os resultados obtidos permitem concluir que a abordagem proposta tende a proporcionar grande impacto social e tecnológico / The use of serious games in many activities, including health cases, like the motor rehabilitation process, has demonstrated results that encourage the development of new applications in this scenario. These activities can be more interesting and funnier by using games, as well as help the patients to execute the steps of the rehabilitation process. In these applications, strategies to maintain the user\'s motivation level during the game are very important. Thus, in this research, we investigated the context adaptation on serious games using techniques of Affective Computing. The proposal consists of a framework that makes the cost of implementing affective adaptation in games lower to programmers and allows the physiotherapists to configure the adaptations that will be executed in the game, according to the profile of the patients. In order to verify the feasibility of the proposal, two games for motor rehabilitation and a version of the framework were implemented, allowing the realization of experiments with programmers, physiotherapists, and patients. The results obtained allow us to conclude that the proposed approach tends to provide great social and technological impact

Uma das principais motivações da Inteligência Artificial no contexto dos sistemas de entretenimento digital é criar personagens adaptáveis a novas situações, pouco previsíveis, com aprendizado rápido, memória de situações passadas e uma grande diversidade de comportamentos consistente e convincente ao longo do tempo. De acordo com recentes estudos desenvolvidos nos campos da Neurociência e da Psicologia, a capacidade de resolução de problemas não está unicamente atrelada à facilidade na manipulação de símbolos, mas também à exploração das características do ambiente e à interação social, que pode ser expressa na forma de fenômenos emocionais. Os resultados desses estudos confirmam o papel fundamental que cumprem a personalidade e as emoções nas atividades de percepção, planejamento, raciocínio, criatividade, aprendizagem, memória e tomada de decisão. Quando módulos para a manipulação de personalidade e emoções são incorporados à teoria de agentes, é possível a construção de Agentes com Comportamento Convincente (Believable Agents). O objetivo principal deste trabalho é desenvolver e implementar uma arquitetura de agentes inteligentes para construir personagens sintéticos cujos estados afetivos influenciam em suas atividades cognitivas. Para o desenvolvimento de tal arquitetura utilizou-se o modelo BDI (Beliefs, Desires e Intentions) como base e aos módulos existentes em uma implementação desse modelo foi incluído um Módulo Afetivo. Esse Módulo Afetivo é constituído por três submódulos (Personalidade, Humor e Emoção) e deve impactar nas atividades cognitivas de percepção, memória e tomada de decisão do agente. Duas provas de conceito (experimentos) foram construídas : a simulação do problema do ``Dilema do Prisioneiro Iterado\'\' e a versão computadorizada do ``Jogo da Memória\'\'. A construção desses experimentos permitiu avaliar empiricamente a influência da personalidade, humor e emoção nas atividades cognitivas dos agentes, e consequentemente no seu comportamento. Os resultados evidenciam que a utilização da nova arquitetura permite a construção de agentes com comportamentos mais coerentes, adaptativos e cooperativos quando comparados aos de agentes construídos com arquiteturas cujas atividades cognitivas não consideram o estado afetivo, e também produz um comportamento mais próximo de um agente humano que de um comportamento ótimo ou aleatório. Essa evidência de sucesso, apresentada nos resultados, mostra que os agentes construídos com a arquitetura proposta nessa dissertação indicam um avanço na direção do desenvolvimento dos Agentes com Comportamento Convincente. / One of the main motivations of Artificial Intelligence in the context of the digital entertainment systems is to create characters that are adaptable to new situations, unpredictable, fast learners, enable with memory of past situations and a variety of consistent and convincing behavior over time. According to recent studies conducted in the fields of Neuroscience and Psychology, the ability to solve problems is not only related to the capacity to manipulate symbols, but also to the ability to explore the environment and to engage into social interaction, which can be expressed as emotional phenomena. The results of these studies confirm the key role the personality and emotions play in the activities of perception, attention, planning, reasoning, creativity, learning, memory and decision making. When modules for handling personality and emotion, are incorporated in a theory of agents, it is possible to build Believable Agents. The main objective of this work is to develop and implement an intelligent agent architecture to build synthetic characters whose affective states influence their cognitive activities. To develop such architecture the BDI model (Beliefs, Desires and Intentions) was used as a basis, to which an Affective Module was included. The Affective Module consists of three sub-modules (Personality, Mood and Emotion), which influence the cognitive activities of perception, memory and decision making. Finally, two proofs of concept were built: the simulation of the problem of ``Iterated Prisoner\'s Dilemma\'\' and the computerized version of the ``Memory Game.\'\' The construction of these experiments allowed to evaluate empirically the influence of personality, mood and emotion in cognitive activities of agents and consequently in their behavior. The results show that using the proposed architecture one can build agents with more consistent, adaptive and cooperative behaviors when compared to agents built with architectures whose affective states do not influence their cognitive activities. It also produces a behavior that is closer to a human user than that of optimal or random behavior. This evidence of success, presented in the obtained results, show that agents built with the proposed architecture indicate an advance towards the development of Believable Agents.

Este trabalho apresenta o processo de construção de um modelo de inferência de emoções que um aluno sente em relação a outros alunos durante interação síncrona em um contexto de jogo colaborativo de aprendizagem. A inferência de emoções está psicologicamente fundamentada na abordagem da avaliação cognitiva e foram investigadas relações entre objetivos e normas comportamentais do aluno e aspectos de sua personalidade. Especificamente, foram empregados o modelo OCC de emoções e o modelo Big-Five (Cinco Grandes Fatores) de traços de personalidade para a fundamentação teórica da modelagem. O modelo afetivo representa a vergonha e orgulho apresentados pelo aluno em resposta à avaliação cognitiva de suas próprias ações e a reprovação e admiração apresentadas pelo aluno em resposta a ações de seu parceiro de jogo, a partir da avaliação do comportamento observável dos parceiros representado por suas interações no jogo colaborativo, em relação a normas comportamentais do aluno. A fim de suportar a incerteza presente na informação afetiva e cognitiva do aluno, adotou-se uma representação deste conhecimento através de Rede Bayesiana. Um refinamento qualitativo parcial e a respectiva parametrização quantitativa do modelo probabilístico foram efetuados a partir da análise de uma base de casos obtida através da condução de experimentos. A fim de prover um ambiente experimental, foi concebido e prototipado um jogo colaborativo no qual dois indivíduos conjugam esforços a fim de resolver problemas lógicos comuns à dupla, através de ações coordenadas, negociação simples e comunicação estruturada, em competição com outras duplas. / This work presents the construction of a model to infer emotions a student feels towards other students during synchronous interaction in the context of a collaborative learning game. The emotions inference is psychologically based on cognitive appraisal theory. Some relations between students’ personality and their goals and behavioral standards were also investigated. This modeling was based on OCC emotion model and Big-Five personality model. The affective model represents the student’s proud and shame as an answer to the cognitive appraisal of her/his own attributed interactions, and the student’s admiration and reproach as an answer to the cognitive appraisal of her/his partner attributed interactions, both according to the student’s behavioral standards. Bayesian Network knowledge representation was employed to better stand for the uncertainty present in the student’s cognitive and affective information. Employing a data-driven procedure, the probabilistic model was partially refined in terms of qualitative relations and quantitative parameters. Experimental data were obtained by using a game prototype implemented in order to support a collaborative dynamics of coordinated action, simple negotiation and structured communication, through which students interacted in order to solve shared problems, during synchronous competition with other students.

For several decades, pediatricians used to believe that neonates do not feel pain. The American Academy of Pediatrics (AAP) recognized neonates' sense of pain in 1987. Since then, there have been many studies reporting a strong association between repeated pain exposure (under-treatment) and alterations in brain structure and function. This association has led to the increased use of anesthetic medications. However, recent studies found that the excessive use of analgesic medications (over-treatment) can cause many side effects. The current standard for assessing neonatal pain is discontinuous and suffers from inter-observer variations, which can lead to over- or under-treatment. Therefore, it is critical to address the shortcomings of the current standard and develop continuous and less subjective pain assessment tools. This dissertation introduces an automatic and comprehensive neonatal pain assessment system. The presented system is different from the previous ones in three principal ways. First, it is specifically designed to assess pain of neonates using data captured while they are hospitalized in the Neonatal Intensive Care Units (NICU). Second, it dynamically analyzes neonatal pain as it unfolds in a particular pattern over time. Third, it combines visual, vocal, and physiological signals to create a system that continues to assess pain even when one or more signals become temporarily unavailable. The presented system has four main components. The first three components consist of novel algorithms for analyzing the visual, vocal, and physiological signals separately. The last component combines all the three signals to create a multimodal pain assessment system. The performance of the system in recognizing pain events is comparable to that of trained nurses; hence, it demonstrates the feasibility of automatic pain assessment in typical neonatal care environments.

Classification of emotions plays a very important role in affective computing and has real-world applications in fields as diverse as entertainment, medical, defense, retail, and education. These applications include video games, virtual reality, pain recognition, lie detection, classification of Autistic Spectrum Disorder (ASD), analysis of stress levels, and determining attention levels. This vast range of applications motivated us to study automatic emotion recognition which can be done by using facial expression, speech, and physiological data. A person’s physiological signals such are heart rate, and blood pressure are deeply linked with their emotional states and can be used to identify a variety of emotions; however, they are less frequently explored for emotion recognition compared to audiovisual signals such as facial expression and voice. In this thesis, we investigate a multimodal approach to emotion recognition using physiological signals by showing how these signals can be combined and used to accurately identify a wide range of emotions such as happiness, sadness, and pain. We use the deep convolutional neural network for our experiments. We also detail comparisons between gender-specific models of emotion. Our investigation makes use of deep convolutional neural networks, which are the latest state of the art in supervised learning, on two publicly available databases, namely DEAP and BP4D+. We achieved an average emotion recognition accuracy of 98.89\% on BP4D+ and on DEAP it is 86.09\% for valence, 90.61\% for arousal, 90.48\% for liking and 90.95\% for dominance. We also compare our results to the current state of the art, showing the superior performance of our method.