Les Agents Conversationnels Animés (ACA) sont des personnages virtuels interactifs et expressifs, dont l'aspect est très souvent « humain », exploitant différentes modalités telles que la face, le langage, les gestes, le regard ou encore la prosodie de la voix. Le but est qu'ils s'expriment en langage naturel et puissent dialoguer avec des interlocuteurs humains. Pour développer un ACA, il faut d'abord comprendre que des aspects tels que personnalité, les émotions et leur apparence sont extrêmement importants. Le travail qui est présenté dans cette thèse a pour objectif d'augmenter l'acceptabilité et la crédibilité des agents au moyen de la personnalité, considérée comme une notion centrale à l'interaction ACA-humain. On propose un modèle qui dote l'ACA de facettes de personnalité et de buts de communication « cachés » et qui module ainsi ses actions conversationnelles. Ce travail présente également une application de jeu de type “puzzle”, intégrant un ACA doté de facettes de personnalité et de buts « cachés », qui a servi de support à plusieurs expérimentations et à l'évaluation du modèle proposé. / Embodied Conversational Agents (ECAs) are intelligent software entities with an embodiment used to communicate with users, using natural language. Their purpose is to exhibit the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication. Researchers in the field of ECAs try to create agents that can be more natural, believable and easy to use. Designing an ECA requires understanding that manner, personality, emotion, and appearance are very important issues to be considered. In this thesis, we are interested in increasing believability of ECAs by placing personality at the heart of the human-agent verbal interaction. We propose a model relating personality facets and hidden communication goals that can influence ECA behaviors. Moreover, we apply our model in agents that interact in a puzzle game application. We develop five distinct personality oriented agents using an expressive communication language and a plan-based BDI approach for modeling and managing dialogue according to our proposed model. In summary, we present and test an innovative approach to model mental aspects of ECAs trying to increase their believability and to enhance human-agent affective communication. With this research, we improve the understanding on how ECAs with expressive and affective characteristics can establish and maintain long-term human-agent relationships.

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / The research in Pervasive Computing have directed their efforts to the modeling and development of programmable and interactive environments, able to assist users in their daily activities. The Systems Group of Mobile Computing (gMob) of Federal University of Santa Maria, developed the project ClinicSpace, a system to aid clinical tasks in hospitals based on Pervasive Computing technologies. On the other hand, the Affective Computing, which is a relatively new field of research in computer science, came up with the proposal to identify and synthesize human feelings in machinery, in order to make the interaction with computing devices more enjoyable and less frustrating. Given that pervasive systems work with context information to adapt their applications according to user needs, it becomes feasible through the Affective Computing, identify emotional characteristics of a person for this type of information can be used as context element, making the adjustment more precise applications pervasive. In this context, this research proposes a model of acquisition of affective context using techniques of inference of stress from psychological tools, emotional self-analysis and stressful classification of clinical tasks. Integrating the concepts of Affective Computing and Pervasive Computing, this model aims to classify the state of stress of the users of ClinicSpace and use it as an affective context, increasing the wealth of contextual information used in this system. To develop the proposed model, studies were performed on the Perceived Stress Scale (PSS), responsible for identifying stress in an individual, the use of AffectButtons to identify the user s emotional state, and the development of a questionnaire, applied to health care professionals to identify and classify the degree of stress of clinical activities. Based on these three approaches was possible to develop a classification model of stress for ClinicSpace users (doctors). To validate the proposal, was implemented a prototype of Affective Context Service for architecture ClinicSpace, which was tested in the Annas Dias Hospital (Ibirubá - RS). The results showed that the model proposed in this work is able to classify the state of stress for clinicians that this type of information can be used as an element of context in pervasive systems. / As pesquisas na área da Computação Pervasiva têm direcionado seus esforços para a modelagem e desenvolvimento de ambientes programáveis e interativos capazes de auxiliar os usuários em suas atividades rotineiras. O Grupo de Sistemas de Computação Móvel (GMob) da Universidade Federal de Santa Maria, desenvolve o projeto ClinicSpace, um sistema de auxílio à tarefas clínicas em ambientes hospitalares baseado em tecnologias da Computação Pervasiva. Por outro lado, a Computação Afetiva, que é um campo de pesquisa relativamente novo na informática, surgiu com a proposta de identificar e sintetizar sentimentos humanos em máquinas, com a finalidade de tornar a interação com dispositivos computacionais mais agradável e menos frustrante. Tendo em vista que sistemas pervasivos trabalham com informações de contexto para adaptar suas aplicações de acordo com as necessidades do usuário, torna-se viável através da Computação Afetiva, identificar características emocionais de uma pessoa para que esse tipo de informação possa ser usada como dados de contexto, tornando a adaptação das aplicações pervasivas mais precisas. Neste contexto, o presente trabalho propõe um modelo de aquisição de contexto afetivo utilizando técnicas de inferência de estresse a partir de ferramentas psicológicas, autoanálise emocional e classificação estressora de tarefas clínicas. Integrando os conceitos de Computação Afetiva e Computação Pervasiva, este modelo visa classificar o estado de estresse dos usuários do ClinicSpace e utilizá-lo como elemento de contexto afetivo, aumentando a riqueza de informações de contexto utilizadas neste sistema. Para o desenvolvimento do modelo proposto, foram realizados estudos sobre a Escala de Estresse Percebido (PSS Perceived Stress Scale), responsável pela identificação de estresse em um indivíduo, o uso de AffectButtons para identificar o estado emocional do usuário, e o desenvolvimento de um questionário, aplicado em profissionais da área da saúde, para identificar e classificar o grau de estresse de atividades clínicas. Com base nessas três abordagens foi possível elaborar um modelo de classificação de estresse para os usuários do ClinicSpace (médicos). Para validar a proposta, foi implementado um protótipo de Serviço de Contexto Afetivo para a arquitetura ClinicSpace, que foi testado no Hospital Annas Dias (Ibirubá RS). Os resultados mostraram que o modelo proposto neste trabalho é capaz de classificar o estado de estresse de profissionais clínicos para que este tipo de informação possa ser utilizado como elemento de contexto em sistemas pervasivos.

Este trabalho propõe um agente pedagógico animado que possui o objetivo de fornecer suporte emocional ao aluno: motivando-o e encorajando-o, fazendo-o acreditar em suas próprias habilidades e promovendo um estado de espírito positivo no aluno, que é melhor para o seu aprendizado. Este suporte cuidadoso do agente, suas táticas afetivas, é expresso através de comportamentos emotivos e mensagens de encorajamento do personagem animado. Devido à tendência social humana de antropomorfizar software, nós acreditamos que um agente de software pode realizar esse papel afetivo. Para escolher as táticas afetivas adequadas, o agente deve conhecer as emoções do aluno. O agente proposto infere as seguintes emoções do aluno: alegria/tristeza, satisfação/frustração, raiva/gratidão e vergonha a partir do comportamento observável do aluno, isto é, as ações do aluno na interface do sistema educacional. A inferência das emoções é fundamentada psicologicamente na teoria cognitiva das emoções. Mais especificamente, nós usamos o modelo OCC o qual é baseado na abordagem cognitivista das emoções e é possível de ser implementado computacionalmente. Devido a natureza dinâmica da informação sobre o estado afetivo do aluno, nós adotamos uma abordagem BDI para implementar o modelo afetivo do usuário e o diagnóstico afetivo. Além disso, em nosso trabalho nós nos beneficiamos da capacidade de raciocínio do BDI para o agente deduzir o appraisal do aluno, que lhe permite inferir as emoções do aluno. Como um caso de estudo, o agente proposto é implementado como o Agente Mediador de MACES: um ambiente para ensino colaborativo à distância modelado com uma arquitetura multiagente e baseado psicologicamente na abordagem Sociocultural de Vygotsky. / This work proposes an animated pedagogical agent that has the role of providing emotional support to the student: motivating and encouraging him, making him believe in his self-ability, and promoting a positive mood in him, which fosters learning. This careful support of the agent, its affective tactics, is expressed through emotional behaviour and encouragement messages of the lifelike character. Due to human social tendency of anthropomorphising software, we believe that a software agent can accomplish this affective role. In order to choose the adequate affective tactics, the agent should also know the student’s emotions. The proposed agent recognises the student’s emotions: joy/distress, satisfaction/disappointment, anger/gratitude, and shame, from the student’s observable behaviour, i. e. his actions in the interface of the educational system. The inference of emotions is psychologically grounded on the cognitive theory of emotions. More specifically, we use the OCC model which is based on the cognitive approach of emotion and can be computationally implemented. Due to the dynamic nature of the student’s affective information, we adopted a BDI approach to implement the affective user model and the affective diagnosis. Besides, in our work we profit from the reasoning capacity of the BDI approach in order for the agent to deduce the student’s appraisal, which allows it to infer the student’s emotions. As a case study, the proposed agent is implemented as the Mediating Agent of MACES: an educational collaborative environment modelled as a multi-agent system and pedagogically based on the sociocultural theory of Vygotsky.

This thesis describes that it is possible to drive synthetic emotions of an interface agent with an electronic nose system developed at AASS. The e-Nose can be used for quality control, and the detected distortion from a known smell sensation prototype is interpreted to a 3D-representation of emotional states, which in turn points to a set of pre-defined muscle contractions. This extension of a rule based motivation system, which we call Facial Expression Driver, is incorporated to a model for sensor fusion with active perception, to provide a general design for a more complex system with additional senses. To be consistent with the biologically inspired sensor fusion model a muscle based animated facial model was chosen as a test bed for the expression of current emotion. The social agent’s facial expressions demonstrate its tolerance to the detected distortion in order to manipulate the user to restore the system to functional balance. Only a few of the known projects use chemically based sensing to drive a face in real-time, whether they are virtual characters or animatronics. This work may inspire a future android implementation of a head with electro active polymers as synthetic facial muscles.

This is a study of how emotional engagement can be measured and be taken into account in the development of websites. We believe that emotions become relevant only when viewed in correlation to how users experiencing a webpage. In this study, we developed a tool for use together with user tests where the information about the user's emotional engagement can add new value to the evaluation. The tool allows the user to mark, on the website, his/her emotions and level of engagement represented by colored circles, generating quantiative data on how users feel about the website. The tool is part of a method for evaluating the emotional engagement that we have developed which consists of user testing supported by the tool and accompanied by interviews. / Detta är en studie och metodutveckling i hur emotionellt engagemang kan mätas och således tas till vara på i utvecklingen av webbplatser. Vi anser att känslor blir väsentliga först när de sätts i korrelation till hur användare i övrigt upplever en webbsida. I denna studie utvecklades ett verktyg avsett att användas i samband med användartester där information om användarens känslor och engagemang kan tillföra ett mervärde till användartestet. Verktyget låter användaren sätta ut markörer för känslor och engagemang representerade av färgcirklar på webbsidan vilket genererar kvantitativ data för hur användare känner inför webbplatsen och interaktionen med den. Verktyget är en del av den metod för utvärdering av emotionellt engagemang som vi utvecklat vilket består av användartester stödda av verktyget vilket följs upp av intervjuer.

Recent research has indicated that the pupil diameter (PD) in humans varies with their affective states. However, this signal has not been fully investigated for affective sensing purposes in human-computer interaction systems. This may be due to the dominant separate effect of the pupillary light reflex (PLR), which shrinks the pupil when light intensity increases. In this dissertation, an adaptive interference canceller (AIC) system using the H∞ time-varying (HITV) adaptive algorithm was developed to minimize the impact of the PLR on the measured pupil diameter signal. The modified pupil diameter (MPD) signal, obtained from the AIC was expected to reflect primarily the pupillary affective responses (PAR) of the subject. Additional manipulations of the AIC output resulted in a processed MPD (PMPD) signal, from which a classification feature, PMPDmean, was extracted. This feature was used to train and test a support vector machine (SVM), for the identification of stress states in the subject from whom the pupil diameter signal was recorded, achieving an accuracy rate of 77.78%. The advantages of affective recognition through the PD signal were verified by comparatively investigating the classification of stress and relaxation states through features derived from the simultaneously recorded galvanic skin response (GSR) and blood volume pulse (BVP) signals, with and without the PD feature. The discriminating potential of each individual feature extracted from GSR, BVP and PD was studied by analysis of its receiver operating characteristic (ROC) curve. The ROC curve found for the PMPDmean feature encompassed the largest area (0.8546) of all the single-feature ROCs investigated. The encouraging results seen in affective sensing based on pupil diameter monitoring were obtained in spite of intermittent illumination increases purposely introduced during the experiments. Therefore, these results confirmed the benefits of using the AIC implementation with the HITV adaptive algorithm to isolate the PAR and the potential of using PD monitoring to sense the evolving affective states of a computer user.

O estado emocional dos usuários influencia na tomada de decisão e é essencial para o conhecimento e explicação do comportamento dos usuários com aplicações computacionais. Sistemas computacionais fazem parte da vida cotidiana, influenciando o comportamento humano e estimulando mudanças nos estados emocionais. A avaliação das emoções dos usuários, durante a interação com aplicações computacionais, pode ajudar a fornecer interfaces flexíveis e melhores sistemas de recomendação. No entanto, as emoções são complexas e difíceis de identificar ou avaliar. Pesquisas anteriores demonstraram que o uso de sensores individuais, em um cenário do mundo real, não fornece uma avaliação emocional precisa. Acredita-se que somente uma visão holística pode permitir tirar conclusões significativas sobre o estado emocional dos usuários. Assim, nesta Tese, propõe-se uma abordagem chamada UserSense, que considera vários sensores para estimar os estados emocionais dos usuários em tempo de interação. A abordagem proposta, a partir de múltiplos sensores, considera várias entradas de usuários (como a fala, movimentos faciais, frequência cardíaca e atividades) e utiliza métodos de inteligência artificial para mapear essas diferentes respostas em um ou mais estados emocionais. Para a validação, os dados coletados pelos sensores durante a interação do usuário foram confrontados com os dos especialistas (psicólogos). A Teoria Componencial das Emoções e o Espaço Emocional Semântico de Scherer são utilizados na fundamentação teórica da abordagem UserSense que, baseada no Middleware Adaptativo OpenCom, incorpora funcionalidades para carregar e descarregar novos recursos, necessários a infraestrutura proposta. Os resultados experimentais mostram que a combinação de resultados gerados por vários sensores, fornece uma avaliação mais precisa dos estados emocionais do que considerar sensores individualmente. / Users emotional state influence decision making and is essential for the knowledge and explanation of users behavior with computer applications. Computer systems are part of everyday life, influencing human behavior and stimulating changes in users emotional states. The assessment of users emotions during interaction with computer applications can help to provide tailorable interfaces and better recommendations systems. However, emotions are complex and difficult to identify or assess. Previous studies have shown that the use of single sensors, in a real-world scenario, does not provide accurate emotional assessment. We believe that only this holistic view can allow us to draw significant conclusions about the users emotional states. Hence, in this Thesis we propose a approach called UserSense, that takes into account multiple sensors to estimate the users emotional states at interaction time. The proposed multi-sensing approach considers several inputs from users (such as speech, facial movements, heart rate and activities), and uses artificial intelligent methods to map these different responses into one or more emotional states; for validation, the data collected by the sensors during the user interaction were confronted with specialists (psychologists). Componential Emotion Theory and Scherers Emotional Semantic Space are used in the theoretical foundation of the UserSense approach, that takes into account Adaptive Middleware OpenCom to embed functionalities to load and unload new required features to proposed infrastructure. The experimental results show that the combination of outputs generated by multiple sensors provides a more accurate assessment of emotional states than considering sensors individually.

L'essor de l'informatique affective ouvre aujourd'hui la porte à la création de dispositifs artificiels dotés d'une forme d'intelligence sociale et émotionnelle. L’étude de l'interaction homme-machine dans ce contexte offre de nombreuses pistes de recherche. Parmi celles-ci se trouve la question de la personnalité : comment il est possible de modéliser certaines caractéristiques d’une personnalité artificielle ? Comment ces caractéristiques influencent le déroulement de l'interaction entre l'homme et la machine ? Cette question globale soulève plusieurs questions de recherche : Comment définir la personnalité ? Sur quels modèles et théories issus de la psychologie peut-on s'appuyer pour développer une personnalité artificielle ? Quelle méthodologie adopter pour aborder l’implémentation d'un concept psychologique complexe ? Qu'apporte le développement informatique d'une personnalité à l'interaction homme-machine ? Au domaine de la psychologie de la personnalité ? Comment évaluer expérimentalement ces apports ? Pour aborder ces questions, nos travaux se positionnent dans une optique pluridisciplinaire, au croisement de l’informatique et de la psychologie. Au regard de sa pertinence pour une approche computationnelle, nous avons modélisé la régulation du Soi comme une composante de la personnalité. Ce concept est approché à partir de la théorie du regulatory focus. Sur cette base théorique, un cadre de travail conceptuel et un modèle computationnel sont proposés. Un questionnaire mesurant le regulatory focus a également été développé et validé. Ces propositions théoriques sont mises en œuvre dans deux implémentations data-driven (dimensionnelle vs socio-cognitive) dotant des agents de regulatory focus en utilisant de l’apprentissage automatique. Deux études utilisateurs (interaction unique avec agent artificiel vs sessions répétées avec agent animé), présentées dans un cadre ludique, ont permis d’étudier la perception du regulatory focus chez un agent et son impact sur l'interaction. Nos résultats encouragent l’utilisation du regulatory focus en informatique affective et ouvrent des perspectives sur les liens théoriques et méthodologiques entre informatique et psychologie. / The development of affective computing is leading to the design of artificial devices endowed with a form of social and emotional intelligence. The study of human-computer interaction in this context offers many research tracks. Among them is the question of personality: how to model some characteristics of an artificial personality? How these characteristics will influence the course of interaction with users? This goal rises several research questions: how to define personality? On which models and theories from psychology should we rely to define some artificial personality? Which methodology will help to address the implementation of such a complex psychological concept? What could artificial personality bring to the field of human-computer interaction? And to the psychology of personality? How to experimentally evaluate these contributions? To address these issues, this thesis takes a multidisciplinary approach, at the crossing of computing science and psychology. Given its relevance to a computational approach, we modeled self-regulation as a component of personality. This concept is approached from the regulatory focus theory. On this theoretical basis, a conceptual framework and a computational model are proposed. Our theoretical proposals led to two data-driven implementations (dimensional vs. socio-cognitive) which endowed our artificial agents with regulatory focus by using machine-learning. A French questionnaire measuring regulatory focus was designed and validated. Two user studies (brief interaction with artificial agents vs. repeated sessions with animated agents), where the regulatory focus of agents is conveyed via game strategies, enabled the study of regulatory focus perception and its impact on the interaction. Our results support the use of regulatory focus in affective computing and open perspectives on the theoretical and methodological links between computer science and psychology.

This thesis questions the idea that; quantified self technology can make us understand our bodies better and facilitate a healthy relationship with the body. Instead, it proposes that a healthy relationship to the body is developed through technology that facilitates a somatic practice, involving the bodily experience in the interpretations of the data. This is achieved by bringing in values inspired by the practice of mindfulness as an alternative to the existing design values promoted in technological solutions dealing with the body. I have designed a first prototype, Inner Mirror, which explores adolescents’ body relationship through screen-based visuals. Inner Mirror detects adolescents’ arousals to visualize their emotional changes in abstract representations that they are invited to connect to specific emotional experiences. The process of designing Inner Mirror will be described thoroughly in the paper. The process was a continuous negotiation between the ideas and values that I brought into the project and the adolescents’ worldview. This is described through a first-person perspective and a participatory design approach. Two school classes of 43 adolescents (between 13 to 14 years old) have participated in the project through three workshops. In the end, I test the prototype in two different settings: on myself, adopting the first-person perspective and together with the adolescents. Finally, three concepts that emerged in the design work are evaluated to suggest a direction for future work.

Tesis por compendio / [ES] El uso de la realidad virtual (RV) se ha incrementado notablemente en la comunidad científica para la investigación del comportamiento humano. En particular, la RV inmersiva ha crecido debido a la democratización de las gafas de realidad virtual o head mounted displays (HMD), que ofrecen un alto rendimiento con una inversión económica. Uno de los campos que ha emergido con fuerza en la última década es el Affective Computing, que combina psicofisiología, informática, ingeniería biomédica e inteligencia artificial, desarrollando sistemas que puedan reconocer emociones automáticamente. Su progreso es especialmente importante en el campo de la investigación del comportamiento humano, debido al papel fundamental que las emociones juegan en muchos procesos psicológicos como la percepción, la toma de decisiones, la creatividad, la memoria y la interacción social. Muchos estudios se han centrado en intentar obtener una metodología fiable para evocar y automáticamente identificar estados emocionales, usando medidas fisiológicas objetivas y métodos de aprendizaje automático. Sin embargo, la mayoría de los estudios previos utilizan imágenes, audios o vídeos para generar los estados emocionales y, hasta donde llega nuestro conocimiento, ninguno de ellos ha desarrollado un sistema de reconocimiento emocional usando RV inmersiva. Aunque algunos trabajos anteriores sí analizan las respuestas fisiológicas en RV inmersivas, estos no presentan modelos de aprendizaje automático para procesamiento y clasificación automática de bioseñales. Además, un concepto crucial cuando se usa la RV en investigación del comportamiento humano es la validez: la capacidad de evocar respuestas similares en un entorno virtual a las evocadas por el espacio físico. Aunque algunos estudios previos han usado dimensiones psicológicas y cognitivas para comparar respuestas entre entornos reales y virtuales, las investigaciones que analizan respuestas fisiológicas o comportamentales están mucho menos extendidas. Según nuestros conocimientos, este es el primer trabajo que compara entornos físicos con su réplica en RV, empleando respuestas fisiológicas y algoritmos de aprendizaje automático y analizando la capacidad de la RV de transferir y extrapolar las conclusiones obtenidas al entorno real que se está simulando. El objetivo principal de la tesis es validar el uso de la RV inmersiva como una herramienta de estimulación emocional usando respuestas psicofisiológicas y comportamentales en combinación con algoritmos de aprendizaje automático, así como realizar una comparación directa entre un entorno real y virtual. Para ello, se ha desarrollado un protocolo experimental que incluye entornos emocionales 360º, un museo real y una virtualización 3D altamente realista del mismo museo. La tesis presenta novedosas contribuciones del uso de la RV inmersiva en la investigación del comportamiento humano, en particular en lo relativo al estudio de las emociones. Esta ayudará a aplicar metodologías a estímulos más realistas para evaluar entornos y situaciones de la vida diaria, superando las actuales limitaciones de la estimulación emocional que clásicamente ha incluido imágenes, audios o vídeos. Además, en ella se analiza la validez de la RV realizando una comparación directa usando una simulación altamente realista. Creemos que la RV inmersiva va a revolucionar los métodos de estimulación emocional en entornos de laboratorio. Además, su sinergia junto a las medidas fisiológicas y las técnicas de aprendizaje automático, impactarán transversalmente en muchas áreas de investigación como la arquitectura, la salud, la evaluación psicológica, el entrenamiento, la educación, la conducción o el marketing, abriendo un nuevo horizonte de oportunidades para la comunidad científica. La presente tesis espera contribuir a caminar en esa senda. / [EN] In recent years the scientific community has significantly increased its use of virtual reality (VR) technologies in human behaviour research. In particular, the use of immersive VR has grown due to the introduction of affordable, high performance head mounted displays (HMDs). Among the fields that has strongly emerged in the last decade is affective computing, which combines psychophysiology, computer science, biomedical engineering and artificial intelligence in the development of systems that can automatically recognize emotions. The progress of affective computing is especially important in human behaviour research due to the central role that emotions play in many background processes, such as perception, decision-making, creativity, memory and social interaction. Several studies have tried to develop a reliable methodology to evoke and automatically identify emotional states using objective physiological measures and machine learning methods. However, the majority of previous studies used images, audio or video to elicit emotional statements; to the best of our knowledge, no previous research has developed an emotion recognition system using immersive VR. Although some previous studies analysed physiological responses in immersive VR, they did not use machine learning techniques for biosignal processing and classification. Moreover, a crucial concept when using VR for human behaviour research is validity: the capacity to evoke a response from the user in a simulated environment similar to the response that might be evoked in a physical environment. Although some previous studies have used psychological and cognitive dimensions to compare responses in real and virtual environments, few have extended this research to analyse physiological or behavioural responses. Moreover, to our knowledge, this is the first study to compare VR scenarios with their real-world equivalents using physiological measures coupled with machine learning algorithms, and to analyse the ability of VR to transfer and extrapolate insights obtained from VR environments to real environments. The main objective of this thesis is, using psycho-physiological and behavioural responses in combination with machine learning methods, and by performing a direct comparison between a real and virtual environment, to validate immersive VR as an emotion elicitation tool. To do so we develop an experimental protocol involving emotional 360º environments, an art exhibition in a real museum, and a highly-realistic 3D virtualization of the same art exhibition. This thesis provides novel contributions to the use of immersive VR in human behaviour research, particularly in relation to emotions. VR can help in the application of methodologies designed to present more realistic stimuli in the assessment of daily-life environments and situations, thus overcoming the current limitations of affective elicitation, which classically uses images, audio and video. Moreover, it analyses the validity of VR by performing a direct comparison using highly-realistic simulation. We believe that immersive VR will revolutionize laboratory-based emotion elicitation methods. Moreover, its synergy with physiological measurement and machine learning techniques will impact transversely in many other research areas, such as architecture, health, assessment, training, education, driving and marketing, and thus open new opportunities for the scientific community. The present dissertation aims to contribute to this progress. / [CA] L'ús de la realitat virtual (RV) s'ha incrementat notablement en la comunitat científica per a la recerca del comportament humà. En particular, la RV immersiva ha crescut a causa de la democratització de les ulleres de realitat virtual o head mounted displays (HMD), que ofereixen un alt rendiment amb una reduïda inversió econòmica. Un dels camps que ha emergit amb força en l'última dècada és el Affective Computing, que combina psicofisiologia, informàtica, enginyeria biomèdica i intel·ligència artificial, desenvolupant sistemes que puguen reconéixer emocions automàticament. El seu progrés és especialment important en el camp de la recerca del comportament humà, a causa del paper fonamental que les emocions juguen en molts processos psicològics com la percepció, la presa de decisions, la creativitat, la memòria i la interacció social. Molts estudis s'han centrat en intentar obtenir una metodologia fiable per a evocar i automàticament identificar estats emocionals, utilitzant mesures fisiològiques objectives i mètodes d'aprenentatge automàtic. No obstant això, la major part dels estudis previs utilitzen imatges, àudios o vídeos per a generar els estats emocionals i, fins on arriba el nostre coneixement, cap d'ells ha desenvolupat un sistema de reconeixement emocional mitjançant l'ús de la RV immersiva. Encara que alguns treballs anteriors sí que analitzen les respostes fisiològiques en RV immersives, aquests no presenten models d'aprenentatge automàtic per a processament i classificació automàtica de biosenyals. A més, un concepte crucial quan s'utilitza la RV en la recerca del comportament humà és la validesa: la capacitat d'evocar respostes similars en un entorn virtual a les evocades per l'espai físic. Encara que alguns estudis previs han utilitzat dimensions psicològiques i cognitives per a comparar respostes entre entorns reals i virtuals, les recerques que analitzen respostes fisiològiques o comportamentals estan molt menys esteses. Segons els nostres coneixements, aquest és el primer treball que compara entorns físics amb la seua rèplica en RV, emprant respostes fisiològiques i algorismes d'aprenentatge automàtic i analitzant la capacitat de la RV de transferir i extrapolar les conclusions obtingudes a l'entorn real que s'està simulant. L'objectiu principal de la tesi és validar l'ús de la RV immersiva com una eina d'estimulació emocional usant respostes psicofisiològiques i comportamentals en combinació amb algorismes d'aprenentatge automàtic, així com realitzar una comparació directa entre un entorn real i virtual. Per a això, s'ha desenvolupat un protocol experimental que inclou entorns emocionals 360º, un museu real i una virtualització 3D altament realista del mateix museu. La tesi presenta noves contribucions de l'ús de la RV immersiva en la recerca del comportament humà, en particular quant a l'estudi de les emocions. Aquesta ajudarà a aplicar metodologies a estímuls més realistes per a avaluar entorns i situacions de la vida diària, superant les actuals limitacions de l'estimulació emocional que clàssicament ha inclòs imatges, àudios o vídeos. A més, en ella s'analitza la validesa de la RV realitzant una comparació directa usant una simulació altament realista. Creiem que la RV immersiva revolucionarà els mètodes d'estimulació emocional en entorns de laboratori. A més, la seua sinergia al costat de les mesures fisiològiques i les tècniques d'aprenentatge automàtic, impactaran transversalment en moltes àrees de recerca com l'arquitectura, la salut, l'avaluació psicològica, l'entrenament, l'educació, la conducció o el màrqueting, obrint un nou horitzó d'oportunitats per a la comunitat científica. La present tesi espera contribuir a caminar en aquesta senda. / Marín Morales, J. (2020). Modelling human emotions using immersive virtual reality, physiological signals and behavioural responses [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/148717 / TESIS / Compendio