Video games have become the largest portion of the entertainment industry and everyday life of millions of players around the world. Considering games as cultural artifacts, it seems imperative to study both games and players to understand underlying psychological and behavioral implications of interacting with this medium, especially since video games are rich domains for occurrence of rich affective experiences annotated by and measurable via in-game behavior. This thesis is a presentation of a series of studies that attempt to model player perception and behavior as well as their psychosocial attributes in order to make sense of interrelations of these factors and implications the findings have for game designers and researchers. In separate studies including survey and in-game telemetry data of millions of players, we delve into reliable measures of player psychological need satisfaction, motivation and generational cohort and cross reference them with in-game behavioral patterns by presenting systemic frameworks for classification and regression. We introduce a measurement of perceived need satisfaction and discuss generational effects in playtime and motivation, present a robust prediction model for ordinally processed motivations and review classification techniques when it comes to playstyles derived from player choices. Additionally, social aspects of play, such as social influence and contagion as well as disruptive behavior, is discussed along with advanced statistical models to detect and explain them. / <p>Note: The papers are not included in the fulltext online</p><p>Vid tidpunkten för disputationen var följande delarbete opublicerat: delarbete I (manuskript).</p><p>At the time of the doctoral defence the following paper was unpublished: paper I (manuscript).</p>

在心理與認知科學領域中常使用生理訊號來測量受試者的反應，並反映出人們的心理狀態起伏。本研究探討應用生理訊號識別情緒之可能性，以及將生理訊號與其他情緒辨識結果整合之方法。 在過去的研究中，生理與心理的對應關係，並無太多著墨，可稱為一黑盒子(black-box)的方式。並因上述類別式實驗長時間收集的生理訊號，對於誘發特定情緒反應之因果(cause-effect)並未進行深入的討論。本研究由於實驗的設計與選用材料之故，可一探純粹由刺激引發的情緒下情緒在生理與心理之因果關係，在輸入輸出對應間能有較明確的解釋。 本研究中嘗試監測較短時間(<10sec)的生理資訊，期望以一近乎即時的方式判讀並回饋使用者適當的資訊，對於生理訊號與情緒狀態的關聯性研究，將以IAPS(International Affective Picture System) 素材為來源，進行較過去嚴謹的實驗設計與程序，以探究生理訊號特徵如何應用於情緒分類。 雖然本研究以維度式情緒學說為理論基礎，然考慮到實際應用情境，若有其他以類別式的理論為基礎之系統，如何整合維度式與類別式兩類的資訊，提出可行的轉換方式，亦是本研究的主要課題。 / Physiological signals can be used to measure a subject’s response to a particular stimulus, and infer the emotional status accordingly. This research investigates the feasibility of emotion recognition using physiological measurements in a smart living space. It also addresses important issues regarding the integration of classification results from multiple modalities. Most past research regarded the recognition of emotion from physiological data as a mapping mechanism which can be learned from training data. These data were collected over a long period of time, and can not model the immediate cause-effect relationship effectively. Our research employs a more rigorous experiment design to study the relationship between a specific physiological signal and the emotion status. The newly designed procedure will enable us to identify and validate the discriminating power of each type of physiological signal in recognizing emotion. Our research monitors short term (< 10s) physiological signals. We use the IAPS (International Affective Picture System) as our experiment material. Physiological data were collected during the presentation of various genres of pictures. With such controlled experiments, we expect the cause-effect relation to be better explained than previous black-box approaches. Our research employs dimensional approach for emotion modeling. However, emotion recognition based on audio and/or visual clues mostly adopt categorical method (or basic emotion types). It becomes necessary to integrate results from these different modalities. Toward this end, we have also developed a mapping process to convert the result encoded in dimensional format into categorical data.

The rise of internet connected devices has enabled the home with a vast amount of enhancements to make life more convenient. These internet connected devices can be used to form a community of devices known as the internet of things (IoT). There is great value in IoT devices to promote healthy independent living for older adults. Fall-related injuries has been one of the leading causes of death in older adults. For example, every year more than a third of people over 65 in the U.S. experience a fall, of which up to 30 percent result in moderate to severe injury. Therefore, this thesis proposes an IoT-based fall detection system for smart home environments that not only to send out alerts, but also launches interaction models, such as voice assistance and camera monitoring. Such connectivity could allow older adults to interact with the system without concern of a learning curve. The proposed IoT-based fall detection system will enable family and caregivers to be immediately notified of the event and remotely monitor the individual. Integrated within a smart home environment, the proposed IoT-based fall detection system can improve the quality of life among older adults. Along with the physical concerns of health, psychological stress is also a great concern among older adults. Stress has been linked to emotional and physical conditions such as depression, anxiety, heart attacks, stroke, etc. Increased susceptibility to stress may accelerate cognitive decline resulting in conversion of cognitively normal older adults to MCI (Mild Cognitive Impairment), and MCI to dementia. Thus, if stress can be measured, there can be countermeasures put in place to reduce stress and its negative effects on the psychological and physical health of older adults. This thesis presents a framework that can be used to collect and pre-process physiological data for the purpose of validating galvanic skin response (GSR), heart rate (HR), and emotional valence (EV) measurements against the cortisol and self-reporting benchmarks for stress detection. The results of this framework can be used for feature extraction to feed into a regression model for validating each combination of physiological measurement. Also, the potential of this framework to automate stress protocols like the Trier Social Stress Test (TSST) could pave the way for an IoT-based platform for automated stress detection and management.

O estado emocional dos usuários influencia na tomada de decisão e é essencial para o conhecimento e explicação do comportamento dos usuários com aplicações computacionais. Sistemas computacionais fazem parte da vida cotidiana, influenciando o comportamento humano e estimulando mudanças nos estados emocionais. A avaliação das emoções dos usuários, durante a interação com aplicações computacionais, pode ajudar a fornecer interfaces flexíveis e melhores sistemas de recomendação. No entanto, as emoções são complexas e difíceis de identificar ou avaliar. Pesquisas anteriores demonstraram que o uso de sensores individuais, em um cenário do mundo real, não fornece uma avaliação emocional precisa. Acredita-se que somente uma visão holística pode permitir tirar conclusões significativas sobre o estado emocional dos usuários. Assim, nesta Tese, propõe-se uma abordagem chamada UserSense, que considera vários sensores para estimar os estados emocionais dos usuários em tempo de interação. A abordagem proposta, a partir de múltiplos sensores, considera várias entradas de usuários (como a fala, movimentos faciais, frequência cardíaca e atividades) e utiliza métodos de inteligência artificial para mapear essas diferentes respostas em um ou mais estados emocionais. Para a validação, os dados coletados pelos sensores durante a interação do usuário foram confrontados com os dos especialistas (psicólogos). A Teoria Componencial das Emoções e o Espaço Emocional Semântico de Scherer são utilizados na fundamentação teórica da abordagem UserSense que, baseada no Middleware Adaptativo OpenCom, incorpora funcionalidades para carregar e descarregar novos recursos, necessários a infraestrutura proposta. Os resultados experimentais mostram que a combinação de resultados gerados por vários sensores, fornece uma avaliação mais precisa dos estados emocionais do que considerar sensores individualmente. / Users emotional state influence decision making and is essential for the knowledge and explanation of users behavior with computer applications. Computer systems are part of everyday life, influencing human behavior and stimulating changes in users emotional states. The assessment of users emotions during interaction with computer applications can help to provide tailorable interfaces and better recommendations systems. However, emotions are complex and difficult to identify or assess. Previous studies have shown that the use of single sensors, in a real-world scenario, does not provide accurate emotional assessment. We believe that only this holistic view can allow us to draw significant conclusions about the users emotional states. Hence, in this Thesis we propose a approach called UserSense, that takes into account multiple sensors to estimate the users emotional states at interaction time. The proposed multi-sensing approach considers several inputs from users (such as speech, facial movements, heart rate and activities), and uses artificial intelligent methods to map these different responses into one or more emotional states; for validation, the data collected by the sensors during the user interaction were confronted with specialists (psychologists). Componential Emotion Theory and Scherers Emotional Semantic Space are used in the theoretical foundation of the UserSense approach, that takes into account Adaptive Middleware OpenCom to embed functionalities to load and unload new required features to proposed infrastructure. The experimental results show that the combination of outputs generated by multiple sensors provides a more accurate assessment of emotional states than considering sensors individually.

L'informatique Affective regarde la computation que se rapporte, surgit de, ou influence délibérément les émotions et trouve son domaine d'application naturel dans les interactions homme-machine a haut niveau d'abstraction. L'informatique affective peut être divisée en trois sujets principaux, à savoir: l'affichage,l'identification, et la synthèse. La construction d'une machine intelligente capable dinteragir'de façon naturelle avec son utilisateur passe forcement par ce trois phases. Dans cette thèse nous proposions une architecture basée principalement sur le modèle dite "Multimodal Affective User Interface" de Lisetti et la théorie psychologique des émotions nommé "Component Process Theory" de Scherer. Dans nos travaux nous avons donc recherché des techniques pour l'extraction automatique et en temps-réel des émotions par moyen des expressions faciales et de la prosodie vocale. Nous avons aussi traité les problématiques inhérentes la génération d'expressions sur de différentes plateformes, soit elles des agents virtuel ou robotique. Finalement, nous avons proposé et développé une architecture pour des agents intelligents capable de simuler le processus humaine d'évaluation des émotions comme décrit par Scherer.

<p>This thesis describes that it is possible to drive synthetic emotions of an interface agent with an electronic nose system developed at AASS. The e-Nose can be used for quality control, and the detected distortion from a known smell sensation prototype is interpreted to a 3D-representation of emotional states, which in turn points to a set of pre-defined muscle contractions. This extension of a rule based motivation system, which we call Facial Expression Driver, is incorporated to a model for sensor fusion with active perception, to provide a general design for a more complex system with additional senses. To be consistent with the biologically inspired sensor fusion model a muscle based animated facial model was chosen as a test bed for the expression of current emotion. The social agent’s facial expressions demonstrate its tolerance to the detected distortion in order to manipulate the user to restore the system to functional balance. Only a few of the known projects use chemically based sensing to drive a face in real-time, whether they are virtual characters or animatronics. This work may inspire a future android implementation of a head with electro active polymers as synthetic facial muscles.</p>

<p>This is a study of how emotional engagement can be measured and be taken into account in the development of websites. We believe that emotions become relevant only when viewed in correlation to how users experiencing a webpage. In this study, we developed a tool for use together with user tests where the information about the user's emotional engagement can add new value to the evaluation. The tool allows the user to mark, on the website, his/her emotions and level of engagement represented by colored circles, generating quantiative data on how users <em>feel </em>about the website. The tool is part of a method for evaluating the emotional engagement that we have developed which consists of user testing supported by the tool and accompanied by interviews.</p> / <p>Detta är en studie och metodutveckling i hur emotionellt engagemang kan mätas och således tas till vara på i utvecklingen av webbplatser. Vi anser att känslor blir väsentliga först när de sätts i korrelation till hur användare i övrigt upplever en webbsida. I denna studie utvecklades ett verktyg avsett att användas i samband med användartester där information om användarens känslor och engagemang kan tillföra ett mervärde till användartestet. Verktyget låter användaren sätta ut markörer för känslor och engagemang representerade av färgcirklar på webbsidan vilket genererar kvantitativ data för hur användare <em>känner inför webbplatsen och interaktionen med den. Verktyget är en del av den metod för utvärdering av emotionellt engagemang som vi utvecklat vilket består av användartester stödda av verktyget vilket följs upp av intervjuer.</em></p>

Emotion influences our actions, and this means that emotion has subjective decision value. Emotions, properly interpreted and understood, of those affected by decisions provide feedback to actions and, as such, serve as a basis for decisions. Accordingly, "affective computing" represents a wide range of technological opportunities toward the implementation of emotions to improve human-computer interaction, which also includes insights across a range of contexts of computational sciences into how we can design computer systems to communicate and recognize the emotional states provided by humans. Today, emotional systems such as software-only agents and embodied robots seem to improve every day at managing large volumes of information, and they remain emotionally incapable to read our feelings and react according to them. From a computational viewpoint, technology has made significant steps in determining how an emotional behavior model could be built; such a model is intended to be used for the purpose of intelligent assistance and support to humans. Human emotions are engines that allow people to generate useful responses to the current situation, taking into account the emotional states of others. Recovering the emotional cues emanating from the natural behavior of humans such as facial expressions and bodily kinetics could help to develop systems that allow recognition, interpretation, processing, simulation, and basing decisions on human emotions. Currently, there is a need to create emotional systems able to develop an emotional bond with users, reacting emotionally to encountered situations with the ability to help, assisting users to make their daily life easier. Handling emotions and their influence on decisions can improve the human-machine communication with a wider vision. The present thesis strives to provide an emotional architecture applicable for an agent, based on a group of decision-making models influenced by external emotional information provided by humans, acquired through a group of classification techniques from machine learning algorithms. The system can form positive bonds with the people it encounters when proceeding according to their emotional behavior. The agent embodied in the emotional architecture will interact with a user, facilitating their adoption in application areas such as caregiving to provide emotional support to the elderly. The agent's architecture uses an adversarial structure based on an Adversarial Risk Analysis framework with a decision analytic flavor that includes models forecasting a human's behavior and their impact on the surrounding environment. The agent perceives its environment and the actions performed by an individual, which constitute the resources needed to execute the agent's decision during the interaction. The agent's decision that is carried out from the adversarial structure is also affected by the information of emotional states provided by a classifiers-ensemble system, giving rise to a "decision with emotional connotation" included in the group of affective decisions. The performance of different well-known classifiers was compared in order to select the best result and build the ensemble system, based on feature selection methods that were introduced to predict the emotion. These methods are based on facial expression, bodily gestures, and speech, with satisfactory accuracy long before the final system. / <p>At the time of the doctoral defense, the following paper was unpublished and had a status as follows: Paper 8: Accepted.</p>

Este trabalho propõe um agente pedagógico animado que possui o objetivo de fornecer suporte emocional ao aluno: motivando-o e encorajando-o, fazendo-o acreditar em suas próprias habilidades e promovendo um estado de espírito positivo no aluno, que é melhor para o seu aprendizado. Este suporte cuidadoso do agente, suas táticas afetivas, é expresso através de comportamentos emotivos e mensagens de encorajamento do personagem animado. Devido à tendência social humana de antropomorfizar software, nós acreditamos que um agente de software pode realizar esse papel afetivo. Para escolher as táticas afetivas adequadas, o agente deve conhecer as emoções do aluno. O agente proposto infere as seguintes emoções do aluno: alegria/tristeza, satisfação/frustração, raiva/gratidão e vergonha a partir do comportamento observável do aluno, isto é, as ações do aluno na interface do sistema educacional. A inferência das emoções é fundamentada psicologicamente na teoria cognitiva das emoções. Mais especificamente, nós usamos o modelo OCC o qual é baseado na abordagem cognitivista das emoções e é possível de ser implementado computacionalmente. Devido a natureza dinâmica da informação sobre o estado afetivo do aluno, nós adotamos uma abordagem BDI para implementar o modelo afetivo do usuário e o diagnóstico afetivo. Além disso, em nosso trabalho nós nos beneficiamos da capacidade de raciocínio do BDI para o agente deduzir o appraisal do aluno, que lhe permite inferir as emoções do aluno. Como um caso de estudo, o agente proposto é implementado como o Agente Mediador de MACES: um ambiente para ensino colaborativo à distância modelado com uma arquitetura multiagente e baseado psicologicamente na abordagem Sociocultural de Vygotsky. / This work proposes an animated pedagogical agent that has the role of providing emotional support to the student: motivating and encouraging him, making him believe in his self-ability, and promoting a positive mood in him, which fosters learning. This careful support of the agent, its affective tactics, is expressed through emotional behaviour and encouragement messages of the lifelike character. Due to human social tendency of anthropomorphising software, we believe that a software agent can accomplish this affective role. In order to choose the adequate affective tactics, the agent should also know the student’s emotions. The proposed agent recognises the student’s emotions: joy/distress, satisfaction/disappointment, anger/gratitude, and shame, from the student’s observable behaviour, i. e. his actions in the interface of the educational system. The inference of emotions is psychologically grounded on the cognitive theory of emotions. More specifically, we use the OCC model which is based on the cognitive approach of emotion and can be computationally implemented. Due to the dynamic nature of the student’s affective information, we adopted a BDI approach to implement the affective user model and the affective diagnosis. Besides, in our work we profit from the reasoning capacity of the BDI approach in order for the agent to deduce the student’s appraisal, which allows it to infer the student’s emotions. As a case study, the proposed agent is implemented as the Mediating Agent of MACES: an educational collaborative environment modelled as a multi-agent system and pedagogically based on the sociocultural theory of Vygotsky.

Este trabalho propõe um agente pedagógico animado que possui o objetivo de fornecer suporte emocional ao aluno: motivando-o e encorajando-o, fazendo-o acreditar em suas próprias habilidades e promovendo um estado de espírito positivo no aluno, que é melhor para o seu aprendizado. Este suporte cuidadoso do agente, suas táticas afetivas, é expresso através de comportamentos emotivos e mensagens de encorajamento do personagem animado. Devido à tendência social humana de antropomorfizar software, nós acreditamos que um agente de software pode realizar esse papel afetivo. Para escolher as táticas afetivas adequadas, o agente deve conhecer as emoções do aluno. O agente proposto infere as seguintes emoções do aluno: alegria/tristeza, satisfação/frustração, raiva/gratidão e vergonha a partir do comportamento observável do aluno, isto é, as ações do aluno na interface do sistema educacional. A inferência das emoções é fundamentada psicologicamente na teoria cognitiva das emoções. Mais especificamente, nós usamos o modelo OCC o qual é baseado na abordagem cognitivista das emoções e é possível de ser implementado computacionalmente. Devido a natureza dinâmica da informação sobre o estado afetivo do aluno, nós adotamos uma abordagem BDI para implementar o modelo afetivo do usuário e o diagnóstico afetivo. Além disso, em nosso trabalho nós nos beneficiamos da capacidade de raciocínio do BDI para o agente deduzir o appraisal do aluno, que lhe permite inferir as emoções do aluno. Como um caso de estudo, o agente proposto é implementado como o Agente Mediador de MACES: um ambiente para ensino colaborativo à distância modelado com uma arquitetura multiagente e baseado psicologicamente na abordagem Sociocultural de Vygotsky. / This work proposes an animated pedagogical agent that has the role of providing emotional support to the student: motivating and encouraging him, making him believe in his self-ability, and promoting a positive mood in him, which fosters learning. This careful support of the agent, its affective tactics, is expressed through emotional behaviour and encouragement messages of the lifelike character. Due to human social tendency of anthropomorphising software, we believe that a software agent can accomplish this affective role. In order to choose the adequate affective tactics, the agent should also know the student’s emotions. The proposed agent recognises the student’s emotions: joy/distress, satisfaction/disappointment, anger/gratitude, and shame, from the student’s observable behaviour, i. e. his actions in the interface of the educational system. The inference of emotions is psychologically grounded on the cognitive theory of emotions. More specifically, we use the OCC model which is based on the cognitive approach of emotion and can be computationally implemented. Due to the dynamic nature of the student’s affective information, we adopted a BDI approach to implement the affective user model and the affective diagnosis. Besides, in our work we profit from the reasoning capacity of the BDI approach in order for the agent to deduce the student’s appraisal, which allows it to infer the student’s emotions. As a case study, the proposed agent is implemented as the Mediating Agent of MACES: an educational collaborative environment modelled as a multi-agent system and pedagogically based on the sociocultural theory of Vygotsky.