In my teaching and software development practice, I realized that most applications with human-computer interaction do not respond to users’ emotional needs. The dualism of reason and emotion as two fairly opposite entities that dominated Western philosophy was also reflected in software design. Computing was originally intended to provide applications for military and industrial activities and was primarily associated with cognition and rationality. Today, more and more computer applications interact with users in very complex and sophisticated ways. In human-computer interaction, attention is given to issues of usability and user modeling, but techniques to emotionally engage users or respond to their emotional needs have not been fully developed, even as specialists like Klein, Norman and Picard argued that machines that recognize and express emotions respond better and more appropriately to user interaction (Picard, 1997; Picard & Klein, 2002; Norman, 2004). This study investigated emotion from designers’ perspectives and tentatively concludes that there is little awareness and involvement in emotional design in the IT community. By contrast, participants in this study (36 IT specialists from various fields) strongly supported the idea of emotional design and confirmed the need for methodologies and theoretical models to research emotional design. Based on a review of theory, surveys and interviews, I identified a set of themes for heuristics of emotional design and recommended future research directions. Attention was given to consequences; participants in this study raised issues of manipulation, ethical responsibilities of designers, and the need for regulations, and recommended that emotional design should carry standard ethical guidelines for games and any other applications. The research design utilized a mixed QUAN-qual methodological model proposed by Creswell (2003) and Gay, Mills, and Airasian (2006), which was modified to equally emphasize both quantitative and qualitative stages. An instrument in the form of a questionnaire was designed, tested and piloted in this study and will be improved and used in future research.

As robots become more and more prevalent in our everyday life, making sure that our interactions with them are natural and satisfactory is of paramount importance. Given the propensity of humans to treat machines as social actors, and the integral role affect plays in human life, providing robots with affective responses is a step towards making our interaction with them more intuitive. To the end of promoting more natural, satisfying and effective human-robot interaction and enhancing robotic behavior in general, an integrative framework of time-varying affective robotic behavior was designed and implemented on a humanoid robot. This psychologically inspired framework (TAME) encompasses 4 different yet interrelated affective phenomena: personality Traits, affective Attitudes, Moods and Emotions. Traits determine consistent patterns of behavior across situations and environments and are generally time-invariant; attitudes are long-lasting and reflect likes or dislikes towards particular objects, persons, or situations; moods are subtle and relatively short in duration, biasing behavior according to favorable or unfavorable conditions; and emotions provide a fast yet short-lived response to environmental contingencies. The software architecture incorporating the TAME framework was designed as a stand-alone process to promote platform-independence and applicability to other domains. In this dissertation, the effectiveness of affective robotic behavior was explored and evaluated in a number of human-robot interaction studies with over 100 participants. In one of these studies, the impact of Negative Mood and emotion of Fear was assessed in a mock-up search-and-rescue scenario, where the participants found the robot expressing affect more compelling, sincere, convincing and "conscious" than its non-affective counterpart. Another study showed that different robotic personalities are better suited for different tasks: an extraverted robot was found to be more welcoming and fun for a task as a museum robot guide, where an engaging and gregarious demeanor was expected; whereas an introverted robot was rated as more appropriate for a problem solving task requiring concentration. To conclude, multi-faceted robotic affect can have far-reaching practical benefits for human-robot interaction, from making people feel more welcome where gregariousness is expected to making unobtrusive partners for problem solving tasks to saving people's lives in dangerous situations.

Zugl.: Magdeburg, Univ., Diss., 2009

Zugl.: Paderborn, Universiẗat, Diss., 2009.

Effective game music is typically adaptive, interactive, or both. Changes in game music are usually influenced by the current state of the game or the actions of the player. To provide another dimension of interactivity, it would be useful to know the affective state of the human player. Biosignals are continuous signals generated by a person that can be measured over time, and have been shown to reflect affective state. This project demonstrates that control signals can be gathered from the player and mapped to musical parameters. Using a heart rate sensor and galvanic skin response sensor built from open source designs, we have used biosignals to control music playback while playing four games from different genres. A system for controlling game music with biosignals is computationally cheap, and can provide data that is useful to other game systems. The prototype developed for this project is basic, but with further research and development, we believe such a system will greatly improve the immersive experience of video games by involving the player on a new level. / text

Emotions are thought to be one of the key factors that critically influences human decision-making. Emotion regulation can help to mitigate emotion related decision biases and eventually lead to a better decision performance. Serious games emerged as a new angle introducing technological methods to learning emotion regulation, where meaningful biofeedback information communicates player's emotional state. Games are a series of interesting choices, where design of those choices could support an educational platform to learning emotion regulation. Such design could benefit digital serious games as those choices could be informed though player's physiology about emotional states in real time. This thesis explores design and evaluation methods for creating serious games where emotion regulation can be learned and practiced. Design of a digital serious game using physiological measures of emotions was investigated and evaluated. Furthermore, it investigates emotions and the effect of emotion regulation on decision performance in digital serious games. The scope of this thesis was limited to digital serious games for emotion regulation training using psychophysiological methods to communicate player's affective information. Using the psychophysiological methods in design and evaluation of digital serious games, emotions and their underlying neural mechanism have been explored. Effects of emotion regulation have been investigated where decision performance has been measured and analyzed. The proposed metrics for designing and evaluating such affective serious games have been extensively evaluated. The research methods used in this thesis were based on both quantitative and qualitative aspects, with true experiment and evaluation research, respectively. Digital serious games approach to emotion regulation was investigated, player's physiology of emotions informs design of interactions where regulation of those emotions could be practiced. The results suggested that two different emotion regulation strategies, suppression and cognitive reappraisal, are optimal for different decision tasks contexts. With careful design methods, valid serious games for training those different strategies could be produced. Moreover, using psychophysiological methods, underlying emotion neural mechanism could be mapped. This could inform a digital serious game about an optimal level of arousal for a certain task, as evidence suggests that arousal is equally or more important than valence for decision-making. The results suggest that it is possible to design and develop digital serious game applications that provide helpful learning environment where decision makers could practice emotion regulation and subsequently improve their decision-making. If we assume that physiological arousal is more important than physiological valence for learning purposes, results show that digital serious games designed in this thesis elicit high physiological arousal, suitable for use as an educational platform.

Computer agents are required to make appropriate decisions quickly and efficiently. As the environments in which they act become increasingly complex, efficient decision-making becomes significantly more challenging. This thesis examines the positive ways in which human emotions influence people’s ability to make good decisions in complex, uncertain contexts, and develops computational analogues of these beneficial functions, demonstrating their usefulness in agent decision-making and communication. For decision-making by a single agent in large-scale environments with stochasticity and high uncertainty, the thesis presents GRUE (Goal Re-prioritization Using Emotion), a decision-making technique that deploys emotion-inspired computational operators to dynamically re-prioritize the agent’s goals. In two complex domains, GRUE is shown to result in improved agent performance over many existing techniques. Agents working in groups benefit from communicating and sharing information that would otherwise be unobservable. The thesis defines an affective signaling mechanism, inspired by the beneficial communicative functions of human emotion, that increases coordination. In two studies, agents using the mechanism are shown to make faster and more accurate inferences than agents that do not signal, resulting in improved performance. Moreover, affective signals confer performance increases equivalent to those achieved by broadcasting agents’ entire private state information. Emotions are also useful signals in agents’ interactions with people, influencing people’s perceptions of them. A computer-human negotiation study is presented, in which virtual agents expressed emotion. Agents whose emotion expressions matched their negotiation strategy were perceived as more trustworthy, and they were more likely to be selected for future interactions. In addition, to address similar limitations in strategic environments, this thesis uses the theory of reasoning patters in complex game-theoretic settings. An algorithm is presented that speeds up equilibrium computation in certain classes of games. For Bayesian games, with and without a common prior, the thesis also discusses a novel graphical formalism that allows agents’ possibly inconsistent beliefs to be succinctly represented, and for reasoning patterns to be defined in such games. Finally, the thesis presents a technique for generating advice from a game’s reasoning patterns for human decision-makers, and demonstrates empirically that such advice helps people make better decisions in a complex game. / Engineering and Applied Sciences

In my teaching and software development practice, I realized that most applications with human-computer interaction do not respond to users’ emotional needs. The dualism of reason and emotion as two fairly opposite entities that dominated Western philosophy was also reflected in software design. Computing was originally intended to provide applications for military and industrial activities and was primarily associated with cognition and rationality. Today, more and more computer applications interact with users in very complex and sophisticated ways. In human-computer interaction, attention is given to issues of usability and user modeling, but techniques to emotionally engage users or respond to their emotional needs have not been fully developed, even as specialists like Klein, Norman and Picard argued that machines that recognize and express emotions respond better and more appropriately to user interaction (Picard, 1997; Picard & Klein, 2002; Norman, 2004). This study investigated emotion from designers’ perspectives and tentatively concludes that there is little awareness and involvement in emotional design in the IT community. By contrast, participants in this study (36 IT specialists from various fields) strongly supported the idea of emotional design and confirmed the need for methodologies and theoretical models to research emotional design. Based on a review of theory, surveys and interviews, I identified a set of themes for heuristics of emotional design and recommended future research directions. Attention was given to consequences; participants in this study raised issues of manipulation, ethical responsibilities of designers, and the need for regulations, and recommended that emotional design should carry standard ethical guidelines for games and any other applications. The research design utilized a mixed QUAN-qual methodological model proposed by Creswell (2003) and Gay, Mills, and Airasian (2006), which was modified to equally emphasize both quantitative and qualitative stages. An instrument in the form of a questionnaire was designed, tested and piloted in this study and will be improved and used in future research.

Este trabalho apresenta um framework que permite a programação de agentes capazes de perceberem seus próprios estados emocionais. O framework foi desenvolvido em Java com base na plataforma multi-agente Jason, estendendo a base de crenças de agentes Jason a fim de utilizar a ontologia afetiva desenvolvida. Além disso, o ambiente foi construído a partir de uma base de conhecimento que descreve rotinas em ambientes simulados. Um mecanismo de avaliação das emoções baseando-se nas anotações dos objetos foi construído apoiado por uma ontologia de preferência sobre essas anotações. Dessa forma, aplicações de entretenimento poderiam utilizar o sistema ou as bases de conhecimento apresentadas para diferentes propósitos. A criação de um mapa onde os personagens atuam, e a criação da rotina de cada personagem e suas preferências são alguns exemplos de utilizações. Para validação do framework desenvolvido, dois exemplos foram construídos. O primeiro utilizou a maior parte dos grupos afetivos da ontologia proposta, com a finalidade principal de demonstrar o modelo implementado. Já o segundo usa apenas um grupo emotivo e serve para demonstrar a utilização conjunta de todas as ontologias apresentadas. / This work presents a framework built on top of the Jason platform (BORDINI et al., 2004) to allow the development of software agents that have emotional states. The framework was developed in Java and extends the belief base of Jason agents so as to use an ontology for the OCC affective model (ORTONY; COLLINS; CLORE, 1988) that has been created as part of this work. The developed belief base allows an agent to perceive its own emotions throw inferring new beliefs based on the agent’s appraisal of the state of the environment. In addition, a model of agents’ routine tasks was defined, as was a model for agents’ preferences about aspects of environment, helping automate the ascription of emotional states. Finally, in order to validate the developed framework, two applications were developed. The first demonstrates the use of various different emotions from the affective model and the second uses in a single application all the ontologies and models developed as part of this work.

Este trabalho apresenta o processo de construção de um modelo de inferência de emoções que um aluno sente em relação a outros alunos durante interação síncrona em um contexto de jogo colaborativo de aprendizagem. A inferência de emoções está psicologicamente fundamentada na abordagem da avaliação cognitiva e foram investigadas relações entre objetivos e normas comportamentais do aluno e aspectos de sua personalidade. Especificamente, foram empregados o modelo OCC de emoções e o modelo Big-Five (Cinco Grandes Fatores) de traços de personalidade para a fundamentação teórica da modelagem. O modelo afetivo representa a vergonha e orgulho apresentados pelo aluno em resposta à avaliação cognitiva de suas próprias ações e a reprovação e admiração apresentadas pelo aluno em resposta a ações de seu parceiro de jogo, a partir da avaliação do comportamento observável dos parceiros representado por suas interações no jogo colaborativo, em relação a normas comportamentais do aluno. A fim de suportar a incerteza presente na informação afetiva e cognitiva do aluno, adotou-se uma representação deste conhecimento através de Rede Bayesiana. Um refinamento qualitativo parcial e a respectiva parametrização quantitativa do modelo probabilístico foram efetuados a partir da análise de uma base de casos obtida através da condução de experimentos. A fim de prover um ambiente experimental, foi concebido e prototipado um jogo colaborativo no qual dois indivíduos conjugam esforços a fim de resolver problemas lógicos comuns à dupla, através de ações coordenadas, negociação simples e comunicação estruturada, em competição com outras duplas. / This work presents the construction of a model to infer emotions a student feels towards other students during synchronous interaction in the context of a collaborative learning game. The emotions inference is psychologically based on cognitive appraisal theory. Some relations between students’ personality and their goals and behavioral standards were also investigated. This modeling was based on OCC emotion model and Big-Five personality model. The affective model represents the student’s proud and shame as an answer to the cognitive appraisal of her/his own attributed interactions, and the student’s admiration and reproach as an answer to the cognitive appraisal of her/his partner attributed interactions, both according to the student’s behavioral standards. Bayesian Network knowledge representation was employed to better stand for the uncertainty present in the student’s cognitive and affective information. Employing a data-driven procedure, the probabilistic model was partially refined in terms of qualitative relations and quantitative parameters. Experimental data were obtained by using a game prototype implemented in order to support a collaborative dynamics of coordinated action, simple negotiation and structured communication, through which students interacted in order to solve shared problems, during synchronous competition with other students.