Human communication is inherently multimodal. The information conveyed through body language, facial expression, gaze, intonation, speaking style etc. are all important components of everyday communication. An issue within computer science concerns how to provide multimodal agent based systems. Those are systems that interact with users through several channels. These systems can include Virtual Humans. A Virtual Human might for example be a complete creature, i.e. a creature with a whole body including head, arms, legs etc. but it might also be a creature with only a head, a Talking Head. The aim of the Virtual Human Markup Language (VHML) is to control Virtual Humans regarding speech, facial animation, facial gestures and body animation. These parts have previously been implemented and investigated separately, but VHML aims to combine them. In this thesis VHML is verified, validated and evaluated in order to reach that aim and thus VHML is made more solid, homogenous and complete. Further, a Virtual Human has to communicate with the user and even though VHML supports a number of other ways of communication, an important communication channel is speech. The Virtual Human has to be able to interact with the user, therefore a dialogue between the user and the Virtual Human has to be created. These dialogues tend to expand tremendously, hence the Dialogue Management Tool (DMT) was developed. Having a toolmakes it easier for programmers to create and maintain dialogues for the interaction. Finally, in order to demonstrate the work done in this thesis a Talking Head application, The Mystery at West Bay Hospital, has been developed and evaluated. This has shown the usefulness of the DMT when creating dialogues. The work that has been accomplished within this project has contributed to simplify the development of Talking Head applications.

<p>Human communication is inherently multimodal. The information conveyed through body language, facial expression, gaze, intonation, speaking style etc. are all important components of everyday communication. An issue within computer science concerns how to provide multimodal agent based systems. Those are systems that interact with users through several channels. These systems can include Virtual Humans. A Virtual Human might for example be a complete creature, i.e. a creature with a whole body including head, arms, legs etc. but it might also be a creature with only a head, a Talking Head. The aim of the Virtual Human Markup Language (VHML) is to control Virtual Humans regarding speech, facial animation, facial gestures and body animation. These parts have previously been implemented and investigated separately, but VHML aims to combine them. In this thesis VHML is verified, validated and evaluated in order to reach that aim and thus VHML is made more solid, homogenous and complete. Further, a Virtual Human has to communicate with the user and even though VHML supports a number of other ways of communication, an important communication channel is speech. The Virtual Human has to be able to interact with the user, therefore a dialogue between the user and the Virtual Human has to be created. These dialogues tend to expand tremendously, hence the Dialogue Management Tool (DMT) was developed. Having a toolmakes it easier for programmers to create and maintain dialogues for the interaction. Finally, in order to demonstrate the work done in this thesis a Talking Head application, The Mystery at West Bay Hospital, has been developed and evaluated. This has shown the usefulness of the DMT when creating dialogues. The work that has been accomplished within this project has contributed to simplify the development of Talking Head applications.</p>

In this thesis a new concept called domain knowledge management for informationproviding dialogue systems is introduced. Domain knowledge management includes issues related to representation and use of domain knowledge as well as access of background information sources, issues that previously have been incorporated in dialogue management. The work on domain knowledge management reported in this thesis can be divided in two parts. On a general theoretical level, knowledge sources and models used for dialogue management, including domain knowledge management, are studied and related to the capabilities they support. On a more practical level, domain knowledge management is examined in the contexts of a dialogue system framework and a specific instance of this framework, the ÖTRAF system. In this system domain knowledge management is implemented in a separate module, a Domain Knowledge Manager. The use of a specialised Domain Knowledge Manager has a number of advantages. The first is that dialogue management becomes more focused as it only has to consider dialogue phenomena, while domain-specific reasoning is handled by the Domain Knowledge Manager. Secondly, porting of a system to new domains is facilitated since domain-related issues are separated out in specialised domain knowledge sources. The third advantage with a separate module for domain knowledge management is that domain knowledge sources can be easily modified, exchanged, and reused. / <p>Report code: LiU-Tek-Lic-2001:27.</p>

Les récents progrès dans le domaine du traitement du langage ont apporté un intérêt significatif à la mise en oeuvre de systèmes de dialogue parlé. Ces derniers sont des interfaces utilisant le langage naturel comme medium d'interaction entre le système et l'utilisateur. Le module de gestion de dialogue choisit le moment auquel l'information qu'il choisit doit être échangée avec l'utilisateur. Ces dernières années, l'optimisation de dialogue parlé en utilisant l'apprentissage par renforcement est devenue la référence. Cependant, une grande partie des algorithmes utilisés nécessite une importante quantité de données pour être efficace. Pour gérer ce problème, des simulations d'utilisateurs ont été introduites. Cependant, ces modèles introduisent des erreurs. Par un choix judicieux d'algorithmes, la quantité de données d'entraînement peut être réduite et ainsi la modélisation de l'utilisateur évitée. Ces travaux concernent une partie des contributions présentées. L'autre partie des travaux consiste à proposer une modélisation à partir de données réelles des utilisateurs au moyen de l'apprentissage par renforcement inverse / Recent advancements in the area of spoken language processing and the wide acceptance of portable devices, have attracted signicant interest in spoken dialogue systems.These conversational systems are man-machine interfaces which use natural language (speech) as the medium of interaction.In order to conduct dialogues, computers must have the ability to decide when and what information has to be exchanged with the users. The dialogue management module is responsible to make these decisions so that the intended task (such as ticket booking or appointment scheduling) can be achieved.Thus learning a good strategy for dialogue management is a critical task.In recent years reinforcement learning-based dialogue management optimization has evolved to be the state-of-the-art. A majority of the algorithms used for this purpose needs vast amounts of training data.However, data generation in the dialogue domain is an expensive and time consuming process. In order to cope with this and also to evaluatethe learnt dialogue strategies, user modelling in dialogue systems was introduced. These models simulate real users in order to generate synthetic data.Being computational models, they introduce some degree of modelling errors. In spite of this, system designers are forced to employ user models due to the data requirement of conventional reinforcement learning algorithms can learn optimal dialogue strategies from limited amount of training data when compared to the conventional algorithms. As a consequence of this, user models are no longer required for the purpose of optimization, yet they continue to provide a fast and easy means for quantifying the quality of dialogue strategies. Since existing methods for user modelling are relatively less realistic compared to real user behaviors, the focus is shifted towards user modelling by means of inverse reinforcement learning. Using experimental results, the proposed method's ability to learn a computational models with real user like qualities is showcased as part of this work.

Les développements issus de la recherche en gestion du dialogue homme-machine portent essentiellement sur le dialogue utilitaire et délaissent le dialogue à caractère ludique ou récréatif. Une description détaillée du contexte et la reconnaissance des buts ne suffisent pas à appréhender les enjeux de ce type de dialogue. La thèse vise à démontrer qu'un système s'appuyant sur une reconnaissance robuste et fine des actes de dialogue et des intentions associée à une prise en compte de l'altérité, de l'éthique et des émotions peut faire émerger une personnalité dialogique à même d'interagir de façon crédible avec un humain et de reproduire certaines performances dans le contexte d'un dialogue de séduction. L'objectif ne consiste pas à faire en sorte que le système puisse être confondu avec un humain comme cela est le cas dans les tests d'intelligence mais plutôt faire en sorte que le système puisse être dirigé par l'intérêt de la conversation. A cette fin, les recherches portent sur la définition de la relation d'altérité JE-TU appliquée au dialogue par l'intermédiaire de la théorie des jeux notamment à travers les concepts de définition de soi, d'éthique de la discussion et de modèles d'émotions. Plusieurs pistes sont explorées dans le but de réunir un corpus d'étude, notamment des prototypes de jeux collaboratifs. Au final, le modèle de personnage est développé sur la base d'un corpus de scripts de cinéma. Ce modèle repose sur une nouvelle taxonomie de phénomènes de dialogue incluant des actes perlocutoires et une approche différente de la connaissance permettant d'inclure l'éthique et le lien d'altérité en son sein. Les stratégies qui régissent le dialogue sont alors décrites de manière beaucoup plus précise. Une scène extraite d'un film servant de cadre applicatif aux expérimentations permet de valider l'architecture du système de dialogue. / Most of the developments issued from the research in man computer dialogue management are mainly focused on functional dialogue and put aside entertainment dialogue. A sharper context description and goals recognition remains quite limited to fully grasp the stakes of type of dialogue. The thesis aims to demonstrate that a system based on a thin and robust recognition of dialogue acts and intentions linked to a consideration for the concept of relation of otherness, ethics and emotions could leads to the emergence of a dialogic personality. Such a system would be able to interact with a human being in a credible way and reproduce some of its achievements in the context of a seduction dialogue. Unlike the common approach developed for intelligence tests, the purpose is not to mimic a human, but to manage the dialogue based on the notion of interest. To achieve this goal, researches deal with the relation of otherness applied to dialogue within game theory and especially through the concepts of Self-definition, Discourse Ethics and Emotions Modeling. The corpora collection process follows many ways including collaborative games. At the end, the character model is developed using a movie script corpora on the basis of a new dialogue phenomena taxonomy including perlocutionary acts and a new approach of knowledge that incorporate ethics and the relation of otherness. It leads to a thinner description of dialogue strategies. A scene extracted from a movie aims to validate the final architecture as an experimental framework.

Les besoins croissants en formation et en entrainement au travail d’équipe ont motivé l’utilisationd’Environnements de réalité Virtuelle Collaboratifs de Formation (EVCF) qui permettent aux utilisateurs de travailler avec des agents autonomes pour réaliser une activité collective. L’idée directrice est que la coordination efficace entre les membres d’une équipe améliore la productivité et réduit les erreurs individuelles et collectives. Cette thèse traite de la mise en place et du maintien de la coordination au sein d’une équipe de travail composée d’agents et d’humains interagissant dans un EVCF.L’objectif de ces recherches est de doter les agents virtuels de comportements conversationnels permettant la coopération entre agents et avec l’utilisateur dans le but de réaliser un but commun.Nous proposons une architecture d’agents Collaboratifs et Conversationnels, dérivée de l’architecture Belief-Desire-Intention (C2-BDI), qui gère uniformément les comportements délibératifs et conversationnels comme deux comportements dirigés vers les buts de l’activité collective. Nous proposons un modèle intégré de la coordination fondé sur l’approche des modèles mentaux partagés, afin d’établir la coordination au sein de l’équipe de travail composée d’humains et d’agents. Nous soutenons que les interactions en langage naturel entre les membres d’une équipe modifient les modèles mentaux individuels et partagés des participants. Enfin, nous décrivons comment les agents mettent en place et maintiennent la coordination au sein de l’équipe par le biais de conversations en langage naturel. Afin d’établir un couplage fort entre la prise de décision et le comportement conversationnel collaboratif d’un agent, nous proposons tout d’abord une approche fondée sur la modélisation sémantique des activités humaines et de l’environnement virtuel via le modèle mascaret puis, dans un second temps, une modélisation du contexte basée sur l’approche Information State. Ces représentations permettent de traiter de manière unifiée les connaissances sémantiques des agents sur l’activité collective et sur l’environnement virtuel ainsi que des informations qu’ils échangent lors de dialogues.Ces informations sont utilisées par les agents pour la génération et la compréhension du langage naturel multipartite. L’approche Information State nous permet de doter les agents C2BDI de capacités communicatives leur permettant de s’engager pro-activement dans des interactions en langue naturelle en vue de coordonner efficacement leur activité avec les autres membres de l’équipe. De plus, nous définissons les protocoles conversationnels collaboratifs favorisant la coordination entre les membres de l’équipe. Enfin, nous proposons dans cette thèse un mécanisme de prise de décision s’inspirant de l’approche BDI qui lie les comportements de délibération et de conversation des agents. Nous avons mis en oeuvre notre architecture dans trois différents scénarios se déroulant dans des EVCF. Nous montrons que les comportements conversationnels collaboratifs multipartites des agents C2BDI facilitent la coordination effective de l’utilisateur avec les autres membres de l’équipe lors de la réalisation d’une tâche partagée. / Growing needs of educational and training requirements motivate the use of collaborative virtual environments for training (CVET) that allows human users to work together with autonomous agents to perform a collective activity. The vision is inspired by the fact that the effective coordination improves productivity, and reduces the individual and team errors. This work addresses the issue of establishing and maintaining the coordination in a mixed human-agent teamwork in the context of CVET. The objective of this research is to provide human-like conversational behavior of the virtual agents in order to cooperate with a user and other agents to achieve shared goals.We propose a belief-desire-intention (BDI) like Collaborative Conversational agent architecture(C2BDI) that treats both deliberative and conversational behaviors uniformly as guided by the goal-directed shared activity. We put forward an integrated model of coordination which is founded on the shared mental model based approaches to establish coordination in a human-agent teamwork. We argue that natural language interaction between team members can affect and modify the individual and shared mental models of the participants. Finally, we describe the cultivation of coordination in a mixed human-agent teamwork through natural language conversation. In order to establish the strong coupling between decision making and the collaborative conversational behavior of the agent, we propose first, the Mascaret based semantic modeling of human activities and the VE, and second, the information state based context model. This representation allows the treatment of semantic knowledge of the collaborative activity and virtual environment, and information exchanged during the dialogue conversation in a unified manner. This knowledge can be used by the agent for multiparty natural language processing (understanding and generation) in the context of the CEVT. To endow the communicative capabilities to C2BDI agent, we put forward the information state based approach for the natural language processing of the utterances. We define collaborative conversation protocols that ensure the coordination between team members. Finally, in this thesis, we propose a decision making mechanism, which is inspired by the BDI based approach and provides the interleaving between deliberation and conversational behavior of the agent. We have applied the proposed architecture to three different scenarios in the CVET. We found that the multiparty collaborative conversational behavior of C2BDI agent is more constructive and facilitates the user to effectively coordinate with other team members to perform a shared task.