Robots and other computational agents are increasingly becoming part of our daily lives. They will need to be able to learn to perform new tasks, adapt to novel situations, and understand what is wanted by their human users, most of whom will not have programming skills. To achieve these ends, agents must learn from humans using methods of communication that are naturally accessible to everyone. This thesis presents and formalizes interactive shaping, one such teaching method, where agents learn from real-valued reward signals that are generated by a human trainer. In interactive shaping, a human trainer observes an agent behaving in a task environment and delivers feedback signals. These signals are mapped to numeric values, which are used by the agent to specify correct behavior. A solution to the problem of interactive shaping maps human reward to some objective such that maximizing that objective generally leads to the behavior that the trainer desires. Interactive shaping addresses the aforementioned needs of real-world agents. This teaching method allows human users to quickly teach agents the specific behaviors that they desire. Further, humans can shape agents without needing programming skills or even detailed knowledge of how to perform the task themselves. In contrast, algorithms that learn autonomously from only a pre-programmed evaluative signal often learn slowly, which is unacceptable for some real-world tasks with real-world costs. These autonomous algorithms additionally have an inflexibly defined set of optimal behaviors, changeable only through additional programming. Through interactive shaping, human users can (1) specify and teach desired behavior and (2) share task knowledge when correct behavior is already indirectly specified by an objective function. Additionally, computational agents that can be taught interactively by humans provide a unique opportunity to study how humans teach in a highly controlled setting, in which the computer agent’s behavior is parametrized. This thesis answers the following question. How and to what extent can agents harness the information contained in human-generated signals of reward to learn sequential decision-making tasks? The contributions of this thesis begin with an operational definition of the problem of interactive shaping. Next, I introduce the tamer framework, one solution to the problem of interactive shaping, and describe and analyze algorithmic implementations of the framework within multiple domains. This thesis also proposes and empirically examines algorithms for learning from both human reward and a pre-programmed reward function within an MDP, demonstrating two techniques that consistently outperform learning from either feedback signal alone. Subsequently, the thesis shifts its focus from the agent to the trainer, describing two psychological studies in which the trainer is manipulated by either changing their perceived role or by having the agent intentionally misbehave at specific times; we examine the effect of these manipulations on trainer behavior and the agent’s learned task performance. Lastly, I return to the problem of interactive shaping, for which we examine a space of mappings from human reward to objective functions, where mappings differ by how much the agent discounts reward it expects to receive in the future. Through this investigation, a deep relationship is identified between discounting, the level of positivity in human reward, and training success. Specific constraints of human reward are identified (i.e., the “positive circuits” problem), as are strategies for overcoming these constraints, pointing towards interactive shaping methods that are more effective than the already successful tamer framework. / text

京都大学 / 0048 / 新制・課程博士 / 博士(情報学) / 甲第17925号 / 情博第507号 / 新制||情||90(附属図書館) / 30745 / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 西田 豊明, 教授 河原 達也, 教授 黒橋 禎夫 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DGAM

京都大学 / 0048 / 新制・課程博士 / 博士(情報学) / 甲第19110号 / 情博第556号 / 新制||情||98(附属図書館) / 32061 / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 西田 豊明, 教授 乾 敏郎, 教授 河原 達也 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DGAM

L'essor des travaux en informatique affective voit la naissance de diverses questions de recherches pour étudier les interactions agents /humains. Parmi elles, se pose la question de l'impact des relations interpersonnelles sur les stratégies de communications. Les interactions entre un agent conversation et un utilisateur humain prennent généralement place dans des environnements collaboratifs où les interlocuteurs partagent des buts communs. La relation interpersonnelle que les individus créent durant leurs interactions affecte leurs stratégies de communications. Par ailleurs, des individus qui collaborent pour atteindre un but commun sont généralement amenés à négocier. Ce type de négociation permet aux négociateurs d'échanger des informations afin de mieux collaborer. L'objectif cette thèse est d'étudier l'impact de la relation interpersonnelle de dominance sur les stratégies de négociation collaborative entre un agent et un humain. Ce travail se base sur des études en psychologie sociale qui ont défini les comportements liés à la manifestation de la dominance dans une négociation. Nous proposons un modèle de négociation collaborative dont le modèle décisionnel est régi par la relation de dominance. En effet, en fonction de sa position dans le spectre de dominance, l'agent est capable d'exprimer une stratégie de négociation spécifique. En parallèle, l'agent simule une relation interpersonnelle de dominance avec son interlocuteur. Pour ce faire, nous avons doté l'agent d'un modèle de théorie de l'esprit qui permet à l'agent de raisonner sur les comportements de son interlocuteur afin de prédire sa position dans le spectre de dominance. Ensuite, il adapte sa stratégie de négociation vers une stratégie complémentaire à celle détectée chez son interlocuteur. Nos résultats ont montré que les comportements de dominance exprimés par notre agent sont correctement perçus. Par ailleurs, le modèle de la théorie de l'esprit est capable de faire de bonnes prédictions avec seulement une représentation partielle de l'état mental de l'interlocuteur. Enfin, la simulation de la relation interpersonnelle de dominance a un impact positif sur la négociation: les négociateurs atteignent de bon taux de gains communs. De plus, la relation de dominance augmente le sentiment d'appréciation entre les négociateurs et la négociation est perçue comme confortable. / The rise of work in affective computing sees the emergence of various research questions to study agent / human interactions. Among them raises the question of the impact of interpersonal relations on the strategies of communication. Human/agent interactions usually take place in collaborative environments in which the agent and the user share common goals. The interpersonal relations which individuals create during their interactions affects their communications strategies. Moreover, individuals who collaborate to achieve a common goal are usually brought to negotiate. This type of negotiation allows the negotiators to efficiently exchange information and their respective expertise in order to better collaborate. The objective of this thesis is to study the impact of the interpersonal relationship of dominance on collaborative negotiation strategies between an agent and a human. This work is based on studies from social psychology to define the behaviours related to the manifestation of dominance in a negotiation. We propose a collaborative negotiation model whose decision model is governed by the interpersonal relation of dominance. Depending on its position in the dominance spectrum, the agent is able to express a specific negotiation strategy. In parallel, the agent simulates an interpersonal relationship of dominance with his interlocutor. To this aim, we provided the agent with a model of theory of mind that allows him to reason about the behaviour of his interlocutor in order to predict his position in the dominance spectrum. Afterwards, the agent adapts his negotiation strategy to complement the negotiation strategy detected in the interlocutor. Our results showed that the dominance behaviours expressed by our agent are correctly perceived by human participants. Furthermore, our model of theory of mind is able de make accurate predictions of the interlocutor behaviours of dominance with only a partial representation of the other's mental state. Finally, the simulation of the interpersonal relation of dominance has a positive impact on the negotiation: the negotiators reach a good rate of common gains and the negotiation is perceived comfortable which increases the liking between the negotiators.

Dans une interaction humain-agent, l’engagement de l’utilisateur est un élément essentiel pour atteindre l’objectif de l’interaction. Dans cette thèse, nous étudions comment l’engagement de l’utilisateur pourrait être favorisé par le comportement de l’agent. Nous nous concentrons sur les stratégies de comportement verbal de l’agent qui concernent respectivement la forme, le timing et le contenu de ses énoncés. Nous présentons des études empiriques qui concernent certains aspects du comportement de politesse de l’agent, du comportement d’interruption de l’agent, et les sujets de conversation que l’agent adresse lors de l’interaction. Basé sur les résultats de la dernière étude, nous proposons un Gestionnaire de Sujets axé sur l’engagement (modèle computationnel) qui personnalise les sujets d’une interaction dans des conversations où l’agent donne des informations à un utilisateur humain. Le Modèle de Sélection des Sujets du Gestionnaire de Sujets décide sur quoi l’agent devrait parler et quand. Pour cela, il prend en compte la perception par l’agent de l’utilisateur, qui est dynamiquement mis à jour, ainsi que l’état mental et les préférences de l’agent. Le Modèle de Transition de Sujets du Gestionnaire de Sujet, basé sur une étude empirique, calcule comment l’agent doit présenter les sujets dans l’interaction en cours sans perdre la cohérence de l’interaction. Nous avons implémenté et évalué le Gestionnaire de Sujets dans un agent virtuel conversationnel qui joue le rôle d’un visiteur dans un musée. / In human-agent interaction the engagement of the user is an essential aspect to complete the goal of the interaction. In this thesis we study how the user’s engagement could be favoured by the agent’s behaviour. We thereby focus on the agent’s verbal behaviour considering strategies that regard respectively the form, timing, and content of utterances : We present empirical studies that regard (aspects of) the agent’s politeness behaviour, interruption behaviour, and the topics that the agent addresses in the interaction. Based on the outcomes of the latter study we propose an engagement-driven Topic Manager (computational model) that personalises the topics of an interaction in human-agent information-giving chat. The Topic Selection component of the Topic Manager decides what the agent should talk about and when. For this it takes into account the agent’s dynamically updated perception of the user as well as the agent’s own mental state. The Topic Transition component of the Topic Manager, based upon an empirical study, computes how the agent should introduce the topics in the ongoing interaction without loosing the coherence of the interaction. We implemented and evaluated the Topic Manager in a conversational virtual agent that plays the role of a visitor in amuseum.