This thesis aims to answer the following question : “How could gestures and facial expressions be used to control the behavior of an interactive entertaining application?”. An answer to this question is presented and illustrated in the context of mixed reality interactive storytelling. The first part focuses on the description of the Artificial Intelligence (AI) mechanisms that are used to model and control the behavior of the application. We present an efficient real-time hierarchical planning engine, and show how active modalities (such as intentional gestures) and passive modalities (such as facial expressions) can be integrated into the planning algorithm, in such a way that the narrative (driven by the behavior of the virtual characters inside the virtual world) can effectively evolve in accordance with user interactions. The second part is devoted to the automatic recognition of user interactions. After briefly describing the implementation of a simple but robust rule-based gesture recognition system, the emphasis is set on facial expression recognition. A complete solution integrating state-of-the-art techniques along with original contributions is drawn. It includes face detection, facial feature extraction and analysis. The proposed approach combines statistical learning and probabilistic reasoning in order to deal with the uncertainty associated with the process of modeling facial expressions.

Submitted by PPG Ci?ncia da Computa??o (ppgcc@pucrs.br) on 2018-05-08T18:37:11Z No. of bitstreams: 1 RAFAEL_CAU?_CARDOSO_TES.pdf: 14431785 bytes, checksum: 227194ed28a9e55c3ab1fbedebf06922 (MD5) / Approved for entry into archive by Caroline Xavier (caroline.xavier@pucrs.br) on 2018-05-15T19:07:36Z (GMT) No. of bitstreams: 1 RAFAEL_CAU?_CARDOSO_TES.pdf: 14431785 bytes, checksum: 227194ed28a9e55c3ab1fbedebf06922 (MD5) / Made available in DSpace on 2018-05-15T19:14:18Z (GMT). No. of bitstreams: 1 RAFAEL_CAU?_CARDOSO_TES.pdf: 14431785 bytes, checksum: 227194ed28a9e55c3ab1fbedebf06922 (MD5) Previous issue date: 2018-03-27 / Sistemas multiagentes freq?entemente cont?m ambientes complexos e din?micos, nos quais os planos dos agentes podem falhar a qualquer momento durante a execu??o do sistema. Al?m disso, novos objetivos podem aparecer para os quais n?o existem nenhum plano dispon?vel. T?cnicas de planejamento s?o bem adequadas para lidar com esses problemas. H? uma quantidade extensa de pesquisa em planejamento centralizado para um ?nico agente, por?m, at? ent?o planejamento multiagente n?o foi completamente explorado na pr?tica. Plataformas multiagentes tipicamente proporcionam diversos mecanismos para coordena??o em tempo de execu??o, frequentemente necess?rios em planejamento online. Neste contexto, planejamento multiagente descentralizado pode ser eficiente e eficaz, especialmente em dom?nios fracamente acoplados, al?m de garantir algumas propriedades importantes em sistemas de agentes como privacidade e autonomia. N?s abordamos esse problema ao apresentar uma t?cnica para planejamento multiagente online que combina aloca??o de objetivos, planejamento individual utilizando rede de tarefas hier?rquicas (HTN), e coordena??o em tempo de execu??o para apoiar a realiza??o de objetivos sociais em sistemas multiagentes. Especificamente, n?s apresentamos um framework chamado Decentralised Online Multi-Agent Planning (DOMAP). Experimentos com tr?s dom?nios fracamente acoplados demonstram que DOMAP supera quatro planejadores multiagente do estado da arte com respeito a tempo de planejamento e tempo de execu??o, particularmente nos problemas mais dif?ceis. / Multi-agent systems often contain dynamic and complex environments where agents? course of action (plans) can fail at any moment during execution of the system. Furthermore, new goals can emerge for which there are no known plan available in any of the agents? plan library. Automated planning techniques are well suited to tackle both of these issues. Extensive research has been done in centralised planning for singleagents, however, so far multi-agent planning has not been fully explored in practice. Multi-agent platforms typically provide various mechanisms for runtime coordination, which are often required in online planning (i.e., planning during runtime). In this context, decentralised multi-agent planning can be efficient as well as effective, especially in loosely-coupled domains, besides also ensuring important properties in agent systems such as privacy and autonomy. We address this issue by putting forward an approach to online multi-agent planning that combines goal allocation, individual Hierarchical Task Network (HTN) planning, and coordination during runtime in order to support the achievement of social goals in multi-agent systems. In particular, we present a planning and execution framework called Decentralised Online Multi-Agent Planning (DOMAP). Experiments with three loosely-coupled planning domains show that DOMAP outperforms four other state-of-the-art multi agent planners with regards to both planning and execution time, particularly in the most difficult problems.

Les agents robotiques mobiles ou aériens sont confrontés au besoin de planifier des actions avec information incomplètesur l'état du monde. Dans ce contexte, cette thèse propose un cadre de modélisation et de résolution de problèmes deplanification de perception et de mission pour un drone hélicoptère qui évolue dans un environnement incertain etpartiellement observé afin de détecter et de reconnaître des cibles. Nous avons fondé notre travail sur les ProcessusDécisionnels Markoviens Partiellement Observables (POMDP), car ils proposent un schéma d'optimisation général pour lestâches de perception et de décision à long terme. Une attention particulière est donnée à la modélisation des sortiesincertaines de l'algorithme de traitement d'image en tant que fonction d'observation. Une analyse critique de la mise enoeuvre en pratique du modèle POMDP et du critère d'optimisation associé est proposée. Afin de respecter les contraintes desécurité et de sûreté de nos robots aériens, nous proposons ensuite une approche pour tenir compte des propriétés defaisabilité d'actions dans des domaines partiellement observables : le modèle AC-POMDP, qui sépare l'informationconcernant la vérification des propriétés du modèle, de celle qui renseigne sur la nature des cibles. Enfin, nous proposonsun cadre d'optimisation et d'exécution en parallèle de politiques POMDP en temps contraint. Ce cadre est basé sur uneoptimisation anticipée et probabilisée des états d'exécution futurs du système. Nous avons embarqué ce cadrealgorithmique sur les hélicoptères autonomes de l'Onera, et l'avons testé en vol et en environnement réel sur une missionde détection et reconnaissance de cibles. / Mobile and aerial robots are faced to the need of planning actions with incomplete information about the state of theworld. In this context, this thesis proposes a modeling and resolution framework for perception and mission planningproblems where an autonomous helicopter must detect and recognize targets in an uncertain and partially observableenvironment. We founded our work on Partially Observable Markov Decision Processes (POMDPs), because it proposes ageneral optimization framework for perception and decision tasks under long-term horizon. A special attention is given tothe outputs of the image processing algorithm in order to model its uncertain behavior as a probabilistic observationfunction. A critical study on the POMDP model and its optimization criterion is also conducted. In order to respect safetyconstraints of aerial robots, we then propose an approach to properly handle action feasibility constraints in partiallyobservable domains: the AC-POMDP model, which distinguishes between the verification of environmental properties andthe information about targets' nature. Furthermore, we propose a framework to optimize and execute POMDP policies inparallel under time constraints. This framework is based on anticipated and probabilistic optimization of future executionstates of the system. Finally, we embedded this algorithmic framework on-board Onera's autonomous helicopters, andperformed real flight experiments for multi-target detection and recognition missions.