Este trabalho investiga a aplicação de rede de comportamentos estendidas ao domínio de jogos de computador. Redes de comportamentos estendidas (RCE) são uma classe de arquiteturas para seleção de ações capazes de selecionar bons conjuntos de ações para agentes complexos situados em ambientes contínuos e dinâmicos. Foram aplicadas com sucesso na Robocup, mas nunca foram aplicadas a jogos. PHISH-Nets, um modelo de redes de comportamentos capaz de selecionar apenas uma ação por vez, foi aplicado à modelagem de personagens, com bons resultados. Apesar de RCEs serem aplicáveis a um conjunto de domínios maior, nunca foram usadas para modelagem de personagens. Apresenta-se como projetar um agente controlado por uma rede de comportamentos para o domínio do Unreal Tournament e como integrar a rede de comportamentos a sensores nebulosos e comportamentos baseados em máquinas de estado-finito aumentadas. Investiga-se a qualidade da seleção de ações e a correção do mecanismo em uma série de experimentos. A performance é medida através da comparação das pontuações de um agente baseado em redes de comportamentos com outros dois agentes. Um dos agentes foi implementado por outro grupo e usava sensores, efetores e comportamentos diferentes. O outro agente era idêntico ao agente baseado em RCEs, exceto pelo mecanismo de controle empregado. A modelagem de personalidade é investigada através do projeto e análise de cinco estereótipos: Samurai, Veterano, Berserker, Novato e Covarde. Apresenta-se três maneiras de construir personalidades e situa-se este trabalho dentro de outras abordagems de projeto de personalidades. Conclui-se que a rede de comportamentos estendida é um bom mecanismo de seleção de ações para o domínio de jogos de computador e um mecanismo interessante para a construção de agentes com personalidades simples. / This work investigates the application of extended behavior networks to the computer game domain. We use as our test bed the game Unreal Tournament. Extended Behavior Networks (EBNs) are a class of action selection architectures capable of selecting a good set of actions for complex agents situated in continuous and dynamic environments. They have been successfully applied to the Robocup, but never before used in computer games. PHISH-Nets, a behavior network model capable of selecting just single actions, was applied to character modeling with promising results. Although extended behavior networks are applicable to a larger domain, they had not been used to character modeling before. We present how to design an agent with extended behavior networks, fuzzy sensors and finite-state machine based behaviors. We investigate the quality of the action selection mechanism and its correctness in a series of experiments. The performance is assessed comparing the scores of an agent using an extended behavior network against a plain reactive agent with identical sensory-motor apparatus and against a totally different agent built around finite-state machines. We investigate how EBNs fare on agent personality modeling via the design and analysis of five stereotypes in Unreal Tournament. We discuss three ways to build character personas and situate our work within other approaches. We conclude that extended behavior networks are a good action selection architecture for the computer game domain and an interesting mechanism to build agents with simple personalities.

Este trabalho investiga a aplicação de rede de comportamentos estendidas ao domínio de jogos de computador. Redes de comportamentos estendidas (RCE) são uma classe de arquiteturas para seleção de ações capazes de selecionar bons conjuntos de ações para agentes complexos situados em ambientes contínuos e dinâmicos. Foram aplicadas com sucesso na Robocup, mas nunca foram aplicadas a jogos. PHISH-Nets, um modelo de redes de comportamentos capaz de selecionar apenas uma ação por vez, foi aplicado à modelagem de personagens, com bons resultados. Apesar de RCEs serem aplicáveis a um conjunto de domínios maior, nunca foram usadas para modelagem de personagens. Apresenta-se como projetar um agente controlado por uma rede de comportamentos para o domínio do Unreal Tournament e como integrar a rede de comportamentos a sensores nebulosos e comportamentos baseados em máquinas de estado-finito aumentadas. Investiga-se a qualidade da seleção de ações e a correção do mecanismo em uma série de experimentos. A performance é medida através da comparação das pontuações de um agente baseado em redes de comportamentos com outros dois agentes. Um dos agentes foi implementado por outro grupo e usava sensores, efetores e comportamentos diferentes. O outro agente era idêntico ao agente baseado em RCEs, exceto pelo mecanismo de controle empregado. A modelagem de personalidade é investigada através do projeto e análise de cinco estereótipos: Samurai, Veterano, Berserker, Novato e Covarde. Apresenta-se três maneiras de construir personalidades e situa-se este trabalho dentro de outras abordagems de projeto de personalidades. Conclui-se que a rede de comportamentos estendida é um bom mecanismo de seleção de ações para o domínio de jogos de computador e um mecanismo interessante para a construção de agentes com personalidades simples. / This work investigates the application of extended behavior networks to the computer game domain. We use as our test bed the game Unreal Tournament. Extended Behavior Networks (EBNs) are a class of action selection architectures capable of selecting a good set of actions for complex agents situated in continuous and dynamic environments. They have been successfully applied to the Robocup, but never before used in computer games. PHISH-Nets, a behavior network model capable of selecting just single actions, was applied to character modeling with promising results. Although extended behavior networks are applicable to a larger domain, they had not been used to character modeling before. We present how to design an agent with extended behavior networks, fuzzy sensors and finite-state machine based behaviors. We investigate the quality of the action selection mechanism and its correctness in a series of experiments. The performance is assessed comparing the scores of an agent using an extended behavior network against a plain reactive agent with identical sensory-motor apparatus and against a totally different agent built around finite-state machines. We investigate how EBNs fare on agent personality modeling via the design and analysis of five stereotypes in Unreal Tournament. We discuss three ways to build character personas and situate our work within other approaches. We conclude that extended behavior networks are a good action selection architecture for the computer game domain and an interesting mechanism to build agents with simple personalities.

This master thesis investigates the principal capability of artificial evolution to produce tool use behavior in adaptive agents, excluding the application of life-time learning or adaptation mechanisms. Tool use is one aspect of complex behavior that is expected from autonomous agents acting in real-world environments. In order to achieve tool use behavior an agent needs to identify environmental objects as potential tools before it can use the tools in a problem-solving task. Up to now research in robotics has focused on life-time learning mechanisms in order to achieve this. However, these techniques impose great demands on resources, e.g. in terms of memory or computational power. All of them have shown limited results with respect to a general adaptivity. One might argue that even nature does not present any kind of omni-adaptive agent. While humans seem to be a good example of natural agents that master an impressive variety of life conditions and environments (at least from a human perspective, other examples are spectacular survivability observations of octopuses, scorpions or various viruses) even the most advanced engineering approaches can hardly compete with the simplest life-forms in terms of adaptation. This thesis tries to contribute to engineering approaches by promoting the application of artificial evolution as a complementing element with the presentation of successful pioneering experiments. The results of these experiments show that artificial evolution is indeed capable to render tool use behavior at different levels of complexity and shows that the application of artificial evolution might be a good complement to life-time approaches in order to create agents that are able to implicitly extract concepts and display tool use behavior. The author believes that off-loading at least parts of the concept retrieval process to artificial evolution will reduce resource efforts at life-time when creating autonomous agents with complex behavior such as tool use. This might be a first step towards the vision of a higher level of autonomy and adaptability. Moreover, it shows the demand for an experimental verification of commonly accepted limits between qualities of learned and evolved tool use capabilities.

Custom software development and maintenance is one of the key expenses associated with developing automated systems for mass customization. This paper presents a method for reducing the risk associated with this expense by developing a flexible environment for determining and executing dynamic workflow paths. Strategies for developing an autonomous agent-based framework and for identifying and creating web services for specific process tasks are presented. The proposed methods are outlined in two different case studies to illustrate the approach for both a generic process with complex workflow paths and a more specific sequential engineering process.

In the area of steering behaviours of autonomous agents and crowd simulations, there is a plethora of methods for executing the simulations. A very hard-to-achieve goal of crowd simulations is to make them seem natural and accurately reflect real-life crowds. A very important criterion for this goal is to have the agents avoid collisions, both with each other and with the environment. A less important, but important nonetheless, criterion is to not let the time taken or distance covered to reach the goal in the simulation be too high, compared with when not implementing collision avoidance. This paper proposes and explores a novel method of enhancing vector field-based steering with rule-based deviations to implement collision avoidance. This method is called ’DevVec’ (’Deviation + Vector Field steering’). The rules which are used for the deviations are extracted from a user survey, and they describe what the agent should do in different collision avoidance scenarios. The viability of DevVec is tested by comparing it with another already established method, called ’Gradient-based Steering’, in terms of fulfilling the criteria mentioned above. Both methods are used to simulate pedestrians moving throughout different scenes. The results suggest that DevVec has potential, but would require additional time and resources, and perhaps a few changes in future works to be presented in its best possible version. / Inom ämnesområdet för styrbeteenden hos autonoma agenter och simuleringar av folkmassor finns det många metoder för att framställa dessa simuleringar. Ett väldigt svåruppnåeligt mål för denna typ av simuleringar är få dem att verka naturliga och verklighetstrogna. Ett viktigt kriterie för detta mål är att få agenterna att undvika kollisioner, både med varandra och med den kringliggande omgivningen. Ett mindre viktigt, men viktigt oavsett, kriterie är att inte låta en agent ta för lång tid eller gå för långt för att nå sitt mål i simuleringen, i jämförelse med när de inte försöka undvika hinder. Denna studie presenterar och utforskar en ny metod som utökar en vektorfältsbaserat styralgoritm med regelbaserade avvikelser för att ta hänsyn till att undvika kollisioner. Denna nya metod kallas för ’DevVec’ (’Deviation + Vector Field steering’). Reglerna som används för avvikelserna är framtagna från en enkät, och de beskriver vad en agent borde göra vid olika kollision-scenarion. Användbarheten av DevVec prövas genom att jämföra den med en redan etablerad metod som kallas för ’Gradientbaserad styrning’, med avseende på de ovan nämnda kriterierna. Båda metoderna används för att simulera fotgängare i olika omgivningar. Resultaten antyder att DevVec har potential, men att det krävs ytterligare tid och resurser, och troligtvis några ändringar i framtiden för att framställa den bästa möjliga versionen.

<p dir="ltr">Learning, planning, and representing knowledge at multiple levels of temporal abstractions provide an agent with the ability to predict consequences of different courses of actions, which is essential for improving the performance of sequential decision making. However, discovering effective temporal abstractions, which the agent can use as skills, and adopting the constructed temporal abstractions for efficient policy learning can be challenging. Despite significant advancements in single-agent settings, temporal abstractions in multi-agent systems remains underexplored. This thesis addresses this research gap by introducing novel algorithms for discovering and employing temporal abstractions in both cooperative and competitive multi-agent environments. We first develop an unsupervised spectral-analysis-based discovery algorithm, aiming at finding temporal abstractions that can enhance the joint exploration of agents in complex, unknown environments for goal-achieving tasks. Subsequently, we propose a variational method that is applicable for a broader range of collaborative multi-agent tasks. This method unifies dynamic grouping and automatic multi-agent temporal abstraction discovery, and can be seamlessly integrated into the commonly-used multi-agent reinforcement learning algorithms. Further, for competitive multi-agent zero-sum games, we develop an algorithm based on Counterfactual Regret Minimization, which enables agents to form and utilize strategic abstractions akin to routine moves in chess during strategy learning, supported by solid theoretical and empirical analyses. Collectively, these contributions not only advance the understanding of multi-agent temporal abstractions but also present practical algorithms for intricate multi-agent challenges, including control, planning, and decision-making in complex scenarios.</p>

<p dir="ltr">We present a framework for Trustworthy Artificial Intelligence (TAI) that dynamically assesses trust and scrutinizes past decision-making, aiming to identify both individual and community behavior. The modeling of behavior incorporates proposed concepts, namely trust pressure and trust sensitivity, laying the foundation for predicting future decision-making regarding community behavior, consensus level, and decision-making duration. Our framework involves the development and mathematical modeling of trust pressure and trust sensitivity, drawing on social validation theory within the context of environmental decision-making. To substantiate our approach, we conduct experiments encompassing (i) dynamic trust sensitivity to reveal the impact of learning actors between decision-making, (ii) multi-level trust measurements to capture disruptive ratings, and (iii) different distributions of trust sensitivity to emphasize the significance of individual progress as well as overall progress.</p><p dir="ltr">Additionally, we introduce TAI metrics, trustworthy acceptance, and trustworthy fairness, designed to evaluate the acceptance of decisions proposed by AI or humans and the fairness of such proposed decisions. The dynamic trust management within the framework allows these TAI metrics to discern support for decisions among individuals with varying levels of trust. We propose both the metrics and their measurement methodology as contributions to the standardization of trustworthy AI.</p><p dir="ltr">Furthermore, our trustability metric incorporates reliability, resilience, and trust to evaluate systems with multiple components. We illustrate experiments showcasing the effects of different trust declines on the overall trustability of the system. Notably, we depict the trade-off between trustability and cost, resulting in net utility, which facilitates decision-making in systems and cloud security. This represents a pivotal step toward an artificial control model involving multiple agents engaged in negotiation.</p><p dir="ltr">Lastly, the dynamic management of trust and trustworthy acceptance, particularly in varying criteria, serves as a foundation for causal AI by providing inference methods. We outline a mechanism and present an experiment on human-driven causal inference, where participant discussions act as interventions, enabling counterfactual evaluations once actor and community behavior are modeled.</p>

<p dir="ltr">This dissertation presents decentralized and agent-based solutions for organizing machine learning resources, such as datasets and learning models. It aims to democratize the analysis of these resources through a simple yet flexible query structure, automate common ML tasks such as training, testing, model selection, and hyperparameter tuning, and enable privacy-centric building of ML models over distributed datasets. Based on networked multi-agent systems, the proposed approach represents ML resources as autonomous and self-reliant entities. This representation makes the resources easily movable, scalable, and independent of geographical locations, alleviating the need for centralized control and management units. Additionally, as all machine learning and data mining tasks are conducted near their resources, providers can apply customized rules independently of other parts of the system. </p><p><br></p>

This thesis addresses the problem of positional consensus of multi-agent systems. A positional consensus is achieved when the agents converge to a point. Some applications of this class of problem is in mid-air refueling of the aircraft or UAVs, targeting a geographical location, etc. In this research work some positional consensus algorithms have been developed. They can be categorized in two part (i) Broadcast control based algorithm (ii) Distributed control based algorithm. In case of broadcast based algorithm control strategies for a group of agents is developed to achieve positional consensus. The problem is constrained by the requirement that every agent must be given the same control input through a broadcast communication mechanism. Although the control command is computed using state information in a global framework, the control input is implemented by the agents in a local coordinate frame. The mathematical formulation has been done in a linear programming framework that is computationally less intensive than earlier proposed methods. Moreover, a random perturbation input in the control command, that helps to achieve reasonable proximity among agents even for a large number of agents, which was not possible with the existing strategy in the literature, is introduced. This method is extended to achieve positional consensus at a pre-specified location. A comparison between the LP approach and the existing SOCP based approach is also presented. Some of the algorithm has been demonstrated successfully on a robotic platform made from LEGO Mindstorms NXT Robots. In the second case of broadcast based algorithm, a decentralized algorithm for a group of multiple autonomous agents to achieve positional consensus has been developed using the broadcast concept. Even here, the mathematical formulation has done using a linear programming framework. Each agent has some sensing radius and it is capable of sensing position and orientation with other agents within their sensing region. The method is computationally feasible and easy to implement. In case of distributed algorithms, a computationally efficient distributed rendezvous algorithm for a group of autonomous agents has been developed. The algorithm uses a rectilinear decision domain (RDD), as against the circular decision domain assumed in earlier work available in the literature. This helps in reducing its computational complexity considerably. An extensive mathematical analysis has been carried out to prove the convergence of the algorithm. The algorithm has also been demonstrated successfully on a robotic platform made from LEGO Mindstorms NXT Robots.

Les robots opèrent dans le monde réel, dans lequel essayer quelque chose prend beaucoup de temps. Pourtant, les methodes d’apprentissage par renforcement actuels (par exemple, deep reinforcement learning) nécessitent de longues périodes d’interaction pour trouver des politiques efficaces. Dans cette thèse, nous avons exploré des algorithmes qui abordent le défi de l’apprentissage par essai-erreur en quelques minutes sur des robots physiques. Nous appelons ce défi “Apprentissage par renforcement micro-data”. Dans la première contribution, nous avons proposé un nouvel algorithme d’apprentissage appelé “Reset-free Trial-and-Error” qui permet aux robots complexes de s’adapter rapidement dans des circonstances inconnues (par exemple, des dommages) tout en accomplissant leurs tâches; en particulier, un robot hexapode endommagé a retrouvé la plupart de ses capacités de marche dans un environnement avec des obstacles, et sans aucune intervention humaine. Dans la deuxième contribution, nous avons proposé un nouvel algorithme de recherche de politique “basé modèle”, appelé Black-DROPS, qui: (1) n’impose aucune contrainte à la fonction de récompense ou à la politique, (2) est aussi efficace que les algorithmes de l’état de l’art, et (3) est aussi rapide que les approches analytiques lorsque plusieurs processeurs sont disponibles. Nous avons aussi proposé Multi-DEX, une extension qui s’inspire de l’algorithme “Novelty Search” et permet de résoudre plusieurs scénarios où les récompenses sont rares. Dans la troisième contribution, nous avons introduit une nouvelle procédure d’apprentissage du modèle dans Black-DROPS qui exploite un simulateur paramétré pour permettre d’apprendre des politiques sur des systèmes avec des espaces d’état de grande taille; par exemple, cette extension a trouvé des politiques performantes pour un robot hexapode (espace d’état 48D et d’action 18D) en moins d’une minute d’interaction. Enfin, nous avons exploré comment intégrer les contraintes de sécurité, améliorer la robustesse et tirer parti des multiple a priori en optimisation bayésienne. L'objectif de la thèse était de concevoir des méthodes qui fonctionnent sur des robots physiques (pas seulement en simulation). Par conséquent, tous nos approches ont été évaluées sur au moins un robot physique. Dans l’ensemble, nous proposons des méthodes qui permettre aux robots d’être plus autonomes et de pouvoir apprendre en poignée d’essais / Robots have to face the real world, in which trying something might take seconds, hours, or even days. Unfortunately, the current state-of-the-art reinforcement learning algorithms (e.g., deep reinforcement learning) require big interaction times to find effective policies. In this thesis, we explored approaches that tackle the challenge of learning by trial-and-error in a few minutes on physical robots. We call this challenge “micro-data reinforcement learning”. In our first contribution, we introduced a novel learning algorithm called “Reset-free Trial-and-Error” that allows complex robots to quickly recover from unknown circumstances (e.g., damages or different terrain) while completing their tasks and taking the environment into account; in particular, a physical damaged hexapod robot recovered most of its locomotion abilities in an environment with obstacles, and without any human intervention. In our second contribution, we introduced a novel model-based reinforcement learning algorithm, called Black-DROPS that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. We additionally proposed Multi-DEX, a model-based policy search approach, that takes inspiration from novelty-based ideas and effectively solved several sparse reward scenarios. In our third contribution, we introduced a new model learning procedure in Black-DROPS (we call it GP-MI) that leverages parameterized black-box priors to scale up to high-dimensional systems; for instance, it found high-performing walking policies for a physical damaged hexapod robot (48D state and 18D action space) in less than 1 minute of interaction time. Finally, in the last part of the thesis, we explored a few ideas on how to incorporate safety constraints, robustness and leverage multiple priors in Bayesian optimization in order to tackle the micro-data reinforcement learning challenge. Throughout this thesis, our goal was to design algorithms that work on physical robots, and not only in simulation. Consequently, all the proposed approaches have been evaluated on at least one physical robot. Overall, this thesis aimed at providing methods and algorithms that will allow physical robots to be more autonomous and be able to learn in a handful of trials