<p>Autonomous systems are often deployed in dynamic environments and are challenged with unexpected changes (novelties) in the environments where they receive novel data that was not seen during training. Given the uncertainty, they should be able to operate without (or with limited) human intervention and they are expected to (1) Adapt to such changes while still being effective and efficient in performing their multiple tasks. The system should be able to provide continuous availability of its critical functionalities. (2) Make informed decisions independently from any central authority. (3) Be Cognitive: learns the new context, its possible actions, and be rich in knowledge discovery through mining and pattern recognition. (4) Be Reflexive: reacts to novel unknown data as well as to security threats without terminating on-going critical missions. These characteristics combine to create the workflow of autonomous decision-making process in multi-agent environments (i.e.,) any action taken by the system must go through these characteristic models to autonomously make an ideal decision based on the situation. </p> <p><br></p> <p>In this dissertation, we propose novel learning-based models to enhance the decision-making process in autonomous multi-agent systems where agents are able to detect novelties (i.e., unexpected changes in the environment), and adapt to it in a timely manner. For this purpose, we explore two complex and highly dynamic domains </p> <p>(1) Transportation Networks (e.g., Ridesharing application): where we develop AdaPool: a novel distributed diurnal-adaptive decision-making framework for multi-agent autonomous vehicles using model-free deep reinforcement learning and change point detection. (2) Multi-agent games (e.g., Monopoly): for which we propose a hybrid approach that combines deep reinforcement learning (for frequent but complex decisions) with a fixed-policy approach (for infrequent but straightforward decisions) to facilitate decision-making and it is also adaptive to novelties. (3) Further, we present a domain agnostic approach for decision making without prior knowledge in dynamic environments using Bootstrapped DQN. Finally, to enhance security of autonomous multi-agent systems, (4) we develop a machine learning based resilience testing of address randomization moving target defense. Additionally, to further  improve the decision-making process, we present (5) a novel framework for multi-agent deep covering option discovery that is designed to accelerate exploration (which is the first step of decision-making for autonomous agents), by identifying potential collaborative agents and encouraging visiting the under-represented states in their joint observation space. </p>

Formation control in multi-agent systems represents a groundbreaking intersection of various research fields with lots of emerging applications in various technologies. The realm of space exploration also can benefit significantly from formation control, facilitating a wide range of functions from astronomical observations, and climate monitoring to enhancing telecommunications, and on-orbit servicing and assembly. In this thesis, we present a novel 3D formation control scheme for directed graphs in a leader-follower configuration, achieving (almost) global convergence to the desired shape. Specifically, we introduce three controlled variables representing bispherical coordinates that uniquely describe the formation in 3D. Acyclic triangulated directed graphs (a class of minimally acyclic persistent graphs) are used to model the inter-agent sensing topology, while the agents’ dynamics are governed by the single-integrator model and 2nd order nonlinear version representing spacecraft formation flight. The analysis demonstrates that the proposed decentralized robust formation controller using prescribed performance control ensures (almost) global asymptotic stability while avoiding potential shape ambiguities in the final formation. Furthermore, the control laws are implementable in arbitrarily oriented local coordinate frames of follower agents using only low-cost onboard vision sensors, making them suitable for practical applications. Formation maneuvering and collision avoidance among agents are also addressed which play crucial roles in the safety of space operations. Finally, we validate our formation control approach by simulation studies. / Formationskontroll i system med flera agenter representerar en banbrytande skärningspunkt av olika forskningsområden med massor av nya tillämpningar inom olika teknologier. Rymdutforskningens rike kan också dra stor nytta av formationskontroll, underlättar ett brett utbud av funktioner från astronomiska observationer och klimat övervakning för att förbättra telekommunikation och service och montering i omloppsbana. I denna avhandling presenterar vi ett nytt 3D-formationskontrollschema för riktade grafer i en ledare-följare-konfiguration, vilket uppnår (nästan) global konvergens till önskad form. Specifikt introducerar vi tre kontrollerade variabler som representerar bisfäriska koordinater som unikt beskriver formationen i 3D. Acykliska triangulerade riktade grafer (en klass av minimalt acykliska beständiga grafer) används för att modellera avkänningstopologin mellan agenter, medan agenternas dynamik styrs av singelintegratormodellen och 2:a ordningen olinjär version som representerar rymdfarkostbildningsflygning. Analysen visar att den föreslagna decentraliserade robusta formationskontrollanten använder föreskriven prestanda kontroll säkerställer (nästan) global asymptotisk stabilitet samtidigt som potentiell form undviks oklarheter i den slutliga formationen. Dessutom är kontrolllagarna implementerbara i godtyckligt orienterade lokala koordinatramar för efterföljare som endast använder lågkostnad ombord visionsensorer, vilket gör dem lämpliga för praktiska tillämpningar. Formationsmanövrering och undvikande av kollisioner mellan agenter tas också upp som spelar avgörande roller i säkerheten vid rymdoperationer. Slutligen validerar vi vår strategi för formningskontroll genom simuleringsstudier

[pt] O compartilhamento e o gerenciamento de dados pessoais são atividades desafiadoras devido à grande quantidade de dados gerados, carregados e digitalizados por cidadãos para utilizar serviços, online ou não. Esse desafio afeta não apenas os cidadãos, mas também os controladores e processadores de dados, que são responsáveis pela segurança, privacidade, anonimato e uso de dados fundados em bases legais e no propósito inicial quando os dados foram solicitados. Nesse cenário, a proteção e regulamentação dedados entram em cena para organizar esse ambiente, propondo direitos e deveres aos agentes envolvidos. No entanto, cada país é livre para criar e empregar sua própria regulamentação de dados, como o GDPR na União Europeia e a LGPD no Brasil. Portanto, embora o objetivo seja proteger os cidadãos, as regulamentações podem apresentar regras diferentes com base em sua jurisdição. Nesse cenário, as ontologias surgem para identificar as entidades e relacionamentos e mostrá-los em um nível de abstração elevado, facilitando o alinhamento das ontologias com diferentes regulamentações. Para isso, desenvolvemos um meta modelo baseado em ontologias da GDPR para possibilitar a representação da LGPD com foco na base legal do consentimento. Além disso, propusemos o GoDReP (Geraçãod e Cenários de Regulamentação de Dados) para permitir que os atores representem a interpretação de sua legislação em um cenário de aplicação específico. Apresentamos então três cenários diferentes para exercitar a aplicação do GoDReP. Além disso, nesta tese, também propomos uma arquitetura de sistema multiagente normativo e inteligente (RegulAI) para representar os direitos e obrigações apresentados pela regulamentação de dados pessoais, bem como o processo de tomada de decisão dos agentes.Por fim, desenvolvemos um estudo de caso aplicando o RegulAI no cenário de open banking. / [en] Sharing and managing personal data are challenging due to the massive amount of data generated, uploaded, and digitalized, informed by data subjects to utilize services, online or not. This challenge disrespects not only the data subjects, but also data controllers and processors, which are responsible for security, privacy, anonymity, and data usage under the legal basis applied and the initial purpose when the data were required. In this scenario, data protection and regulation take place to organize this environment proposing rights and duties to the involved agents. However, each country is free to create and employ its data regulation, e.g., GDPR in European Union and LGPD in Brazil. Therefore, although the goal is to protect the data subjects, the regulations can present different rules based on their jurisdiction. In this scenario, ontologies emerge to identify the entities and relationships to show them at a high abstraction level, facilitating ontology alignment with different regulations. To do so, we developed a metamodel based on GDPR ontologies to enable the LGPD representation focused on the consent legal basis. Moreover, we proposed GoDReP (Generation of Data Regulation Plots) to allow actors to represent their law s interpretation in a specific application scenario. As a result, we set three scenarios to exercise the GoDReP application. Moreover, in this thesis, we also propose an intelligent normative multiagent system architecture (RegulAI) to represent the personal data regulation rights and obligations, as well as the agent s decision-making process. Finally, we developed a use case applying RegulAI in the open banking scenario.

<p dir="ltr">Current artificial intelligence (AI) systems are proficient at tasks in a closed-world setting where the rules are often rigid. However, in real-world applications, the environment is usually open and dynamic. In this work, we investigate the effects of such dynamic environments on AI systems and develop ways to mitigate those effects. Central to our exploration is the concept of \textit{novelties}. Novelties encompass structural changes, unanticipated events, and environmental shifts that can confound traditional AI systems. We categorize novelties based on their representation, anticipation, and impact on agents, laying the groundwork for systematic detection and adaptation strategies. We explore novelties in the context of stochastic games. Decision-making in stochastic games exercises many aspects of the same reasoning capabilities needed by AI agents acting in the real world. A multi-agent stochastic game allows for infinitely many ways to introduce novelty. We propose an extension of the deep reinforcement learning (DRL) paradigm to develop agents that can detect and adapt to novelties in these environments. To address the sample efficiency challenge in DRL, we introduce a hybrid approach that combines fixed-policy methods with traditional DRL techniques, offering enhanced performance in complex decision-making tasks. We present a novel method for detecting anticipated novelties in multi-agent games, leveraging information theory to discern patterns indicative of collusion among players. Finally, we introduce DABLER, a pioneering deep reinforcement learning architecture that dynamically adapts to changing environmental conditions through broad learning approaches and environment recognition. Our findings underscore the importance of developing AI systems equipped to navigate the uncertainties of the open world, offering promising pathways for advancing AI research and application in real-world settings.</p>

Les systèmes complexes, naturels et artificiels, ont reçu récemment une attention renouvelée : les systèmes naturels, notamment biologiques, du fait de la nécessité de les appréhender dans une démarche systémique ; les systèmes artificiels, du fait de la dématérialisation de l'ordinateur amorcée avec l'informatique ubiquitaire. L'art, de son côté, explore depuis toujours le détournement des dernières avancées scientifiques et technologiques pour la création d'oeuvres singulières. Le travail mené depuis dix ans se situe à la croisée de ces chemins, dans le cadre unificateur des systèmes multi-agents. Je me suis intéressé plus particulièrement à l'interaction homme-machine dans le contexte de l'informatique ambiante, dans l'idée d'une construction automatique et d'une régulation dynamique de systèmes d'interaction. Ce travail est alimenté par la recherche menée dans le cadre de la simulation à base d'agents, aussi bien du point de vue des concepts et outils développés, que du point de vue de l'inspiration tirée des mécanismes d'auto-organisation et de régulation des systèmes étudiés. L'art fournit quant à lui un cadre expérimental original par la mise en scène métaphorique, dans des performances numériques interactives, des situations étudiées.

Cette thèse s'inscrit dans le cadre d'un projet de recherche et de développement qui vise à mettre en place des technologies de simulation permettant de reproduire des comportements humains dans une ville. L'objectif de nos travaux est de définir des algorithmes permettant de simuler les déplacements d'une grande quantité de piétons dans un environnement urbain, en temps réel, et de manière crédible. Pour ce type d'exercice, plusieurs solutions existent. Ces solutions sont principalement développées à partir de deux types d'approches : les approches microscopiques, où les piétons sont modélisés comme des agents autonomes, et les approches macroscopiques, où les piétons sont considérés comme soumis à des lois d'écoulement continues ou discrètes. Notre position est que ces deux approches ne s'opposent pas, contrairement à ce qui ressort de la pratique courante, mais se complètent mutuellement. Privilégier l'une au détriment de l'autre fait courir le risque de produire des solutions partiellement satisfaisantes. Aussi nous sommes nous proposés de clarifier le cadre formel permettant d'appréhender la complexité des déplacements. En ligne avec plusieurs études statistiques et psychologiques sur le déplacement des piétons, nous explicitons un déplacement crédible comme un déplacement économe en énergie métabolique. Nous nous inspirons des jeux de congestion et du paradigme multi-agent pour proposer une formulation générique du problème de déplacement des piétons : nous introduisons la notion de ressources de navigation, que nous décrivons comme des régions de l'espace que les agents utilisent pour atteindre leurs destinations, et via lesquelles les agents interagissent pour estimer leurs dépenses énergétiques de manière robuste. Nous proposons une stratégie de déplacement basée sur les heuristiques taboues et nous considérons le principe influence et réaction pour implémenter les actions de déplacements. Le concept d'environnement issu du paradigme multi-agent s'avère particulièrement utile pour appréhender la complexité de la simulation. L'environnement est considéré comme un composant indépendant et ontologiquement différent des agents qui est pris en compte à tous les niveaux de décisions. Une importante partie de la dynamique de la simulation peut ainsi être déléguée à l'environnement sans altérer l'autonomie des agents. Cette séparation favorise à la fois la crédibilité des résultats et le passage à l'échelle. Nous avons choisi de comparer notre proposition avec un modèle microscopique standard à travers plusieurs scénarios de simulation. Il ressort de notre comparaison que notre modèle permet de reproduire des résultats plus crédibles du point de vue d'un observateur extérieur et plus proches des études empiriques connues sur les déplacements des piétons.

The demand for automated decision making, learning and inference in uncertain, risk sensitive and dynamically changing situations presents a challenge: to design computational approaches that promise to be widely deployable and flexible to adapt on the one hand, while offering reliable guarantees on safety on the other. The tension between these desiderata has created a gap that, in spite of intensive research and contributions made from a wide range of communities, remains to be filled. This represents an intriguing challenge that provided motivation for much of the work presented in this thesis. With these desiderata in mind, this thesis makes a number of contributions towards the development of algorithms for automated decision-making and inference under uncertainty. To facilitate inference over unobserved effects of actions, we develop machine learning approaches that are suitable for the construction of models over dynamical laws that provide uncertainty bounds around their predictions. As an example application for conservative decision-making, we apply our learning and inference methods to control in uncertain dynamical systems. Owing to the uncertainty bounds, we can derive performance guarantees of the resulting learning-based controllers. Furthermore, our simulations demonstrate that the resulting decision-making algorithms are effective in learning and controlling under uncertain dynamics and can outperform alternative methods. Another set of contributions is made in multi-agent decision-making which we cast in the general framework of optimisation with interaction constraints. The constraints necessitate coordination, for which we develop several methods. As a particularly challenging application domain, our exposition focusses on collision avoidance. Here we consider coordination both in discrete-time and continuous-time dynamical systems. In the continuous-time case, inference is required to ensure that decisions are made that avoid collisions with adjustably high certainty even when computation is inevitably finite. In both discrete-time and finite-time settings, we introduce conservative decision-making. That is, even with finite computation, a coordination outcome is guaranteed to satisfy collision-avoidance constraints with adjustably high confidence relative to the current uncertain model. Our methods are illustrated in simulations in the context of collision avoidance in graphs, multi-commodity flow problems, distributed stochastic model-predictive control, as well as in collision-prediction and avoidance in stochastic differential systems. Finally, we provide an example of how to combine some of our different methods into a multi-agent predictive controller that coordinates learning agents with uncertain beliefs over their dynamics. Utilising the guarantees established for our learning algorithms, the resulting mechanism can provide collision avoidance guarantees relative to the a posteriori epistemic beliefs over the agents' dynamics.

Esquemas de Rejeição de Cargas (RC) por subfrequência, utilizados para manter a frequência de operação de um Sistema Elétrico de Potência (SEP) próxima ao seu valor nominal, precisam ser criteriosamente projetados a fim de diminuir os riscos de colapso generalizado do sistema. Entretanto, pelos métodos convencionais, a quantidade de carga a ser rejeitada não leva em consideração a dinamicidade intrínseca do sistema, sendo baseada em conjecturas estáticas sobre porções do SEP. Como resultado, a redução da carga geralmente não é eficiente, gerando rejeição insuficiente ou excessiva. Neste cenário, este trabalho propõe um novo esquema para o controle da frequência em comparação aos processos de RC usualmente empregados. Com o propósito de superar as limitações e melhorar as principais funções desses processos é proposto um Sistema Multi Agentes (SMA) centralizado que irá coordenar as diversas etapas de monitoramento, processamento e tomada de decisão nos barramentos disponíveis para corte em situações de subfrequência. Busca-se dessa forma, desconectar o menor montante de cargas do sistema, por um curto espaço de tempo e com menor perturbação da frequência. Neste sentido, uma malha de controle fechada foi desenvolvida a partir da simulação de um sistema elétrico de potência completo via o Real Time Digital Simulator (RTDS). O SMA foi embarcado em um sistema integrado de hardware e software em tempo real para teste e validação da metodologia proposta. No contexto delineado, uma métrica de avaliação foi proposta para comparar o método proposto com outras duas filosofias convencionais de RC. Os resultados obtidos permitem evidenciar o bom desempenho do SMA frente às duas filosofias convencionais, principalmente no que se refere ao montante de carga a ser rejeitado, com boa aproximação do valor esperado. / Automatic Under Frequency Load Shedding (AUFLS) schemes, used to maintain the frequency of an electric power system close to the nominal value, need to be carefully designed in order to reduce the risk of a widespread system collapse. However, the conventional methods do not take into account the inherent dynamics of an electric system and they are based on static assumptions. As a result, the shedding is generally not efficient, causing insufficient or excessive load discontinuity. In this scenario, this work proposes a new scheme for controlling the frequency compared to the AUFLS processes usually employed. In order to overcome the limitations of the methods usually employed and to improve the main functions of the AUFLS schemes, this work proposes a centralized MultiAgent System (MAS) that will coordinate the various stages of the monitoring and decision making process. The MAS seeks to disconnect a minimum amount of loads, in a short period of time and with less disturbance of the system frequency. A Hardware in Loop (HIL) configuration was developed from the simulation of a full electric system using the Real Time Digital Simulator (RTDS). The MAS was embedded in a real time system, consisting of hardware and software to test and validate the proposed methodology. In addition, a scoring metric evaluation is defined in order to compare other two conventional AUFLS philosophies. The results show good performance of the proposed MAS. The shedding was carried out in a single step and the amount of load shed was very close to the expected value.

Made available in DSpace on 2016-03-22T17:26:22Z (GMT). No. of bitstreams: 1 dm2_Luciano_vargas.pdf: 637463 bytes, checksum: b08b63e8c6a347cd2c86fc24fdfd8986 (MD5) Previous issue date: 2009-03-31 / The study and development of systems to control interactions in multiagent systems is an open problem in Artificial Intelligence. The system of social exchange values of Piaget is a social approach that allows for the foundations of the modeling of interactions between agents, where the interactions are seen as service exchanges between pairs of agents, with the evaluation of the realized or received services, thats is, the investments and profits in the exchange, and credits and debits to be charged or received, respectively, in future exchanges. This evaluation may be performed in different ways by the agents, considering that they may have different exchange personality traits. In an exchange process along the time, the different ways in the evaluation of profits and losses may cause disequilibrium in the exchange balances, where some agents may accumulate profits and others accumulate losses. To solve the exchange equilibrium problem, we use the Partially Observable Markov Decision Processes (POMDP) to help the agent decision of actions that can lead to the equilibrium of the social exchanges. Then, each agent has its own internal process to evaluate its current balance of the results of the exchange process between the other agents, observing its internal state, and with the observation of its partner s exchange behavior, it is able to deliberate on the best action it should perform in order to get the equilibrium of the exchanges. Considering an open multiagent system, it is necessary a mechanism to recognize the different personality traits, to build the POMDPs to manage the exchanges between the pairs of agents. This recognizing task is done by Hidden Markov Models (HMM), which, from models of known personality traits, can approximate the personality traits of the new partners, just by analyzing observations done on the agent behaviors in exchanges. The aim of this work is to develop an hybrid agent architecture for the self-regulation of social exchanges between personalitybased agents in a open multiagent system, based in the BDI (Beliefs, Desires, Intentions) architecture, where the agent plans are obtained from optimal policies of POMDPs, which model personality traits that are recognized by HMMs. To evaluate the proposed approach some simulations were done considering (known or new) different personality traits / O estudo e desenvolvimento de sistemas para o controle de interações em sistemas multiagentes é um tema em aberto dentro da Inteligência Artificial. O sistema de valores de trocas sociais de Piaget é uma abordagem social que possibilita fundamentar a modelagem de interações de agentes, onde as interações são vistas como trocas de serviços entre pares de agentes, com a valorização dos serviços realizados e recebidos, ou seja, investimentos e ganhos na troca realizada, e, também os créditos e débitos a serem cobrados ou recebidos, respectivamente, em trocas futuras. Esta avaliação pode ser realizada de maneira diferenciada pelos agentes envolvidos, considerando que estes apresentam traços de personalidade distintos. No decorrer de processo de trocas sociais a forma diferenciada de avaliar os ganhos e perdas nas interações pode causar desequilíbrio nos balanços de trocas dos agentes, onde alguns agentes acumulam ganhos e outros acumulam perdas. Para resolver a questão do equilíbrio das trocas, encontrou-se nos Processos de Decisão de Markov Parcialmente Observáveis (POMDP) uma metodologia capaz de auxiliar a tomada de decisões de cursos de ações na busca do equilíbrio interno dos agentes. Assim, cada agente conta com um mecanismo próprio para avaliar o seu estado interno, e, de posse das observações sobre o comportamento de troca dos parceiros, torna-se apto para deliberar sobre as melhores ações a seguir na busca do equilíbrio interno para o par de agentes. Com objetivo de operar em sistema multiagentes aberto, torna-se necessário um mecanismo para reconhecer os diferentes traços de personalidade, viabilizando o uso de POMDPs nestes ambientes. Esta tarefa de reconhecimento é desempenhada pelos Modelos de Estados Ocultos de Markov (HMM), que, a partir de modelos de traços de personalidade conhecidos, podem inferir os traços aproximados de novos parceiros de interações, através das observações sobre seus comportamentos nas trocas. O objetivo deste trabalho é desenvolver uma arquitetura de agentes híbrida para a auto-regulação de trocas sociais entre agentes baseados em traços de personalidade em sistemas multiagentes abertos. A arquitetura proposta é baseada na arquitetura BDI (Beliefs, Desires, Intentions), onde os planos dos agentes são obtidos através de políticas ótimas de POMDPs, que modelam traços de personalidade reconhecidos através de HMMs. Para avaliar a proposta, foram realizadas simulações envolvendo traços de personalidade conhecidos e novos traços

Esquemas de Rejeição de Cargas (RC) por subfrequência, utilizados para manter a frequência de operação de um Sistema Elétrico de Potência (SEP) próxima ao seu valor nominal, precisam ser criteriosamente projetados a fim de diminuir os riscos de colapso generalizado do sistema. Entretanto, pelos métodos convencionais, a quantidade de carga a ser rejeitada não leva em consideração a dinamicidade intrínseca do sistema, sendo baseada em conjecturas estáticas sobre porções do SEP. Como resultado, a redução da carga geralmente não é eficiente, gerando rejeição insuficiente ou excessiva. Neste cenário, este trabalho propõe um novo esquema para o controle da frequência em comparação aos processos de RC usualmente empregados. Com o propósito de superar as limitações e melhorar as principais funções desses processos é proposto um Sistema Multi Agentes (SMA) centralizado que irá coordenar as diversas etapas de monitoramento, processamento e tomada de decisão nos barramentos disponíveis para corte em situações de subfrequência. Busca-se dessa forma, desconectar o menor montante de cargas do sistema, por um curto espaço de tempo e com menor perturbação da frequência. Neste sentido, uma malha de controle fechada foi desenvolvida a partir da simulação de um sistema elétrico de potência completo via o Real Time Digital Simulator (RTDS). O SMA foi embarcado em um sistema integrado de hardware e software em tempo real para teste e validação da metodologia proposta. No contexto delineado, uma métrica de avaliação foi proposta para comparar o método proposto com outras duas filosofias convencionais de RC. Os resultados obtidos permitem evidenciar o bom desempenho do SMA frente às duas filosofias convencionais, principalmente no que se refere ao montante de carga a ser rejeitado, com boa aproximação do valor esperado. / Automatic Under Frequency Load Shedding (AUFLS) schemes, used to maintain the frequency of an electric power system close to the nominal value, need to be carefully designed in order to reduce the risk of a widespread system collapse. However, the conventional methods do not take into account the inherent dynamics of an electric system and they are based on static assumptions. As a result, the shedding is generally not efficient, causing insufficient or excessive load discontinuity. In this scenario, this work proposes a new scheme for controlling the frequency compared to the AUFLS processes usually employed. In order to overcome the limitations of the methods usually employed and to improve the main functions of the AUFLS schemes, this work proposes a centralized MultiAgent System (MAS) that will coordinate the various stages of the monitoring and decision making process. The MAS seeks to disconnect a minimum amount of loads, in a short period of time and with less disturbance of the system frequency. A Hardware in Loop (HIL) configuration was developed from the simulation of a full electric system using the Real Time Digital Simulator (RTDS). The MAS was embedded in a real time system, consisting of hardware and software to test and validate the proposed methodology. In addition, a scoring metric evaluation is defined in order to compare other two conventional AUFLS philosophies. The results show good performance of the proposed MAS. The shedding was carried out in a single step and the amount of load shed was very close to the expected value.