The security and resiliency of electric power supply to serve critical facilities are of high importance in today's world. Instead of building large electric power grids and high capacity transmission lines, an intelligent microgrid (or smart grid) can be considered as a promising power supply alternative. In recent years, multi-agent systems have been proposed to provide intelligent energy control and management systems in microgrids. Multi-agent systems offer their inherent benefits of flexibility, extensibility, autonomy, reduced maintenance and more. The implementation of a control network based on multi-agent systems that is capable of making intelligent decisions on behalf of the user has become an area of intense research. Many previous works have proposed multi-agent system architectures that deal with buying and selling of energy within a microgrid and algorithms for auction systems. The others proposed frameworks for multi-agent systems that could be further developed for real life control of microgrid systems. However, most proposed methods ignore the process of sharing energy resources among multiple distinct sets of prioritized loads. It is important to study a scenario that emphasizes on supporting critical loads during outages based on the user's preferences and limited capacity. The situation becomes further appealing when an excess DER capacity after supplying critical loads is allocated to support non-critical loads that belong to multiple users. The previous works also ignore the study of dynamic interactions between the agents and the physical systems. It is important to study the interaction and time delay when an agent issues a control signal to control a physical device in a microgrid and when the command is executed. Agents must be able to respond to the information sensed from the external environment quickly enough to manage the microgrid in a timely fashion. The ability of agents to disconnect the microgrid during emergencies should also be studied. These issues are identified as knowledge gaps that are of focus in this thesis. The objective of this research is to design, develop and implement a multi-agent system that enables real-time management of a microgrid. These include securing critical loads and supporting non-critical loads belonging to various owners with the distributed energy resource that has limited capacity during outages. The system under study consists of physical (microgrid) and cyber elements (multi-agent system). The cyber part or the multi-agent system is of primary focus of this work. The microgrid simulation has been implemented in Matlab/Simulink. It is a simplified distribution circuit that consists of one distributed energy resources (DER), loads and the main grid power supply. For the multi-agent system implementation, various open source agent building toolkits are compared to identify the most suitable agent toolkit for implementation in the proposed multi-agent system. The agent architecture is then designed by dividing overall goal of the system into several smaller tasks and assigning them to each agent. The implementation of multi-agent system was completed by identifying Roles (Role Modeling) and Responsibilities (Social and Domain Responsibilities) of agents in the system, and modeling the Knowledge (Facts), rules and ontology for the agents. Finally, both microgrid simulation and multi-agent system are connected together via TCP/IP using external java programming and a third party TCP server in the Matlab/Simulink environment. In summary, the multi-agent system is designed, developed and implemented in several simulation test cases. It is expected that this work will provide an insight into the design and development of a multi-agent system, as well as serving as a basis for practical implementation of an agent-based technology in a microgrid environment. Furthermore, the work also contributes to new design schemes to increase multi-agent system's intelligence. In particular, these include control algorithms for intelligently managing the limited supply from a DER during emergencies to secure critical loads, and at the same time supporting non-critical loads when the users need the most. / Master of Science

One of the major planning problems faced by medium and large manufacturing enterprises is the distribution of production over various (production) facilities. The need for cross-facility capacity management is most evident in the high-tech industries having capital-intensive equipment and short technology life cycle. There have been solutions proposed in the literature that are based on the lagragian decomposition method which separate the overall multiple product problem into a number of single product problems. We believe that multi-agent systems, given their distributed problem solving approach can be used to solve this problem, in its entirety, more effectively. According to other researchers who have worked in this field, auction theoretic mechanisms are a good way to solve complex production planning problems. This research study develops a multi-agent system and negotiation protocol based on combinatorial auction framework to solve the given multi-facility planning problem. The output of this research is a software library, which can be used as a multi-agent system model of the multi-product, multi-facility capacity allocation problem. The negotiation protocol for the agents is based on an iterative combinatorial auction framework which can be used for making allocation decisions in this environment in real-time. A simulator based on this library is created to validate the multi-agent model as well as the auction theoretic framework for different scenarios in the problem domain. The planning software library is created using open source standards so that it can be seamlessly integrated with scheduling library being developed as a part of the Advanced Planning and Scheduling (APS) system project or any other software suite which might require this functionality. The research contribution of this study is in terms of a new multi-agent architecture for an Advanced Planning and Control (APS) system as well as a novel iterative combinatorial auction mechanism which can be used as an agent negotiation protocol within this architecture. The theoretical concepts introduced by this research are implemented in the MultiPlanner production planning tool which can be used for generating master production plans for manufacturing enterprises. The validation process carried out on both the iterative combinatorial framework and the agent-based production planning methodology demonstrate that the proposed solution strategies can be used for integrated decision making in the multi-product, multi-facility production planning domain. Also, the software tool developed as part of this research is a robust, platform independent tool which can be used by manufacturing enterprises to make relevant production planning decisions. / Master of Science

In the rapidly advancing field of autonomous vehicles, ensuring the security and reliability of self-driving systems is crucial. Autonomous vehicle systems, such as cooperative adaptive cruise control (CACC), must undergo significant research and testing before their integration into commercial intelligent transportation systems. CACC considers multiple vehicles in close proximity as a single entity, or platoon, with each vehicle equipped with a controller that uses sensor-based measurements and vehicle-to-vehicle (V2V) communication to control inter-vehicle spacing. While this system offers numerous potential benefits for traffic safety and efficiency, it is also susceptible to False Data Injection (FDI) attacks, which can cause the system to behave in potentially life-threatening ways. Testing these scenarios in the real world is infeasible due to expense, safety concerns, and the use of theoretical technologies. This study presents an implementation of a vehicle platoon in a simulated environment where the vehicles' controllers were tuned to maintain desired inter-vehicle spacing. Various FDI signals were then implemented to demonstrate the feasibility of malicious attacks, including a novel parameterized sinusoidal FDI signal. Furthermore, acknowledging the necessity for future anomaly detection schemes and noise filtration, a theoretical optimal attack—generated using a model of the sinusoidal FDI attack and identification of optimal FDI values—was also evaluated.

Pursuit-evasion problems comprise a set of pursuers that strive to catch oneor several evaders, often in a constrained environment. This thesis proposesand compares heuristic algorithms for pursuit-evasion problems wherein several double integrator agents pursue a single evader in a bounded subset of theEuclidean plane. Different methods for assigning surrounding target points tothe pursuers are tested numerically. In addition, a method which finds the timeoptimal strategy for pursuing a static target in an unconstrained setting is presented, and is then used to pursue the assigned, dynamic, target. Numericalresults show that the time optimal strategy for pursuing a static target translateswell to the dynamic problem.

The application of reinforcement learning to dynamic industrial scheduling has gained increasing attention due to its capability to optimize complex manufacturing processes. With the advent of Industry 4.0 and the rise of smart manufacturing, new challenges arise that require innovative approaches, particularly in environments where there is a high degree of variability and uncertainty. Previous research has demonstrated that reinforcement learning, in particular Monte Carlo methods, is highly effective in optimizing resource allocation in job-shop scheduling scenarios. Even though Monte Carlo methods are effective where reward functions are clear and retrospective, real-world manufacturing systems often require more dynamic decision-making capabilities in real-time, for which temporaldifference methods are more appropriate. Despite the effectiveness of reinforcement learning in this area, there is a gap in understanding how different reward functions affect the learning process. In this study, we systematically examined multiple reward functions within a temporal difference system, applying a sensitivity analysis to assess their effects during the training and evaluation phases. Our results demonstrated that the overall performance of the production line improved despite the inherent complexity and challenges posed by temporal difference methods. Our findings demonstrate the effectiveness of multi-agent reinforcement learning for improving manufacturing efficiency, and provide implications for future research on scalable, real-time industrial scheduling.

In modern manufacturing environments, it is essential to be able to react autonomously and dynamically to unpredictable events in an automated manner in order to schedule production in a cost-effective manner. One of the prerequisites for the development of this technology is the progressive integration of cyberphysical systems into industrial sectors. Data generated by the industry constitutes the basis for operative and strategic decision-making in this context. Collecting these data in real time, transforming it if necessary, and analyzing it in order to ensure time-critical decision-making is a major challenge. This paper presents a novel approach that simplifies dynamic scheduling through efficient action selection. YoloRL, the method presented in this paper, which is based on reinforcement learning, which allows for a reduction in the complexity of the training process in a substantial way. For the purpose of identifying promising action sequences, YoloRL does not take into consideration all of the state information of an episode; it only takes into account the initial state. As a result, training complexity is significantly reduced while at the same time robust and adaptive control can be achieved. This study improves the manufacturing system’s performance by minimizing the overall completion time (for any given order). Experimental results indicate that the proposed method results in a faster generalization of the domain knowledge and provides for a powerful policy that is both efficient and reliable in dynamic environments. With YoloRL, overall completion time is reduced by a moderate but quantifiable amount compared with the traditional approach. In accordance with our experimental results, the proposed methodology has the ability to accelerate and stabilize the training process. Thus, a reliable and generalizable policy network is established, which can nevertheless respond dynamically to unforeseen events and changing environmental conditions due to its adaptability. The policy ...

Training autonomous agents that are capable of performing their assigned job without fail is the ultimate goal of deep reinforcement learning. This thesis introduces a dueling Quantile Regression Deep Q-network, where the network learns the state value quantile function and advantage quantile function separately. With this network architecture the agent is able to learn to control simulated robots in the Gazebo simulator. Carefully crafted reward functions and state spaces must be designed for the agent to learn in complex non-stationary environments. When trained for only 100,000 timesteps, the agent is able reach asymptotic performance in environments with moving and stationary obstacles using only the data from the inertial measurement unit, LIDAR, and positional information. Through the use of transfer learning, the agents are also capable of formation control and flocking patterns. The performance of agents with frozen networks is improved through advice giving in Deep Q-networks by use of normalized Q-values and majority voting.

This thesis addresses the problem of positional consensus of multi-agent systems. A positional consensus is achieved when the agents converge to a point. Some applications of this class of problem is in mid-air refueling of the aircraft or UAVs, targeting a geographical location, etc. In this research work some positional consensus algorithms have been developed. They can be categorized in two part (i) Broadcast control based algorithm (ii) Distributed control based algorithm. In case of broadcast based algorithm control strategies for a group of agents is developed to achieve positional consensus. The problem is constrained by the requirement that every agent must be given the same control input through a broadcast communication mechanism. Although the control command is computed using state information in a global framework, the control input is implemented by the agents in a local coordinate frame. The mathematical formulation has been done in a linear programming framework that is computationally less intensive than earlier proposed methods. Moreover, a random perturbation input in the control command, that helps to achieve reasonable proximity among agents even for a large number of agents, which was not possible with the existing strategy in the literature, is introduced. This method is extended to achieve positional consensus at a pre-specified location. A comparison between the LP approach and the existing SOCP based approach is also presented. Some of the algorithm has been demonstrated successfully on a robotic platform made from LEGO Mindstorms NXT Robots. In the second case of broadcast based algorithm, a decentralized algorithm for a group of multiple autonomous agents to achieve positional consensus has been developed using the broadcast concept. Even here, the mathematical formulation has done using a linear programming framework. Each agent has some sensing radius and it is capable of sensing position and orientation with other agents within their sensing region. The method is computationally feasible and easy to implement. In case of distributed algorithms, a computationally efficient distributed rendezvous algorithm for a group of autonomous agents has been developed. The algorithm uses a rectilinear decision domain (RDD), as against the circular decision domain assumed in earlier work available in the literature. This helps in reducing its computational complexity considerably. An extensive mathematical analysis has been carried out to prove the convergence of the algorithm. The algorithm has also been demonstrated successfully on a robotic platform made from LEGO Mindstorms NXT Robots.

Ce travail de thèse porte sur l'étude des systèmes complexes par une démarche de modélisation et simulation (M&S). La plupart des questionnements sur ces systèmes nécessitent de prendre en compte plusieurs points de vue simultanément. Il faut alors considérer des phénomènes évoluant à des échelles (temporelles et spatiales) et des niveaux de résolutions (de microscopique à macroscopique) différents. De plus, l'expertise nécessaire pour décrire le système vient en général de plusieurs domaines scientifiques. Les défis sont alors de concilier ces points de vues hétérogènes, et d'intégrer l'existant de chaque domaine (formalismes et logiciels de simulation) tout en restant dans le cadre rigoureux de la démarche de M&S. Pour répondre à ces défis, nous mobilisons à la fois des notions de modélisation multi-niveau (intégration de représentations micro/macro), de modélisation hybride (intégration de formalismes discrets/continus), de simulation parallèle, et d'ingénierie logicielle (interopérabilité logiciel, et ingénierie dirigée par les modèles). Nous nous inscrivons dans la continuité des travaux de M&S existants autour de l'approche AA4MM et du formalisme DEVS. Nous étudions en effet dans cette thèse en quoi ces approches sont complémentaires et permettent, une fois combinées dans une démarche d'Ingénierie Dirigée par les Modèles (IDM), de répondre aux défis de la M&S des systèmes complexes. Notre contribution est double. Nous proposons d'une part les spécifications opérationnelles de l'intergiciel de co-simulation MECSYCO permettant de simuler en parallèle un modèle de manière rigoureuse et complètement décentralisée. D'autre part, nous proposons une approche d'IDM permettant de décrire de manière non-ambiguë des modèles, puis de systématiser leur implémentation dans MECSYCO. Nous évaluons les propriétés de notre approche à travers plusieurs preuves de concept portant sur la M&S du trafic autoroutier et sur la résolution numérique d'un système d'équations différentielles / This thesis is focused on the study of complex systems through a modeling and simulation (M&S) process. Most questions about such systems requiere to take simultaneously account of several points of view. Phenomena evolving at different (temporal and spatial) scales and at different levels of resolution (from micro to macro) have to be considered. Moreover, several expert skills belonging to different scientific fields are needed. The challenges are then to reconcile these heterogeneous points of view, and to integrate each domain tools (formalisms and simulation software) within the rigorous framework of the M&S process. In order to solve these issues, we mobilise notions from multi-level modeling, hybrid modeling, parallel simulation and software engineering. Regarding these fields, we study the complementarity of the AA4MM approach and the DEVS formalism into the scope of the model-driven engineering (MDE) approach. Our contribution is twofold. We propose the operational specifications of the MECSYCO co-simulation middleware enabling the parallel simulation of complex systems models in a rigorous and decentralized way. We also define an MDE approach enabling the non-ambiguous description of complex systems models and their automatic implementation in MECSYCO. We show the properties of our approach with several proofs of concept

Submitted by PPG Ci?ncia da Computa??o (ppgcc@pucrs.br) on 2018-05-08T18:37:11Z No. of bitstreams: 1 RAFAEL_CAU?_CARDOSO_TES.pdf: 14431785 bytes, checksum: 227194ed28a9e55c3ab1fbedebf06922 (MD5) / Approved for entry into archive by Caroline Xavier (caroline.xavier@pucrs.br) on 2018-05-15T19:07:36Z (GMT) No. of bitstreams: 1 RAFAEL_CAU?_CARDOSO_TES.pdf: 14431785 bytes, checksum: 227194ed28a9e55c3ab1fbedebf06922 (MD5) / Made available in DSpace on 2018-05-15T19:14:18Z (GMT). No. of bitstreams: 1 RAFAEL_CAU?_CARDOSO_TES.pdf: 14431785 bytes, checksum: 227194ed28a9e55c3ab1fbedebf06922 (MD5) Previous issue date: 2018-03-27 / Sistemas multiagentes freq?entemente cont?m ambientes complexos e din?micos, nos quais os planos dos agentes podem falhar a qualquer momento durante a execu??o do sistema. Al?m disso, novos objetivos podem aparecer para os quais n?o existem nenhum plano dispon?vel. T?cnicas de planejamento s?o bem adequadas para lidar com esses problemas. H? uma quantidade extensa de pesquisa em planejamento centralizado para um ?nico agente, por?m, at? ent?o planejamento multiagente n?o foi completamente explorado na pr?tica. Plataformas multiagentes tipicamente proporcionam diversos mecanismos para coordena??o em tempo de execu??o, frequentemente necess?rios em planejamento online. Neste contexto, planejamento multiagente descentralizado pode ser eficiente e eficaz, especialmente em dom?nios fracamente acoplados, al?m de garantir algumas propriedades importantes em sistemas de agentes como privacidade e autonomia. N?s abordamos esse problema ao apresentar uma t?cnica para planejamento multiagente online que combina aloca??o de objetivos, planejamento individual utilizando rede de tarefas hier?rquicas (HTN), e coordena??o em tempo de execu??o para apoiar a realiza??o de objetivos sociais em sistemas multiagentes. Especificamente, n?s apresentamos um framework chamado Decentralised Online Multi-Agent Planning (DOMAP). Experimentos com tr?s dom?nios fracamente acoplados demonstram que DOMAP supera quatro planejadores multiagente do estado da arte com respeito a tempo de planejamento e tempo de execu??o, particularmente nos problemas mais dif?ceis. / Multi-agent systems often contain dynamic and complex environments where agents? course of action (plans) can fail at any moment during execution of the system. Furthermore, new goals can emerge for which there are no known plan available in any of the agents? plan library. Automated planning techniques are well suited to tackle both of these issues. Extensive research has been done in centralised planning for singleagents, however, so far multi-agent planning has not been fully explored in practice. Multi-agent platforms typically provide various mechanisms for runtime coordination, which are often required in online planning (i.e., planning during runtime). In this context, decentralised multi-agent planning can be efficient as well as effective, especially in loosely-coupled domains, besides also ensuring important properties in agent systems such as privacy and autonomy. We address this issue by putting forward an approach to online multi-agent planning that combines goal allocation, individual Hierarchical Task Network (HTN) planning, and coordination during runtime in order to support the achievement of social goals in multi-agent systems. In particular, we present a planning and execution framework called Decentralised Online Multi-Agent Planning (DOMAP). Experiments with three loosely-coupled planning domains show that DOMAP outperforms four other state-of-the-art multi agent planners with regards to both planning and execution time, particularly in the most difficult problems.