Throughout today’s society, increasingly more areas are being automated. Grocery stores however have been the same for years. Only recently, self-checkout counters and online shopping have been utilised in this business area. This thesis aims to take it to the next step by introducing automated grocery stores using a multi-agent system. Orders will be given to the system, and on a small area, multiple agents will pick the products in a time-efficient way and deliver them to the customer. This can both increase the throughput but also decrease the food waste and energy consumption of grocery stores. This thesis investigates already existing solutions for the multi-agent pickup and delivery problem. It extends these to the important case of batched tasks in order to improve the customer experience. Batches of tasks represent shopping carts, where fast completion of whole batches gives greater customer satisfaction. This notion is not mentioned in related work, where completion of single tasks is the main goal. Because of this, the existing solution does not accommodate the need of batches or the importance of completing whole batches fast and in somewhat linear order. For this purpose, a new metric called batch ordering weighted error (BOWE) was created that takes these factors into consideration. Using BOWE, one existing algorithm has been extended into prioritizing completing whole batches and is now called B-PIBT. This new algorithm has significantly improved BOWE and even batch service time for the algorithm in key cases and is now superior in comparison to the other state-of-the-art algorithms.

Cooperative multi-agent systems (MAS) are finding applications in a wide variety of domains, including sensor networks, robotics, distributed control, collaborative decision support systems, and data mining. A cooperative MAS consists of a group of autonomous agents that interact with one another in order to optimize a global performance measure. A central challenge in cooperative MAS research is to design distributed coordination policies. Designing optimal distributed coordination policies offline is usually not feasible for large-scale complex multi-agent systems, where 10s to 1000s of agents are involved, there is limited communication bandwidth and communication delay between agents, agents have only limited partial views of the whole system, etc. This infeasibility is either due to a prohibitive cost to build an accurate decision model, or a dynamically evolving environment, or the intractable computation complexity. This thesis develops a multi-agent reinforcement learning paradigm to allow agents to effectively learn and adapt coordination policies in complex cooperative domains without explicitly building the complete decision models. With multi-agent reinforcement learning (MARL), agents explore the environment through trial and error, adapt their behaviors to the dynamics of the uncertain and evolving environment, and improve their performance through experiences. To achieve the scalability of MARL and ensure the global performance, the MARL paradigm developed in this thesis restricts the learning of each agent to using information locally observed or received from local interactions with a limited number of agents (i.e., neighbors) in the system and exploits non-local interaction information to coordinate the learning processes of agents. This thesis develops new MARL algorithms for agents to learn effectively with limited observations in multi-agent settings and introduces a low-overhead supervisory control framework to collect and integrate non-local information into the learning process of agents to coordinate their learning. More specifically, the contributions of already completed aspects of this thesis are as follows: Multi-Agent Learning with Policy Prediction: This thesis introduces the concept of policy prediction and augments the basic gradient-based learning algorithm to achieve two properties: best-response learning and convergence. The convergence property of multi-agent learning with policy prediction is proven for a class of static games under the assumption of full observability. MARL Algorithm with Limited Observability: This thesis develops PGA-APP, a practical multi-agent learning algorithm that extends Q-learning to learn stochastic policies. PGA-APP combines the policy gradient technique with the idea of policy prediction. It allows an agent to learn effectively with limited observability in complex domains in presence of other learning agents. The empirical results demonstrate that PGA-APP outperforms state-of-the-art MARL techniques in both benchmark games. MARL Application in Cloud Computing: This thesis illustrates how MARL can be applied to optimizing online distributed resource allocation in cloud computing. Empirical results show that the MARL approach performs reasonably well, compared to an optimal solution, and better than a centralized myopic allocation approach in some cases. A General Paradigm for Coordinating MARL: This thesis presents a multi-level supervisory control framework to coordinate and guide the agents' learning process. This framework exploits non-local information and introduces a more global view to coordinate the learning process of individual agents without incurring significant overhead and exploding their policy space. Empirical results demonstrate that this coordination significantly improves the speed, quality and likelihood of MARL convergence in large-scale, complex cooperative multi-agent systems. An Agent Interaction Model: This thesis proposes a new general agent interaction model. This interaction model formalizes a type of interactions among agents, called {\em joint-even-driven} interactions, and define a measure for capturing the strength of such interactions. Formal analysis reveals the relationship between interactions between agents and the performance of individual agents and the whole system. Self-Organization for Nearly-Decomposable Hierarchy: This thesis develops a distributed self-organization approach, based on the agent interaction model, that dynamically form a nearly decomposable hierarchy for large-scale multi-agent systems. This self-organization approach is integrated into supervisory control framework to automatically evolving supervisory organizations to better coordinating MARL during the learning process. Empirically results show that dynamically evolving supervisory organizations can perform better than static ones. Automating Coordination for Multi-Agent Learning: We tailor our supervision framework for coordinating MARL in ND-POMDPs. By exploiting structured interaction in ND-POMDPs, this tailored approach distributes the learning of the global joint policy among supervisors and employs DCOP techniques to automatically coordinate distributed learning to ensure the global learning performance. We prove that this approach can learn a globally optimal policy for ND-POMDPs with a property called groupwise observability.

Past research on multi-agent simulation with cooperative reinforcement learning (RL) for homogeneous agents focuses on developing sharing strategies that are adopted and used by all agents in the environment. These sharing strategies are considered to be reciprocating because all participating agents have a predefined agreement regarding what type of information is shared, when it is shared, and how the participating agent's policies are subsequently updated. The sharing strategies are specifically designed around manipulating this shared information to improve learning performance. This thesis targets situations where the assumption of a single sharing strategy that is employed by all agents is not valid. This work seeks to address how agents with no predetermined sharing partners can exploit groups of cooperatively learning agents to improve learning performance when compared to Independent learning. Specifically, several intra-agent methods are proposed that do not assume a reciprocating sharing relationship and leverage the pre-existing agent interface associated with Q-Learning to expedite learning. The other agents' functions and their sharing strategies are unknown and inaccessible from the point of view of the agent(s) using the proposed methods. The proposed methods are evaluated on physically embodied agents in the multi-agent cooperative robotics field learning a navigation task via simulation. The experiments conducted focus on the effects of the following factors on the performance of the proposed non-reciprocating methods: scaling the number of agents in the environment, limiting the communication range of the agents, and scaling the size of the environment. / Master of Science

Tesis por compendio / [EN] Multi-Agent Systems (MAS), Argumentation and Automated Planning are three lines of investigations within the field of Artificial Intelligence (AI) that have been extensively studied over the last years. A MAS is a system composed of multiple intelligent agents that interact with each other and it is used to solve problems whose solution requires the presence of various functional and autonomous entities. Multi-agent systems can be used to solve problems that are difficult or impossible to resolve for an individual agent. On the other hand, Argumentation refers to the construction and subsequent exchange (iteratively) of arguments between a group of agents, with the aim of arguing for or against a particular proposal. Regarding Automated Planning, given an initial state of the world, a goal to achieve, and a set of possible actions, the goal is to build programs that can automatically calculate a plan to reach the final state from the initial state. The main objective of this thesis is to propose a model that combines and integrates these three research lines. More specifically, we consider a MAS as a team of agents with planning and argumentation capabilities. In that sense, given a planning problem with a set of objectives, (cooperative) agents jointly construct a plan to satisfy the objectives of the problem while they defeasibly reason about the environmental conditions so as to provide a stronger guarantee of success of the plan at execution time. Therefore, the goal is to use the planning knowledge to build a plan while agents beliefs about the impact of unexpected environmental conditions is used to select the plan which is less likely to fail at execution time. Thus, the system is intended to return collaborative plans that are more robust and adapted to the circumstances of the execution environment. In this thesis, we designed, built and evaluated a model of argumentation based on defeasible reasoning for planning cooperative multi-agent system. The designed system is independent of the domain, thus demonstrating the ability to solve problems in different application contexts. Specifically, the system has been tested in context sensitive domains such as Ambient Intelligence as well as with problems used in the International Planning Competitions. / [ES] Dentro de la Inteligencia Artificial (IA), existen tres ramas que han sido ampliamente estudiadas en los últimos años: Sistemas Multi-Agente (SMA), Argumentación y Planificación Automática. Un SMA es un sistema compuesto por múltiples agentes inteligentes que interactúan entre sí y se utilizan para resolver problemas cuya solución requiere la presencia de diversas entidades funcionales y autónomas. Los sistemas multiagente pueden ser utilizados para resolver problemas que son difíciles o imposibles de resolver para un agente individual. Por otra parte, la Argumentación consiste en la construcción y posterior intercambio (iterativamente) de argumentos entre un conjunto de agentes, con el objetivo de razonar a favor o en contra de una determinada propuesta. Con respecto a la Planificación Automática, dado un estado inicial del mundo, un objetivo a alcanzar, y un conjunto de acciones posibles, el objetivo es construir programas capaces de calcular de forma automática un plan que permita alcanzar el estado final a partir del estado inicial. El principal objetivo de esta tesis es proponer un modelo que combine e integre las tres líneas anteriores. Más específicamente, nosotros consideramos un SMA como un equipo de agentes con capacidades de planificación y argumentación. En ese sentido, dado un problema de planificación con un conjunto de objetivos, los agentes (cooperativos) construyen conjuntamente un plan para resolver los objetivos del problema y, al mismo tiempo, razonan sobre la viabilidad de los planes, utilizando como herramienta de diálogo la Argumentación. Por tanto, el objetivo no es sólo obtener automáticamente un plan solución generado de forma colaborativa entre los agentes, sino también utilizar las creencias de los agentes sobre la información del contexto para razonar acerca de la viabilidad de los planes en su futura etapa de ejecución. De esta forma, se pretende que el sistema sea capaz de devolver planes colaborativos más robustos y adaptados a las circunstancias del entorno de ejecución. En esta tesis se diseña, construye y evalúa un modelo de argumentación basado en razonamiento defeasible para un sistema de planificación cooperativa multiagente. El sistema diseñado es independiente del dominio, demostrando así la capacidad de resolver problemas en diferentes contextos de aplicación. Concretamente el sistema se ha evaluado en dominios sensibles al contexto como es la Inteligencia Ambiental y en problemas de las competiciones internacionales de planificación. / [CA] Dins de la intel·ligència artificial (IA), hi han tres branques que han sigut àmpliament estudiades en els últims anys: Sistemes Multi-Agent (SMA), Argumentació i Planificació Automàtica. Un SMA es un sistema compost per múltiples agents intel·ligents que interactúen entre si i s'utilitzen per a resoldre problemas la solución dels quals requereix la presència de diverses entitats funcionals i autònomes. Els sistemes multiagente poden ser utilitzats per a resoldre problemes que són difícils o impossibles de resoldre per a un agent individual. D'altra banda, l'Argumentació consistiex en la construcció i posterior intercanvi (iterativament) d'arguments entre un conjunt d'agents, amb l'objectiu de raonar a favor o en contra d'una determinada proposta. Respecte a la Planificació Automàtica, donat un estat inicial del món, un objectiu a aconseguir, i un conjunt d'accions possibles, l'objectiu és construir programes capaços de calcular de forma automàtica un pla que permeta aconseguir l'estat final a partir de l'estat inicial. El principal objectiu d'aquesta tesi és proposar un model que combine i integre les tres línies anteriors. Més específicament, nosaltres considerem un SMA com un equip d'agents amb capacitats de planificació i argumentació. En aquest sentit, donat un problema de planificació amb un conjunt d'objectius, els agents (cooperatius) construeixen conjuntament un pla per a resoldre els objectius del problema i, al mateix temps, raonen sobre la viabilitat dels plans, utilitzant com a ferramenta de diàleg l'Argumentació. Per tant, l'objectiu no és només obtindre automàticament un pla solució generat de forma col·laborativa entre els agents, sinó també utilitzar les creences dels agents sobre la informació del context per a raonar sobre la viabilitat dels plans en la seua futura etapa d'execució. D'aquesta manera, es pretén que el sistema siga capaç de tornar plans col·laboratius més robustos i adaptats a les circumstàncies de l'entorn d'execució. En aquesta tesi es dissenya, construeix i avalua un model d'argumentació basat en raonament defeasible per a un sistema de planificació cooperativa multiagent. El sistema dissenyat és independent del domini, demostrant així la capacitat de resoldre problemes en diferents contextos d'aplicació. Concretament el sistema s'ha avaluat en dominis sensibles al context com és la inte·ligència Ambiental i en problemes de les competicions internacionals de planificació. / Pajares Ferrando, S. (2016). Defeasible Argumentation for Cooperative Multi-Agent Planning [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/60159 / Compendio

Cette thèse présente PARDISIM, un modèle de simulation qui propose uneapproche économique institutionnelle pour la simulation du processus dedéveloppement résidentiel. Plutôt que de modéliser le développement résidentielcomme le résultat de choix de localisation au niveau des ménages, PARDISIMmet l’accent sur les objectifs et les interactions des acteurs du développementrésidentiel. L’idée est que les acteurs du développement, y compris les au-torités publiques d’aménagement, jouent un rôle important dans le processusde développement résidentiel. L’approche est donc top-down et se démarquedes approches habituelles bottom-up. Les premiers résultats obtenus montrentque PARDISIM est capable de produire des configurations spatiales réalistes. / This thesis presents PARDISIM, a simulation model that takes an institutionaleconomic approach in the simulation of the residential development process.Rather then modelling the residential development as the result of locationchoices at household level, PARDISIM focusses on the objectives and interac-tions of development actors. The idea behind this approach is that developmentactors, including public planning authorities, play an important role in the pro-cess of residential development. The model is top-down whereas the most recentefforts by other scholars focus instead on a bottom-up approach. Initial testingshows that PARDISIM is capable of producing realistic spatial patterns. / Dit proefschrift presenteert PARDISIM, een simulatiemodel dat een institu-tioneel economische benadering toepast in de simulatie van de ontwikkeling vanwoningbouw. In plaats van deze ontwikkeling te definiëren als gevolg van locatiekeuzes op huishoudniveau, richt PARDISIM zich op de doelstellingen en de in-teracties van actoren met een professioneel belang in de stedelijke ontwikkeling.Het idee achter deze aanpak is dat deze actoren, waaronder ondermeer localeoverheden, een belangrijke rol spelen in het proces van de ontwikkeling vanwoningbouw. Het model is top-down terwijl de meest recente inspanningen inde literatuur een bottom-up benadering toepassen. Uit de eerste testresultatenblijkt dat PARDISIM in staat is realistische, ruimtelijke configuraties te pro-duceren.

Published Article / Radio-frequency Identification (RFID) technology promises to revolutionize business processes. While RFID technology is improving rapidly, a reliable deployment of this technology is still a significant challenge impeding its widespread adoption. In this paper we provide a brief overview of some common fundamental characteristics of RFID data and devices, which pose significant challenges in the design of RFID middleware systems. In addition, the development of a multi-agent RFID middleware solution to address the RFID data and device management challenges is discussed.

This thesis undertakes to provide an architecture and understanding of the incorporation of trust into the paradigm of mobile multi-agent systems. Trust deliberation is a soft security approach to the problem of mobile agent security whereby an agent is protected from the malicious behaviour of others within the system. Using a trust approach capitalises on observing malicious behaviour rather than preventing it. We adopt an architectural approach to trust such than we do not provide a model in itself, numerous mathematical models for the calculation of trust based on a history of observations already exist. Rather we look to provide the framework enabling such models to be utilised by mobile agents. As trust is subjective we envisage a system whereby individual agents will use different trust models or different weighting mechanisms. Three architectures are provided. Centralised whereby the platform itself provides all of the services needed by an agent to make observations and calculate trust. Decentralised in which each individual agent is responsible for making observations, communicating trust and the calculation of its own trust in others. A hybrid architecture such that trust mechanisms are provided by the platform and additionally are embedded within the agents themselves. As an optimisation of the architectures proposed in this thesis, we introduce the notion of trust communities. A community is used as a means to represent the trust information in categorisations dependant upon various properties. Optimisation occurs in two ways; firstly with subjective communities and secondly with system communities. A customised implementation framework of the architectures is introduced in the form of our TEMPLE (Trust Enabled Mobile-agent PLatform Environment) and stands as the underpinning of a case-study implementation in order to provide empirical evidence in the form of scenario test-bed data as to the effectiveness of each architecture. The case study chosen for use in a trust based system is that of a fish market' as given the number of interactions, entities, and migration of agents involved in the system thus, providing substantial output data based upon the trust decisions made by agents. Hence, a good indicator of the effectiveness of equipping agents with trust ability using our architectures.

Agent technology enables the designers of computer based systems to construct software agents that are able to model attitudes. We present a frame-work in which these artifacts aim to express the preferences of both their designers and their owners. Just like human societies need rules, regula-tions, norms and social laws, in order to function, societies of agents need coordination mechanisms in order to work efficiently. We show why some higher level goals of agents are incompatible, e.g. the automatic creation of coalitions among agents, and at the same time being self-interested and boundedly rational. One way to model the outcome of planned interactions between agents is to apply game theory. We use game theory for proving some results, e.g. a \No free lunch" theorem. For more practical applications, however, other approaches are often needed. One such domain is dynamic resource allocation, where agents through auction mechanisms or different kinds of mobile broker techniques solve the problem of coordinating the allocation. We present comparisons of the results of simulations of several of these approaches in a telecommunication networks application. Another interesting domain concerns mobile robots for playing soccer. To model this problem, a novel approach called artificial electrical fields, is used for both navigation and manipulation of objects. / Agentteknologin möjliggör design av mjukvaruagenter som kan representera åsikter. Vi presenterar ett ramverk i vilket både agenternas designrar, såväl som ägare, kan uttrycka sina preferenser. Precis som i verkligheten, där mänskliga samhällen behöver regler och lagar för att fungera, så behöver agenterna normer och koordineringsmekanismer för att fungera effektivt. Vi visar varför några av högnivåmålen i multi-agentsystem är motstridiga, tex rationalitet och förmåga att bygga koalitioner. Ett sätt att modellera interaktioner mellan agenter är att använda spelteori. Vi använder spelteori bland annat för att visa ett "No free lunch"-teorem för agentsystem, men i praktiska tillämpningar, så behöver vi ofta använda andra angreppssätt. En sådan problemdomän är dynamisk resursallokering i telekommunikationssystem, i vilken vi simulerat koordineringar mellan agenter för att lösa problemet. Vi presenterar resultaten av simuleringar av ett flertal olika arkitekturer, bland annat mobila mäklar-agenter och auktionsagenter. En ytterligare domän är robotfotboll till vilken vi utvecklat en heuristik för val av handlingar baserad på artificiella elektroniska fält.

Multi-agent systems are applied to distributed problem-solving applications because of their ability to overcome the limitations that individual agents face when solving complex problems. Large numbers of agents acting as problem-solvers on networks suggest a virtual marketplace. In this marketplace, groups of self-interested agents can interact to solve highly constrained and distributed problems by assuming varying roles and forming “temporary teams”. This dissertation presents a decision making mechanism for multi-agent team formation between self-interested agents in a competitive, open and dynamic environment. An agent perceives environmental uncertainties, and models those uncertainties into simplified categories such as risks and benefits. The dissertation further demonstrates how an agent’s attitudes shape how risk and rewards are weighted when making decisions among multiple alternatives. Accordingly, agent-borne attitudes toward proactive behavior, risk, reward, and urgency are proposed as the basis of the proposed team formation mechanism. Finally, a learning technique assists an agent in continuously learning what attitudes it needs in order to adapt to dynamic environments and increase its resulting rewards. / text

The research described in this thesis stems from ROBOCARE1, a three year research project aimed at developing software and robotic technology for providing intelligent support for elderly people. This thesis deals with two problems which have emerged in the course of the project’s development: Multi-agent coordination with scarce resources. Multi-agent planning is concerned with automatically devising plans or strategies for the coordinated enactment of concurrently executing agents. A common realistic constraint in applications which require the coordination of multiple agents is the scarcity of resources for execution. In these cases, concurrency is affected by limited capacity resources, the presence of which modifies the structure of the planning/coordination problem. Specifically, the first part of this thesis tackles this problem in two contexts, namely when planning is carried out centrally (planning from first principles), and in the context of distributed multi-agent coordination. Domain modeling for scheduling applications. It is often the case that the products of research in AI problem solving are employed to develop applications for supporting human decision processes. Our experience in ROBOCARE as well as other domains has often called for the customization of prototypical software for real applications. Yet the gap between what is often a research prototype and a complete decision support system is seldom easy to bridge.The second part of the thesis focuses on this issue from the point of view of scheduling software deployment.Overall, this thesis presents three contributions within the two problems mentioned above. First, we address the issue of planning in concurrent domains in which the complexity of coordination is dominated by resource constraints. To this end, an integrated planning and scheduling architecture is presented and employed to explore the structural trademarks of multi-agent coordination problems in function of their resource-related characteristics. Theoretical and experimental analyses are carried out revealing which planning strategies are most fit for achieving plans which prescribe efficient coordination subject to scarce resources.We then turn our attention to distributed multi-agent coordination techniques (specifically, a distributed constraint optimization (DCOP) reduction of the coordination problem). Again, we consider the issue of achieving coordinated action in the presence of limited resources. Specifically, resource constraints impose n-ary relations among tasks. In addition, as the number of n-ary relations due to resource contention are exponential in the size of the problem, they cannot be extensionally represented in the DCOP representation of the coordination problem. Thus, we propose an algorithm for DCOP which retains the capability to dynamically post n-ary constraints during problem resolution in order to guarantee resource-feasible solutions. Although the approach is motivated by the multi-agent coordination problem, the algorithm is employed to realize a general architecture for n-ary constraint reasoning and posting.Third, we focus on a somewhat separate issue stemming from ROBOCARE, namely a software engineering methodology for facilitating the process of customizing scheduling components in real-world applications. This work is motivated by the strong applicative requirements of ROBOCARE. We propose a software engineering methodology specific to scheduling technology development. Our experience in ROBOCARE as well as other application scenarios has fostered the development of a modeling framework which subsumes the process of component customization for scheduling applications. The framework aims to minimize the effort involved in deploying automated reasoning technology in practise, and is grounded on the use of a modeling language for defining how domain-level concepts are grounded into elements of a technology-specific scheduling ontology.