Systems consisting of multiple robots are traditionallydifficult to optimize. This project considers such a systemin a simulated warehouse setting, where the robots are todeliver boxes while avoiding collisions. Adding such collisionconstraints complicates the problem. For dynamical multi-agentsystems as these, reinforcement learning algorithms are oftenappropriate. We explore and implement a reinforcement learningalgorithm, called multi-agent rollout, that allows for re-planningduring operation. The algorithm is paired with a base policyof following the shortest path. Simulation results with up to10 robots indicates that the algorithm is promising for largescalemulti-robot systems. We have also discussed the possibilityof using neural networks and partitioning to further increaseperformance. / System med flera robotar har traditionellt sett ansetts mycket svåra att optimera. I detta projekt undersöks ett sådant system i en simulerad lagerlokal, där robotarna skall förflytta lådor samtidigt som de undviker kollisioner. För dessa dynamiska system med flera robotar är förstärkande inlärning ofta lämpligt. Vi undersöker och implementerar en förstärkandeinlärningsalgoritm kallad ”multi-agent rollout” vilken möjliggör omdirigering under drift. Algoritmen används tillsammans med en så kallad ”base policy” som alltid väljer kortaste vägen. Baserat på simulationsresultaten med upp till tio robotar verkar algoritmen lovande för storskaliga flerrobotsystem. Det diskuteras även om möjligheten av att använda neurala nätverk och partitionering för att vidare öka prestandan. / Kandidatexjobb i elektroteknik 2021, KTH, Stockholm

A major challenge in the transportation industry is how to reduce the emissions of greenhouse gases. One way of achieving this in vehicles is to drive more fuel-efficiently. One recently developed technique that has been successful in reducing the fuel consumption is the look-ahead cruise controller, which utilizes future conditions such as road topography. In this this thesis, similar methods are used in order to reduce the fuel consumption of heavy-duty vehicles driving in environments where the required and desired velocity vary. The main focus is on vehicles in urban driving, which must alter their velocity due to, for instance, changing legal speed restrictions and the presence of intersections. The driving missions of such vehicles are here formulated as optimal control problems. In order to restrict the vehicle to drive in a way that does not deviate too much from a normal way of driving, constraints on the velocity are imposed based on statistics from real truck operation. In a first approach, the vehicle model is based on forces and the cost function involves the consumed energy. This problem is solved both offline using Pontryagin's maximum principle and online using a model predictive controller with a quadratic program formulation. Simulations show that 7 % energy can be saved without increasing the trip time nor deviating from a normal way of driving. In a second approach, the vehicle model is extended to include an engine and a gearbox with the objective of minimizing the fuel consumption. A fuel map for the engine and a polynomial function for the gearbox losses are extracted from experimental data and used in the model. This problem is solved using dynamic programming taking into consideration gear changes, coasting with gear and coasting in neutral. Simulations show that by allowing the use of coasting in neutral gear, 13 % fuel can be saved without increasing the trip time or deviating from a normal way of driving. Finally, an implementation of a rule-based controller into an advanced vehicle model in highway driving is performed. The controller identifies sections of downhills where fuel can be saved by coasting in neutral gear. / En stor utmaning för transportsektorn är hur utsläppen av växthusgaser ska minskas. Detta kan åstadkommas i fordon genom att köra bränslesnålare. En nyligen utvecklad teknik som har varit framgångsrik i att minska bränsleförbrukningen är framförhållningsreglering, som använder framtida förhållanden så som vägtopografi. I denna avhandling används liknande metoder för att minska bränsleförbrukningen i tunga fordon som kör i miljöer där önskad och tvingad hastighet varierar. Fokus ligger framförallt på fordon i stadskörning, där hastigheten måste varieras beroende på bland annat hastighetsbegränsningar och korsningar. Denna typ av körning formuleras här som optimala reglerproblem. För att hindra fordonet från att avvika för mycket från ett normalt körbeteende sätts begränsningar på tillåten hastighet baserat på statistik från verklig körning. Problemet angrips först genom att använda en fordonsmodell baserad på krafter och en kriteriefunktion innehållande energiförbrukning. Problemet löses både offline med Pontryagin's maximum princip och online med modellprediktiv reglering baserad på kvadratisk programmering. Simuleringar visar att 7 % energi kan sparas utan att öka körtiden eller avvika från ett normalt körbeteende. Problemet angrips sedan genom att utöka fordonsmodellen till att också innehålla motor och växellåda med målet att minimera bränsleförbrukningen. Specifik bränsleförbrukning och en polynomisk approximation av förlusterna i växellådan är extraherade från experiment och används i simuleringarna. Problemet löses genom dynamisk programmering som tar hänsyn till växling, släpning och frirullning. Simuleringar visar att 13 % bränsle kan sparas utan att öka körtid eller avvika från normalt körbeteende genom att tillåta frirullning. Slutligen görs en implementering av en regelbaserad regulator på en avancerad fordonsmodell för ett fordon i motorvägskörning. Regulatorn identifierar sektioner med nedförsbackar där bränsle kan sparas genom frirulllning. / <p>QC 20171011</p>

Game theory is the study of tactical interactions involving conflicts and cooperations among multiple decision makers called players with applications in diverse disciplines such as economics, biology, management, communication networks, electric power systems and control. This dissertation studies a statistical differential game problem where finite N players optimize their system performance by shaping the distribution of their cost function through cost cumulants. This research integrates game theory with statistical optimal control theory and considers a statistical Nash non-cooperative nonzero-sum game for a nonlinear dynamic system with nonquadratic cost functions. The objective of the statistical Nash game is to find the equilibrium solution where no player has the incentive to deviate once other players maintain their equilibrium strategy. The necessary condition for the existence of the Nash equilibrium solution is given for the m-th cumulant cost optimization using the Hamilton-Jacobi-Bellman (HJB) equations. In addition, the sufficient condition which is the verification theorem for the existence of Nash equilibrium solution is given for the m-th cumulant cost optimization using the Hamilton-Jacobi-Bellman (HJB) equations. However, solving the HJB equations even for relatively low dimensional game problem is not trivial, we propose to use neural network approximate method to find the solution of the HJB partial differential equations for the statistical game problem. Convergence proof of the neural network approximate method solution to exact solution is given. In addition, numerical examples are provided for the statistical game to demonstrate the applicability of the proposed theoretical developments. / Electrical and Computer Engineering

In an optimal visiting problem, we want to control a trajectory that has to pass as close as possible to a collection of target points or regions. We introduce a hybrid control-based approach for the classic problem where the trajectory can switch between a group of discrete states related to the targets of the problem. The model is subsequently adapted to a mean-field game framework, that is when a huge population of agents plays the optimal visiting problem with a controlled dynamics and with costs also depending on the distribution of the population. In particular, we investigate a single continuity equation with possible sinks and sources and the field possibly depending on the mass of the agents. The same problem is also studied on a network framework. More precisely, we study a mean-field game model by proving the existence of a suitable definition of an approximated mean-field equilibrium and then we address the passage to the limit.

Agent-based models are computer models made up of agents that can exist in a finite number of states. The state of the system at any given time is determined by rules governing agents' interaction. The rules may be deterministic or stochastic. Optimization is the process of finding a solution that optimizes some value that is determined by simulating the model. Optimal control of an agent-based model is the process of determining a sequence of control inputs to the model that steer the system to a desired state in the most efficient way. In large and complex models, the number of possible control inputs is too large to be enumerated by computers; hence methods must be developed for use with these models in order to find solutions without searching the entire solution space. Heuristic algorithms have been applied to such models with some success. Such algorithms are discussed; case studies of examples from biology are presented. The lack of a standard format for agent-based models is a major issue facing the study of agent-based models; presentation as polynomial dynamical systems is presented as a viable option. Algorithms are adapted and presented for use in this framework. / Master of Science

In this thesis, we consider an optimal stopping problem interpreted as the task of valuating two so called real options written on an underlying asset following the dynamics of an observable geometric Brownian motion with non-observable drift; we have incomplete information. After exercising the first real option, however, the value of the underlying asset becomes observable with reduced noise; we obtain partial information. We then state some theoretical properties of the value function such as convexity and monotonicity. Furthermore, numerical solutions for the value functions are obtained by stating and solving a linear complementary problem. This is done in a Python implementation using the 2nd order backward differentiation formula and summation-by-parts operators for finite differences combined with an operator splitting method.

Heterogeneity, influenced by diverse factors such as age, gender, immunity, behavior, and spatial distribution, plays a critical role in the dynamics of infectious disease transmission. Discrete mathematical structures, including matrices and graphs, can offer effective tools for modeling the interactions among these diverse factors, resulting heterogeneous epidemiological models. This dissertation explores analytical approaches, specifically utilizing eigenvalues and eigenvectors of discrete structures, to characterize heterogeneity within mathematical models of infectious diseases. Theoretical results, along with numerical simulations, enhance our understanding of heterogeneous epidemiological processes and their significant implications for disease control strategies. In this dissertation, we introduce a unified approach to establish the final size formula in heterogeneous epidemic models, based on a new concept of “total infectious contacts” as an eigenvector-based aggregation of disease compartments. This approach allows us to identify the peak of total infectious contacts, offering a novel method to pinpoint the turning point of a disease outbreak. Furthermore, we examine spatial heterogeneity through two distinct mathematical frameworks: the Lagrangian and Eulerian models. The Lagrangian model assesses the epidemiological consequences of spatio-temporal residence time matrices, while the Eulerian model investigates “Turing instability” as a new underlying mechanism for spatial heterogeneity observed in disease prevalence data.

Résumé : Depuis la fin du XXème siècle, l’augmentation du prix du pétrole brut et les problématiques environnementales poussent l’industrie automobile à développer des technologies plus économes en carburant et générant moins d’émissions de gaz à effet de serre. Parmi ces technologies, les véhicules électriques hybrides constituent une solution viable et performante. En alliant un moteur électrique et un moteur à combustion, ces véhicules possèdent un fort potentiel de réduction de la consommation de carburant sans sacrifier son autonomie. La présence de deux moteurs et de deux sources d’énergie requiert un contrôleur, appelé système de gestion d’énergie, responsable de la commande simultanée des deux moteurs. Les performances du véhicule en matière de consommation dépendent en partie de la conception de ce contrôleur. Les véhicules électriques hybrides rechargeables, plus récents que leur équivalent non rechargeable, se distinguent par l’ajout d’un chargeur interne permettant la recharge de la batterie pendant l’arrêt du véhicule et par conséquent la décharge de celle-ci au cours d’un trajet. Cette particularité ajoute un degré de complexité pour ce qui est de la conception du système de gestion d’énergie. Dans cette thèse, nous proposons un modèle complet du véhicule dédié à la conception du contrôleur. Nous étudions ensuite la dépendance de la commande optimale des deux moteurs par rapport au profil de vitesse suivi au cours d’un trajet ainsi qu’à la quantité d’énergie électrique disponible au début d’un trajet. Cela nous amène à proposer une technique d’auto-apprentissage visant l’amélioration de la stratégie de gestion d’énergie en exploitant un certain nombre de données enregistrées sur les trajets antérieurs. La technique proposée permet l’adaptation de la stratégie de contrôle vis-à-vis du trajet en cours en se basant sur une pseudo-prédiction de la totalité du profil de vitesse. Nous évaluerons les performances de la technique proposée en matière de consommation de carburant en la comparant avec une stratégie optimale bénéficiant de la connaissance exacte du profil de vitesse ainsi qu’avec une stratégie de base utilisée couramment dans l’industrie. // Abstract : Since the end of the XXth century, the increase in crude oil price and the environmental concerns lead the automotive industry to develop technologies that can improve fuel savings and decrease greenhouse gases emissions. Among these technologies, the hybrid electric vehicles stand as a reliable and efficient solution. By combining an electrical motor and an internal combustion engine, these vehicles can bring a noticeable improvement in terms of fuel consumption without sacrificing the vehicle autonomy. The two motors and the two energy storage systems require a control unit, called energy management system, which is responsible for the command decision of both motors. The vehicle performances in terms of fuel consumption greatly depend on this control unit. The plug-in hybrid electric vehicles are a more recent technology compared to their non plug-in counterparts. They have an extra internal battery charger that allows the battery to be charged during OFF state, implying a possible discharge during a trip. This particularity adds complexity when it comes to the design of the energy management system. In this thesis, a complete vehicle model is proposed and used for the design of the controller. A study is then carried out to show the dependence between the optimal control of the motors and the speed profile followed during a trip as well as the available electrical energy at the beginning of a trip. According to this study, a self-learning optimization technique that aims at improving the energy management strategy by exploiting some driving data recorded on previous trips is proposed. The technique allows the adaptation of the control strategy to the current trip based on a pseudo-prediction of the total speed profile. Fuel consumption performances for the proposed technique will be evaluated by comparing it with an optimal control strategy that benefits from the exact a priori knowledge of the speed profile as well as a basic strategy commonly used in industry.

Achieving robust and reliable autonomous operation even in complex unstructured environments is a central goal of field robotics. As the environments and scenarios to which robots are applied have continued to grow in complexity, so has the challenge of properly defining preferences and tradeoffs between various actions and the terrains they result in traversing. These definitions and parameters encode the desired behavior of the robot; therefore their correctness is of the utmost importance. Current manual approaches to creating and adjusting these preference models and cost functions have proven to be incredibly tedious and time-consuming, while typically not producing optimal results except in the simplest of circumstances. This thesis presents the development and application of machine learning techniques that automate the construction and tuning of preference models within complex mobile robotic systems. Utilizing the framework of inverse optimal control, expert examples of robot behavior can be used to construct models that generalize demonstrated preferences and reproduce similar behavior. Novel learning from demonstration approaches are developed that offer the possibility of significantly reducing the amount of human interaction necessary to tune a system, while also improving its final performance. Techniques to account for the inevitability of noisy and imperfect demonstration are presented, along with additional methods for improving the efficiency of expert demonstration and feedback. The effectiveness of these approaches is confirmed through application to several real world domains, such as the interpretation of static and dynamic perceptual data in unstructured environments and the learning of human driving styles and maneuver preferences. Extensive testing and experimentation both in simulation and in the field with multiple mobile robotic systems provides empirical confirmation of superior autonomous performance, with less expert interaction and no hand tuning. These experiments validate the potential applicability of the developed algorithms to a large variety of future mobile robotic systems.

In this thesis, we develop preconditioned iterative methods for the solution of matrix systems arising from PDE-constrained optimization problems. In order to do this, we exploit saddle point theory, as this is the form of the matrix systems we wish to solve. We utilize well-known results on saddle point systems to motivate preconditioners based on effective approximations of the (1,1)-block and Schur complement of the matrices involved. These preconditioners are used in conjunction with suitable iterative solvers, which include MINRES, non-standard Conjugate Gradients, GMRES and BiCG. The solvers we use are selected based on the particular problem and preconditioning strategy employed. We consider the numerical solution of a range of PDE-constrained optimization problems, namely the distributed control, Neumann boundary control and subdomain control of Poisson's equation, convection-diffusion control, Stokes and Navier-Stokes control, the optimal control of the heat equation, and the optimal control of reaction-diffusion problems arising in chemical processes. Each of these problems has a special structure which we make use of when developing our preconditioners, and specific techniques and approximations are required for each problem. In each case, we motivate and derive our preconditioners, obtain eigenvalue bounds for the preconditioners where relevant, and demonstrate the effectiveness of our strategies through numerical experiments. The goal throughout this work is for our iterative solvers to be feasible and reliable, but also robust with respect to the parameters involved in the problems we consider.