Hunter-Prey or Prey-Pursuit problem is a common toy domain for Reinforcement Learning, but the size of the state space is exponential in the parameters such as size of the grid or number of agents. As the size of the state space makes the flat Q-learning impossible to use for different scenarios, this thesis presents an approach to make the size of the state space constant by producing agents that use previously learned knowledge to perform on bigger scenarios containing more agents. Inspired from HRL methods, the method is composed of a parallel subtasks schema dividing the task into choices of simpler subtasks, a state representation technique convenient for this schema and its extension for bigger grids. Experimental results show that proposed method successfully provides agents that perform near to hand-coded agents by using constant sized state space independent from parameters of the domain.

A very general framework for modeling uncertainty in learning environments is given by Partially Observable Markov Decision Processes (POMDPs). In a POMDP setting, the learning agent infers a policy for acting optimally in all possible states of the environment, while receiving only observations of these states. The basic idea for coping with partial observability is to include memory into the representation of the policy. Perfect memory is provided by the belief space, i.e. the space of probability distributions over environmental states. However, computing policies defined on the belief space requires a considerable amount of prior knowledge about the learning problem and is expensive in terms of computation time. In this thesis, we present a reinforcement learning algorithm for solving deterministic POMDPs based on short-term memory. Short-term memory is implemented by sequences of past observations and actions which are called history lists. In contrast to belief states, history lists are not capable of representing optimal policies, but are far more practical and require no prior knowledge about the learning problem. The algorithm presented learns policies consisting of two separate phases. During the first phase, the learning agent collects information by actively establishing a history list identifying the current state. This phase is called the efficient identification strategy. After the current state has been determined, the Q-Learning algorithm is used to learn a near optimal policy. We show that such a procedure can be also used to solve large Markov Decision Processes (MDPs). Solving MDPs with continuous, multi-dimensional state spaces requires some form of abstraction over states. One particular way of establishing such abstraction is to ignore the original state information, only considering features of states. This form of state abstraction is closely related to POMDPs, since features of states can be interpreted as observations of states.

Cette thèse est organisée en deux chapitres. Le premier chapitre sert d’introduction aux concepts et idées utilisés dans le deuxième chapitre (l’article). Le premier chapitre est divisé en trois sections. Dans la première section, nous introduisons l’apprentissage par renforcement en tant que paradigme d’apprentissage automatique et montrons comment ses problèmes sont formalisés à l’aide de processus décisionnels de Markov. Nous formalisons les buts sous forme de rendements attendus et montrons comment les équations de Bellman utilisent la formulation récursive du rendement pour établir une relation entre les valeurs de deux états successifs sous la politique de l’agent. Après cela, nous soutenons que la résolution des équations d’optimalité de Bellman est insoluble et introduisons des algorithmes basés sur des valeurs tels que la programmation dynamique, les méthodes de Monte Carlo et les méthodes de différence temporelle qui se rapprochent de la solution optimale à l’aide de l’itération de politique généralisée. L’approximation de fonctions est ensuite proposée comme moyen de traiter les grands espaces d’états. Nous discutons également de la manière dont les méthodes basées sur les politiques optimisent directement la politique sans optimiser la fonction de valeur. Dans la deuxième section, nous introduisons les jeux de Markov comme une extension des processus décisionnels de Markov pour plusieurs agents. Nous couvrons les différents cadres formés par les différentes structures de récompense et donnons les dilemmes sociaux séquentiels comme exemple du cadre d’incitation mixte. En fin de compte, nous introduisons différentes structures d’information telles que l’apprentissage centralisé qui peuvent aider à faire face à la non-stationnarité in- duite par l’adversaire. Enfin, dans la troisième section, nous donnons un bref aperçu des types d’abstraction d’état et introduisons les métriques de bisimulation comme un concept inspiré de l’abstraction de non-pertinence du modèle qui mesure la similarité entre les états. Dans le deuxième chapitre (l’article), nous approfondissons finalement l’abstraction d’agent en tant que métrique de bisimulation et dérivons un facteur de compression que nous pouvons appliquer à la diplomatie pour révéler l’agence supérieure sur les unités de joueur. / This thesis is organized into two chapters. The first chapter serves as an introduction to the concepts and ideas used in the second chapter (the article). The first chapter is divided into three sections. In the first section, we introduce Reinforcement Learning as a Machine Learning paradigm and show how its problems are formalized using Markov Decision Processes. We formalize goals as expected returns and show how the Bellman equations use the recursive formulation of return to establish a relation between the values of two successive states under the agent’s policy. After that, we argue that solving the Bellman optimality equations is intractable and introduce value-based algorithms such as Dynamic Programming, Monte Carlo methods, and Temporal Difference methods that approximate the optimal solution using Generalized Policy Iteration. Function approximation is then proposed as a way of dealing with large state spaces. We also discuss how policy-based methods optimize the policy directly without optimizing the value function. In the second section, we introduce Markov Games as an extension of Markov Decision Processes for multiple agents. We cover the different settings formed by the different reward structures and give Sequential Social Dilemmas as an example of the mixed-incentive setting. In the end, we introduce different information structures such as centralized learning that can help deal with the opponent-induced non-stationarity. Finally, in the third section, we give a brief overview of state abstraction types and introduce bisimulation metrics as a concept inspired by model-irrelevance abstraction that measures the similarity between states. In the second chapter (the article), we ultimately delve into agent abstraction as a bisimulation metric and derive a compression factor that we can apply to Diplomacy to reveal the higher agency over the player units.