This dissertation addresses the need for using machine learning-based methods rather than traditional rule-based methods for controlling non-playable characters (NPCs). The goal of the Reinforcement Learning Framework for the Unreal Engine is to enable game development studios to create, train, and more easily implement smarter, more compelling AI characters in major video game releases. The framework contains three distinct software libraries: an Unreal Engine reinforcement learning library whose purpose is to enable Unreal Engine levels to act as reinforcement learning environments, a python library which provides convenient abstractions and implementations to the reinforcement learning process, and a flexible connection system responsible for the communication between the two sides of the framework. In this dissertation, I describe the framework in detail, demonstrate the framework’s capability by implementing, training, and evaluating on the cartpole benchmark, and prove the system’s viability by comparing it to similar tools already on the market.

Military, Medical, Exploratory, and Commercial robots have much to gain from exchanging wheels for legs. However, the equations of motion of dynamic bipedal walker models are highly coupled and non-linear, making the selection of an appropriate control scheme difficult. A temporal difference reinforcement learning method known as Q-learning develops complex control policies through environmental exploration and exploitation. As a proof of concept, Q-learning was applied through simulation to a benchmark single pendulum swing-up/balance task; the value function was first approximated with a look-up table, and then an artificial neural network. We then applied Evolutionary Function Approximation for Reinforcement Learning to effectively control the swing-leg and torso of a 3 degree of freedom active dynamic bipedal walker in simulation. The model began each episode in a stationary vertical configuration. At each time-step the learning agent was rewarded for horizontal hip displacement scaled by torso altitude--which promoted faster walking while maintaining an upright posture--and one of six coupled torque activations were applied through two first-order filters. Over the course of 23 generations, an approximation of the value function was evolved which enabled walking at an average speed of 0.36 m/s. The agent oscillated the torso forward then backward at each step, driving the walker forward for forty-two steps in thirty seconds without falling over. This work represents the foundation for improvements in anthropomorphic bipedal robots, exoskeleton mechanisms to assist in walking, and smart prosthetics. / Master of Science

We manipulated delay and magnitude of reinforcers in two concurrent schedules of reinforcement to decrease a prevalent behavior while increasing another behavior already in the participant's repertoire. The first experiment manipulated delay, implementing a five second delay between the behavior and delivery of reinforcement for a behavior targeted for decrease while no delay was implemented after the behavior targeted for increase. The second experiment manipulated magnitude, providing one piece of food for the behavior targeted for decrease while two pieces of food were provided for the behavior targeted for increase. The experiments used an ABAB reversal design. Results suggest that behavior can be decreased without the use of extinction when contingencies favor the desirable behavior.

Hunter-Prey or Prey-Pursuit problem is a common toy domain for Reinforcement Learning, but the size of the state space is exponential in the parameters such as size of the grid or number of agents. As the size of the state space makes the flat Q-learning impossible to use for different scenarios, this thesis presents an approach to make the size of the state space constant by producing agents that use previously learned knowledge to perform on bigger scenarios containing more agents. Inspired from HRL methods, the method is composed of a parallel subtasks schema dividing the task into choices of simpler subtasks, a state representation technique convenient for this schema and its extension for bigger grids. Experimental results show that proposed method successfully provides agents that perform near to hand-coded agents by using constant sized state space independent from parameters of the domain.

Inom det heta forskningsområdet med självflygande drönare sker det en kontinuerlig utveckling både inom forskningen och inom industrin. Det finns flera forskningsproblem kring autonoma fordon, inklusive autonom styrning av drönare. Ett intressant spår för autonom styrning av drönare, är via deep reinforcement learning, dvs. en kombination av djupa neuronnät med reinforcement learning. Problemen som ofta uppkommer är tidskrävande träning, ineffektiv manövrering och problem med oförutsägbarhet och säkerhet. Även höga kostnader kan vara ett problem. Med hjälp av simuleringsprogrammet AirSim har vi fått en möjlighet att testa aktuella algoritmer utan hänsyn till kostnader och andra begränsande faktorer som kan utgöra svårigheter för att arbeta inom detta område. Microsofts egenutvecklade simulator AirSim tillåter användare att via deras applikationsprogrammeringsgränssnitt kommunicera med drönaren i programmet, vilket gör det möjligt att testa olika algoritmer. Frågeställningen som berörs är hur kan den existerande belöningsfunktionen i AirSim simulatorn förbättras med avseende på att undvika hinder och förflytta drönaren från start till mål. Målet med undersökningen är att studera och förbättra AirSims existerande Deep Q-Network algoritm med fokus på belöningsfunktionen och testa den i olika simulerade miljöer. Med hjälp av två olika experiment som utförts i två olika miljöer, observerades belöningen, antalet kollisioner och beteendet agenten hade i simulatorn. Vi lyckades inte få fram tillräckligt med data för att kunna mäta en tydlig förbättring av den modifierade belöningsfunktionens utvärderingsmått, dock kan vi säga att vi lyckades utveckla en belöningsfunktion som presterar bra genom att den undviker hinder och tar sig till mål. För att kunna jämföra vilken av belöningsfunktionerna som är bättre, behövs mer forskning inom ämnet. Med de problem som fanns med att samla in data är slutsatsen att vi inte lyckades förbättra algoritmen då vi vet inte om den presterar bättre eller sämre än den existerande belöningsfunktionen. / Drones are growing popular and so is the research within the field of autonomous drones. There are several research problems around autonomous vehicles overall, but one interesting problem covered by this study is the autonomous manoeuvring of drones. One interesting path for autonomous drones is through deep reinforcement learning, which is a combination of deep neural networks and reinforcement learning. Problems that researchers often encounter within the field stretch from time consuming training, effective manoeuvring to problems with unpredictability and security. Even high costs of testing can be an issue. With the help of simulation programs, we are able to test algorithms without any concerns to cost or other real-world factors that could limit our work. Microsoft’s own simulator AirSim lets users control the vehicle in their simulator through an application programming interface, which enables the possibility to test a variety of algorithms. The research question addressed in this study is how can the pre-existing reward function be improved on avoiding obstacles and move the drone from start to goal. The goal of this study is to find improvements on AirSim’s pre-existing Deep Q-Network algorithm’s reward function and test it in two different simulated environments. By conducting several experiments and storing evaluation metrics produced by the agents, it was possible to observe a result. The observed evaluation metrics included the average reward that the agent received over time, number of collisions and overall performance in the respective environment. We were not successfully able to gather enough data to measure an improvement of the evaluation metrics for the modified reward function. The modified function that was created performed well but did not display any substantially improved performance. To be able to successfully compare if one reward function is better than the other more research needs to be done. With the difficulties of gathering data, the conclusion is that we created a reward function that we can’t tell if it is better or worse than the benchmark reward function.

No description available.

Partial order planning is an important approach that solves planning problems without completely specifying the orderings between the actions in the plan. This property provides greater flexibility in executing plans; hence making the partial order planners a preferred choice over other planning methodologies. However, in order to find partially ordered plans, partial order planners perform a search in plan space rather than in space of world states and an uninformed search in plan space leads to poor efficiency. In this thesis, I discuss applying a reinforcement learning method, called First-visit Monte Carlo method, to partial order planning in order to design agents which do not need any training data or heuristics but are still able to make informed decisions in plan space based on experience. Communicating effectively with the agent is crucial in reinforcement learning. I address how this task was accomplished in plan space and the results from an evaluation of a blocks world test bed.

No description available.

In this master thesis project a LiDAR-based, depth image-based and semantic segmentation image-based reinforcement learning agent is investigated and compared forlearning in simulation and performing in real-time. The project utilize the Deep Deterministic Policy Gradient architecture for learning continuous actions and was designed to control a RC car. One of the first project to deploy an agent in a real scenario after training in a similar simulation. The project demonstrated that with a proper reward function and by tuning driving parameters such as restricting steering, maximum velocity, minimum velocity and performing input data scaling a LiDAR-based agent could drive indefinitely on a simple but completely unseen track in real-time.

Die Arbeit ist Teil eines aktuellen Forschungsprojekts, bei der ein dynamischer zweidimensionaler Verkehrsflusssimulator zur Beschreibung der Binnenschifffahrt auf einer ca. 220 km langen Strecke auf dem Niederrhein entwickelt werden soll. Ziel dieser Arbeit ist es, ein Schifffolgemodell mithilfe von Deep Learning Ansätzen umzusetzen und mittels geeigneter Beschleunigung ein kollisionsfreies Folgen zu realisieren. Dabei sind die gesetzlichen Randbedingungen (Verkehrsregeln, Mindestabstände) sowie hydrodynamische und physikalische Gesetzmäßigkeiten wie minimale und maximale Beschleunigungen und Geschwindigkeiten zu berücksichtigen. Nach der Analyse des Systems sowie der notwendigen Parameter, wird ein Modell entworfen und die Modellparameter bestimmt. Unter Berücksichtigung der Modellparameter wird ein Agent ausgewählt und das System in MATLAB implementiert. Die Parameter sind so gestaltet, dass sich damit ein allgemeines Folgemodell ergibt und beispielsweise auch ein Autofolgemodell realisieren lässt.:1 Einleitung 1.1 Ziel der Arbeit 1.2 Aufbau der Arbeit 2 Stand der Technik 2.1 Traditionelle Folgemodelle 2.2 Reinforcement Learning 2.2.1 Modell 2.2.2 State-value function 2.3 Deep Reinforcement Learning 2.3.1 Künstliches neuronales Netz 3 Mathematische Grundlagen 3.1 Künstliche Neuronen 3.1.1 Aktivierungsfunktionen 3.2 Normierung 3.3 Funktionstypen 4 Analyse 4.1 Analyse der Systemfunktionen der Software 5 Modell 5.1 Aufbau 5.2 Approximatoren 5.3 Parameter 5.4 Szenarien 6 Agent 6.1 Auswahl des Agenten 6.2 Twin-Delayed Deterministic Policy Gradient (TD3) 7 Implementierung 7.1 Environment 7.1.1 Rewardfunktion 7.2 Agent 7.2.1 Netzwerkarchitektur 7.2.1.1 Actor-Netzwerk 7.2.1.2 Critic-Netzwerk 7.2.1.3 Rauschprozesse 7.3 Hyperparameter 7.4 Sonstige Parameter 8 Trainingsprozess 45 8.1 Ornstein-Uhlenbeck-Prozess 8.2 Algorithmus 9 Validierung 9.1 Fahrverhalten bei verschiedenen Charakteristika 9.2 Vergleich mit dem Intelligent Driver Model 10 Zusammenfassung und Ausblick Literaturverzeichnis