Corrida de carros e um gênero popular de jogos eletrônicos e um domínio com vários desafios a serem explorados no âmbito da Inteligência Artificial (IA), tendo recebido atenção crescente nos últimos anos. Naturalmente, um desses desafios e criar pilotos virtuais capazes de aprender sozinhos a correr nas pistas. Neste projeto de mestrado, nos adaptamos e aplicamos técnicas de Aprendizagem por Reforço (Reinforcement Learning) no desenvolvimento de um agente completamente autônomo capaz de correr em pistas de vários formatos dentro do simulador TORCS. Esse jogo de código aberto possui um sistema de física muito elaborado e permite a criação de módulos de IA para controlar os carros, sendo assim um ambiente de testes frequentemente adotado para pesquisas nesse contexto. O objetivo do nosso agente e encontrar ações de controle do acelerador e freio a fim de gastar o menor tempo possível em cada volta. Para atingir tal meta, ele coleta dados na primeira volta, gera um modelo do circuito, segmenta e classifica cada trecho da pista e, finalmente, da voltas no percurso ate atingir um comportamento consistente. Além das questões relacionadas a aprendizagem, este trabalho explora conceitos de Sistemas de Controle, em especial controladores PID (Proporcional, Integrativo, Derivativo), usados para a implementação da heurística do manejo do volante. Também abordamos os fundamentos de alguns assistentes de direção, tais como ABS (Anti-lock Braking System) e controle de estabilidade. Esses princípios são de grande importância para tornar o agente capaz de guiar o carro dentro de um ambiente com simulação física tão próxima a realidade. Nesse ponto e no emprego do sensoriamento para a aquisição de dados, nosso trabalho flerta com a área de Robótica Móvel. Por fim, avaliamos o desempenho de nosso piloto virtual comparando seus resultados com os de controladores baseados em outras técnicas. / Reinforcement learning driver for TORCS car racing simulator.

Corrida de carros e um gênero popular de jogos eletrônicos e um domínio com vários desafios a serem explorados no âmbito da Inteligência Artificial (IA), tendo recebido atenção crescente nos últimos anos. Naturalmente, um desses desafios e criar pilotos virtuais capazes de aprender sozinhos a correr nas pistas. Neste projeto de mestrado, nos adaptamos e aplicamos técnicas de Aprendizagem por Reforço (Reinforcement Learning) no desenvolvimento de um agente completamente autônomo capaz de correr em pistas de vários formatos dentro do simulador TORCS. Esse jogo de código aberto possui um sistema de física muito elaborado e permite a criação de módulos de IA para controlar os carros, sendo assim um ambiente de testes frequentemente adotado para pesquisas nesse contexto. O objetivo do nosso agente e encontrar ações de controle do acelerador e freio a fim de gastar o menor tempo possível em cada volta. Para atingir tal meta, ele coleta dados na primeira volta, gera um modelo do circuito, segmenta e classifica cada trecho da pista e, finalmente, da voltas no percurso ate atingir um comportamento consistente. Além das questões relacionadas a aprendizagem, este trabalho explora conceitos de Sistemas de Controle, em especial controladores PID (Proporcional, Integrativo, Derivativo), usados para a implementação da heurística do manejo do volante. Também abordamos os fundamentos de alguns assistentes de direção, tais como ABS (Anti-lock Braking System) e controle de estabilidade. Esses princípios são de grande importância para tornar o agente capaz de guiar o carro dentro de um ambiente com simulação física tão próxima a realidade. Nesse ponto e no emprego do sensoriamento para a aquisição de dados, nosso trabalho flerta com a área de Robótica Móvel. Por fim, avaliamos o desempenho de nosso piloto virtual comparando seus resultados com os de controladores baseados em outras técnicas. / Reinforcement learning driver for TORCS car racing simulator.

Artificiella neurala nätverk (ANN) kan användas för att lära och efterlikna olika beteenden. I det här projektet används ANN för att kontrollera en bil i en simulatormiljö genom att lära upp nätverken med mänskliga exempel. Syftet med projektet är att ta reda på vilken kombination av parametrar det är som gör att en bil kan kontrolleras av ANN med ett bra resultat. Detta undersöks genom att skapa åtta olika artificiella förare som representerar olika kombinationer av parametrar och sedan jämföra förarnas beteende och resultat för att se vilken förare som klarar sig bäst. På så vis är det sedan möjligt att härleda vilken kombination av parametrar som är den bästa för att kontrollera en bil med ANN. Resultaten från experimenten visar att den bästa kombinationen av parametrar för att styra en artificiell bilförare med ANN är högnivåinput, högnivåoutput och en delad nätverksarkitektur. Framtida arbeten innefattar bland annat hybrider av kombinationer.

No

The purpose of this thesis was to compare the performance of three different imitation learning algorithms with human experts, with limited expert time. The central question was, ”How should one implement imitation learning in a simulated car racing environment, using human experts, to achieve the best performance when access to the experts is limited?”. We limited the work to only consider the three algorithms Behavior Cloning, DAGGER, and HG-DAGGER and limited the implementation to the car racing simulator TORCS. The agents consisted of the same type of feedforward neural network that utilized sensor data provided by TORCS. Through comparison in the performance of the different algorithms on a different amount of expert time, we can conclude that HGDAGGER performed the best. In this case, performance is regarded as a distance covered given set time. Its performance also seemed to scale well with more expert time, which the others did not. This result confirmed previously published results when comparing these algorithms. / Målet med detta examensarbete var att jämföra prestandan av tre olika algoritmer inom området imitationinlärning med mänskliga experter, där experttiden är begränsad. Arbetets frågeställning var, ”Hur ska man implementera imitationsinlärning i en bilsimulator, för att få bäst prestanda, med mänskliga experter där experttiden är begränsad?”. Vi begränsade arbetet till att endast omfatta de tre algoritmerna, Behavior Cloning, DAGGER och HG-DAGGER, och begränsade implementationsmiljön till bilsimulatorn TORCS. Alla agenterna bestod av samma sorts feedforward neuralt nätverk som använde sig av sensordata från TROCS. Genom jämförelse i prestanda på olika mängder experttid kan vi dra slutsatsen att HG-DAGGER gav bäst resultat. I detta fall motsvarar prestanda körsträcka, givet en viss tid. Dess prestanda verkar även utvecklas väl med ytterligare experttid, vilket de övriga inte gjorde. Detta resultat bekräftar tidigare publicerade resultat om jämförelse av de tre olika algoritmerna.

Context. Building autonomous racing car controllers is a growing field of computer science which has been receiving great attention lately. An approach named Artificial Potential Fields (APF) is used widely as a path finding and obstacle avoidance approach in robotics and vehicle motion controlling systems. The use of APF results in a collision free path, it can also be used to achieve other goals such as overtaking and maneuverability. Objectives. The aim of this thesis is to build an autonomous racing car controller that can achieve good performance in terms of speed, time, and damage level. To fulfill our aim we need to achieve optimality in the controller choices because racing requires the highest possible performance. Also, we need to build the controller using algorithms that does not result in high computational overhead. Methods. We used Particle Swarm Optimization (PSO) in combination with APF to achieve optimal car controlling. The Open Racing Car Simulator (TORCS) was used as a testbed for the proposed controller, we have conducted two experiments with different configuration each time to test the performance of our APF- PSO controller. Results. The obtained results showed that using the APF-PSO controller resulted in good performance compared to top performing controllers. Also, the results showed that the use of PSO proved to enhance the performance compared to using APF only. High performance has been proven in the solo driving and in racing competitions, with the exception of an increased level of damage, however, the level of damage was not very high and did not result in a controller shut down. Conclusions. Based on the obtained results we have concluded that the use of PSO with APF results in high performance while taking low computational cost.

Att skapa en bra ANN-förare handlar inte bara om att ha en bra struktur på sitt nätverk. Det är minst lika viktigt att data som används vid träningen av nätverket är av bra kvalité. I detta arbete utvärderas i huvudsak två olika förbehandlingstekniker för att se vilken påverkan de har på slutresultatet. Båda teknikerna, reduceringsutjämning och kategoriutjämning är skapade baserat på resultat från tidigare forskningsarbeten. Förare med olika kombinationer av dessa förbehandlingstekniker och spegling utvärderades på olika banor (spegling innebär att man kopierar alla exempel och vänder håll på dem). Resultatet var tyvärr inte så bra som förväntat då förbehandlingsteknikerna visade sig vara felkonstruerade. Förbehandlingsteknikerna gjorde därför inte sina uppgifter på rätt sätt vilket gav ett lite opålitligt resultat. Det positiva i studien var att förare med en kombination av båda förbehandlingsteknikerna lärde sig bäst. Detta visar på potential hos förbehandlingsteknikerna som därför skulle kunna vidareutvecklas i framtida arbeten.

Den här studien använder sig av TORCS (The Open Racing Car Simulator) som är ett intressant spel att skapa självkörande bilar i då det finns nitton olika typer av sensorer som beskriver omgivningen för agenten. Problemet för denna studie har varit att identifiera vilka av alla dessa sensorer som kan användas i en belöningsfunktion och hur denna sedan skall implementeras. Studien har anammat en kvantitativa experimentell studie där forskningsfrågan är: Hur kan en belöningsfunktion utformas så att agenten klarar av att manövrera i spelet TORCS utan att krocka och med ett konsekvent resultat Den kvantitativ experimentell studien valdes då författarna behövde designa, implementera, utföra experiment och utvärdera resultatet för respektive belöningsfunktion. Det har utförts totalt femton experiment över tolv olika belöningsfunktioner i spelet TORCS på två olika banor E-Track 5(E-5) och Aalborg. De tolv belöningsfunktionerna utförde varsitt experiment på E-5 där de tre som fick bäst resultat: Charlie, Foxtrot och Juliette utförde ett experiment på Aalborg, då denna är en svårare bana. Detta för att kunna styrka om den kan köra på mer än en bana och om belöningsfunktionen då är generell. Juliette är den belöningsfunktion som var ensam med att klara både E-5 och Aalborg utan att krocka. Genom de utförda experimenten drogs slutsatsen att Juliette uppfyller forskningsfrågan då den klarar bägge banorna utan att krocka och när den lyckas får den ett konsekvent resultat. Studien har därför lyckats designa och implementera en belöningsfunktion som uppfyller forskningsfrågan. / For this study TORCS (The Open Racing Car Simulator) have been used, since it is an interesting game to create self-driving cars in. This is due to the fact there is nineteen different sensors available that describes the environment for the agent. The problem for this study has been to identify what sensor can be used in a reward function and how should this reward function be implemented. The study have been utilizing a quantitative experimental method where the research questions have been: How can a reward function be designed so that an Agent can maneuver in TORCS without crashing and at the same time have a consistent result The quantitative experimental method was picked since the writer’s hade to design, implement, conduct experiment and evaluate the result for each reward function. Fifteen experiments have been conducted over twelve reward functions on two different maps: E-Track 5 (E-5) and Aalborg. Each of the twelve reward function conducted an experiment on E-5, where the three once with the best result: Charlie, Foxtrot and Juliette conducted an additional experiment on Aalborg. The test on Aalborg was conducted in order to prove if the reward function can maneuver on more than one map. Juliette was the only reward function that managed to complete a lap on both E-5 and Aalborg without crashing. Based on the conducted experiment the conclusion that Juliette fulfills the research question was made, due to it being capable of completing both maps without crashing and if it succeeded it gets a consistent result. Therefor this study has succeeded in answering the research question.

Pharmaceuticals, personal care products (PPCPs) and other trace organic contaminants (TOrCs) encompass a diverse group of chemicals that are not currently monitored or regulated in US drinking water or wastewater. Researchers have found low levels of TOrCs in aquatic and terrestrial environments all over the globe, and observed negative effects on impacted biota. The primary source of TOrCs in the environment is domestic wastewater discharges. Centralized wastewater treatment plants present greater risks on a global scale, but on a local scale, onsite treatment systems may have more potent impacts on resources that are invaluable to residents, including groundwater, surface waters, and soils. The objective of this thesis is to identify and characterize promising treatment technologies for onsite TOrC remediation. Receptors who could be impacted by TOrC discharges are assessed, and applications that may require alternative treatment are identified. The best treatment technologies are recognized as those that protect sensitive environmental receptors, provide permanent removal pathways for as many TOrCs as possible, and are not prohibitively expensive to install or maintain. Findings from a pilot study show increased removal of conventional pollutants and TOrCs in an aerobic treatment unit (ATU), two types of biofilter, and a hybrid constructed wetland, all relative to septic tank effluent. The constructed wetland achieved the highest nutrient removals with TN concentrations below 10 mg/L throughout the study. A system with an ATU and peat biofilters achieved the highest removals of persistent pharmaceuticals carbamazepine and lamotrigine (>85% and >95%, respectively). / Master of Science / Trace organic contaminants (TOrCs) are chemicals found in pharmaceuticals, laundry detergents, shampoo, flame retardants, food preservatives, and many other products used in a typical home, which are not currently monitored or regulated in US drinking water or wastewater. Researchers have found low levels of TOrCs in waters and soils all over the globe, and observed negative effects on the plants and animals that live in those environments. The primary source of TOrCs in the environment is treated wastewater from centralized wastewater treatment plants, which is usually released to rivers, lakes, and other surface waters. People in rural communities also have TOrCs in their wastewater, which is normally treated using a septic system. Water released in the septic field can add TOrCs to septic field soils, groundwater sources, or nearby surface water sources, and from there these chemicals have the potential to impact human health, soil fertility, livestock health, or fish and other living things in surface waters. The objective of this thesis is to identify and characterize promising treatment technologies that would prevent or limit TOrC impacts to these important resources, which are called “receptors.” Receptors who could be impacted by TOrC discharges are assessed, and the situations in which these treatment technologies would be necessary are identified. The best treatment technologies are recognized as those that protect sensitive environmental receptors, remove as many TOrCs as possible, and are affordable to install or maintain. An experiment was designed to compare the performance of three different technologies that could remove TOrCs from septic tank effluent, including a peat filter and a constructed wetland. The constructed wetland removed the most nitrogen (total nitrogen <10 mg/L throughout the study), and a system with a peat filter removed the greatest amounts of persistent pharmaceuticals carbamazepine and lamotrigine (averaging >85% and >95%, respectively).

The Car Racing competition platform is used for evaluating different car control solutions under competitive conditions [1]. These competitions are organized as part of the IEEE Congress on Evolutionary Computation (CEC) and Computational Intelligence and Games Sym-posium (CIG). The goal is to learn and develop a controller for a car in the TORCS open source racing game [2]. Oussama Khatib [3] (1986) introduced Artificial potential fields (APFs) for the first time while looking for new ways to avoid obstacles for manipulators and mobile robots in real time. In car racing games a novel combination of artificial potential fields as the major control paradigm for car controls in a multi-agent system is being used to coordinate control interests in different scenarios [1]. Here we extend the work of Uusitalo and Stefan J. Johansson by introducing effective collision detection, overtaking maneuvers, run time evaluation and detailed analysis of the track using the concept of multi-agent artificial potential fields MAPFs. The results of our extended car controller in terms of lap time, number of damages and position in the race is improved. / We have concluded by implementing a controller that make use of multi agent based artificial potential field approach to achieve the tricky and complex task of collision detection and overtaking while driving in a car racing game with different other controllers as opponents. We exploited the advantages of APFs to the best of our knowledge using laws of physics and discrete mathematics in order to achieve successful overtaking behavior and overcome its drawbacks as being very complex to implement and high memory requirements for time critical applications e.g. car racing games (Section 3.1, RQ1). Dynamic objects in a fast changing environment like a car racing game are likely to collide more often with each other, thus resulting in higher number of damages. Using APFs instead of traditionally used collision avoidance techniques resulted in less number of damages during the race, thus minimizing the lap’s time which in turn contribute to better position in the race as shown in experiment 3 (Section 3.1, RQ2 and Section 6.6). Overtaking maneuvers are complex and tricky and is the major cause of collision among cars both in real life as well as in car racing games, thus the criteria to measure the performance regarding overtaking behavior of different controllers in the race is based on number of damages taken during the race. The comparison between the participating controllers in terms of damages taken during various rounds of the race is analyzed in experiment 3 (Section 3.1, RQ3 and Section 6.6). The results of the quick race along with opponents shows good results on three tracks while having bad performance on the remaining other track. Our controller got 1st position on the CG track 2 while kept 2nd position on CS Speed way 1 and Alpine 1. It had worse performance on wheel 2 which needs to be optimized in the future for better results on this track and other similar complex and tricky tracks. / +46723266771