Physikalische Simulation ist eine effektive und praktische Methode, um die Probleme der realen Welt zu untersuchen und zu erforschen. Jedoch kann die Simulation wertvolle Ergebnisse für die Robotik nur in enger Verbindung zu den realen Robotern liefern. In der Arbeit haben wie Methoden untersucht, die einen glatten Übergang von simulierten zu realen Robotern für die Steuerung humanoider Roboter erlauben. Wir haben ein Framework entwickelt, in dem Roboter sowohl in realen als auch in simulierten Umgebungen arbeiten können. Wir haben einen Simulator für humanoide Roboter auf konzeptioneller und experimenteller Ebene durch entsprechende Experimente evaluiert. Weiterhin haben wir den Simulator um zusätzliche Modelle erweitert und Parameter mithilfe Evolutionärer Algorithmen optimiert. Schließlich haben wir Bewegungen in Simulationen mit Maschinellem Lernen entwickelt und erfolgreich auf reale Roboter übertragen. Als Resultat können Roboter Teams sowohl in den Simulationsligen als auch in den realen Ligen des RoboCup mit identischen Steuerungen Fußball spielen. Das ergibt eine enge Verbindung zwischen den Entwicklern von simulierten und realen Robotern. / Physical simulation is an effective and practical method, to apply to the study and exploration of real world problems. However, simulation can offer valuable results for robotics only in close connection to real robots. In this thesis, we investigated how to create a mechanism that provides a smooth gradient to transfer humanoid robot control from simulated robot to real robot. We developed a framework for running robots both in real and simulated settings; and evaluated a humanoid robot simulator at a conceptual model level and results level by conducting experiments. Then, we improved the simulator by adding missing models and optimizing parameters with Evolutionary Algorithms. Finally, we developed motions in the simulations, with the help of Machine Learning, and transferred them to real robots successfully. As a result, a robot team can play soccer using identical controls in both the simulation and real RoboCup leagues. This constitutes a close connection between the communities working with simulated and real robots.

Les petits robots humanoïdes sont généralement soumis à de nombreuses imperfections : déformations et jeux mécaniques, défauts électriques et problèmes d'asservissements moteurs. L'objet de ces travaux est l'utilisation de techniques d'apprentissage pour compenser les imperfections du robot réel. L'amélioration de la précision de l'odométrie et de la stabilité de mouvements générés est étudiée. Cette thèse est fortement guidée et inspirée par la participation de l'équipe Rhoban (Rhoban Football Club) à la compétition internationale de robotique, la RoboCup. Depuis 2011, l'équipe concourt chaque année dans la ligue des petits robots humanoïdes complètement autonomes (Humanoid Kid-Size) dans un tournoi de football robotique. L'odométrie proprioceptive estime les déplacements du robot à partir de ses capteurs internes (la caméra n'est pas utilisée) alors que l'odométrie prédictive simule les déplacements engendrés par une séquence donnée d'ordres du mouvement de marche. Deux méthodes de correction sont ici proposées pour les deux odométries. La première se fonde sur une technique de régression non paramétrique (LWPR) et un système externe de capture de mouvement. La deuxième optimise (CMA-ES) un modèle de correction linéaire sans ne nécessiter aucun autre dispositif de mesure. L'odométrie proprioceptive est essentielle à la localisation du robot sur le terrain de football alors que l'odométrie prédictive permet d'entraîner hors ligne une politique de contrôle de la marche. La synthèse de mouvements très dynamiques tels que la marche ou le tir est rendue difficile par la forte contrainte de stabilité bipède et les imperfections des servomoteurs. Des mouvements de tir sont tout d'abord générés par optimisation (CMA-ES) et évalués au travers du modèle dynamique inverse du robot. Le développement d'un simulateur physique a été commencé. Le but est de réduire la distance entre le comportement réel et désiré du robot par correction des mouvements au sein du simulateur. / Small humanoid robots are often affected by many flaws : mechanical wraps and backlashes, electrical issues and motor control problems. This work is aimed at applying machine learning methods to deal with the flaws of the real robot. More precisely, improving the odometry accuracy and generated motion stability is studied. This thesis is highly guided and inspired by the participation of the Rhoban team (Rhoban Football Club) to the international RoboCup competition. Since 2011, the team has been competing each year in a soccer tournament within the fully autonomous small humanoid robots (Kid-Size) league. Proprioceptive odometry estimates the robot displacements from its internal sensors (no camera is used) whereas predictive odometry simulates the displacements created from a sequence of walk orders. Two corrective methods are proposed for the two kinds of odometries. The first one is based on a non parametric regression (LWPR) and a motion capture setup. The second one optimizes (CMA-ES) a linear corrective model without needing any external measure system. The proprioceptive odometry is essential to the localization of the robot on the soccer field. The predictive odometry is used to train a control policy for the walk motion. The generation of very dynamic motions like walking or kicking the ball is difficult due to the biped balance constraint and the many servomotor flaws. To start, kick motions are generated by optimization (CMA-ES) and evaluated based on the inverse dynamic model of the robot. The implementation of a physics simulator has been started. The objective is make the real behaviour of the robot to catch up the target trajectory by correcting the motion within the simulator.

Simulation is often used in research and industry as a low cost, high efficiency alternative to real model testing. Simulation has also been used to develop and test powerful learning algorithms. However, optimized values in simulation do not translate directly to optimized values in application. In fact, heavy optimization in simulation is likely to exploit the simplifications made in simulation. This observation brings to question the utility of learning in simulation. The UT Austin Villa 3D Simulation Team developed an optimization framework on which a robot agent was trained to maximize the speed of an omni-directional walk. The resulting agent won first place in the 2011 RoboCup 3D Simulation League. This thesis presents the adaptation of this optimization framework to learn parameters in simulation that improved the forward walk speed of the real Aldebaran Nao. By constraining the simulation with tree models learned from the real robot, and manually guiding the optimization based on testing on the real robot, the Nao's walk speed was improved by about 30%. / text

Localization is an important problem that must be resolved in order for a robot to make an estimation of its location based on observation and odometry updates. Relying on artificial landmarks such as the lines, circles, and goalposts in the robot soccer domain, current robot localization requires prior knowledge and suffers from uncertainty problems due to partial observation, and thus is less generalizable compared to human beings, who refer to their surroundings for complimentary information. To improve the certainty of the localization model, we propose a framework that recognizes orientation by actively using natural landmarks from the off-field surroundings, extracting these visual features from raw images. Our approach involves identifying visual features and natural landmarks, training with localization information to understand the surroundings, and prediction based on matching of features. This approach can increase the precision of robot orientation and improve localization accuracy by eliminating uncertain hypotheses, and in addition, it is also a general approach that can be extended and applied to other localization problems as well. / text

Thesis (MSc)--Stellenbosch University, 2015. / ENGLISH ABSTRACT: This study has started as part of a research project at Stellenbosch University (SU) that aims at building a team of soccer-playing robots for the RoboCup Small Size League (SSL). In the RoboCup SSL the Decision- Making Module (DMM) plays an important role for it makes all decisions for the robots in the team. This research focuses on the development of some parts of the DMM for the team at SU. A literature study showed that the DMM is typically developed in a hierarchical structure where basic soccer skills form the fundamental building blocks and high-level team behaviours are implemented using these basic soccer skills. The literature study also revealed that strategies in the DMM are usually developed using a hand-coded approach in the RoboCup SSL domain, i.e., a specific and fixed strategy is coded, while in other leagues a Machine Learning (ML) approach, Reinforcement Learning (RL) in particular, is widely used. This led to the following research objective of this thesis, namely to develop basic soccer skills using RL for the RoboCup Small Size League. A second objective of this research is to develop a simulation environment to facilitate the development of the DMM. A high-level simulator was developed and validated as a result. The temporal-difference value iteration algorithm with state-value functions was used for RL, along with a Multi-Layer Perceptron (MLP) as a function approximator. Two types of important soccer skills, namely shooting skills and passing skills were developed using the RL and MLP combination. Nine experiments were conducted to develop and evaluate these skills in various playing situations. The results showed that the learning was very effective, as the learning agent executed the shooting and passing tasks satisfactorily, and further refinement is thus possible. In conclusion, RL combined with MLP was successfully applied in this research to develop two important basic soccer skills for robots in the RoboCup SSL. These form a solid foundation for the development of a complete DMM along with the simulation environment established in this research. / AFRIKAANSE OPSOMMING: Hierdie studie het ontstaan as deel van 'n navorsingsprojek by Stellenbosch Universiteit wat daarop gemik was om 'n span sokkerrobotte vir die RoboCup Small Size League (SSL) te ontwikkel. Die besluitnemingsmodule (BM) speel 'n belangrike rol in die RoboCup SSL, aangesien dit besluite vir die robotte in die span maak. Hierdie navorsing fokus op ontwikkeling van enkele komponente van die BM vir die span by SU. 'n Literatuurstudie het getoon dat die BM tipies ontwikkel word volgens 'n hiërargiese struktuur waarin basiese sokkervaardighede die fundamentele boublokke vorm en hoëvlak spangedrag word dan gerealiseer deur hierdie basiese vaardighede te gebruik. Die literatuur het ook getoon dat strategieë in die BM van die RoboCup SSL domein gewoonlik ontwikkel word deur 'n hand-gekodeerde benadering, dit wil s^e, 'n baie spesifieke en vaste strategie word gekodeer, terwyl masjienleer (ML) en versterkingsleer (VL) wyd in ander ligas gebruik word. Dit het gelei tot die navorsingsdoelwit in hierdie tesis, naamlik om basiese sokkervaardighede vir robotte in die RoboCup SSL te ontwikkel. 'n Tweede doelwit was om 'n simulasie-omgewing te ontwikkel wat weer die ontwikkeling van die BM sou fasiliteer. Hierdie simulator is suksesvol ontwikkel en gevalideer. Die tydwaarde-verskil iterariewe algoritme met toestandwaarde-funksies is gebruik vir VL saam met 'n multi-laag perseptron (MLP) vir funksiebenaderings. Twee belangrike sokkervaardighede, naamlik doelskop- en aangeevaardighede is met hierdie kombinasie van VL en MLP ontwikkel. Nege eksperimente is uitgevoer om hierdie vaardighede in verskillende speelsituasies te ontwikkel en te evalueer. Volgens die resultate was die leerproses baie effektief, aangesien die leer-agent die doelskiet- en aangeetake bevredigend uitgevoer het, en verdere verfyning is dus moontlik. Die gevolgtrekking is dat VL gekombineer met MLP suksesvol toegepas is in hierdie navorsingswerk om twee belangrike, basiese sokkervaardighede vir robotte in die RoboCup SSL te ontwikkel. Dit vorm 'n sterk fondament vir die ontwikkeling van 'n volledige BM tesame met die simulasie-omgewing wat in hierdie werk daargestel is.

Neste trabalho é apresentado o desenvolvimento de um modelo de programação paralela inspirado na estrutura geral dos sistemas conexionistas como proposta por Rumelhart, McClelland e Hinton em seu livro intitulado The Parallel Distributed Processing Perspective (MCCLELLAND, RUMELHART e HINTON, 1983). Este modelo tem como objetivo servir como base para o desenvolvimento de um sistema autônomo de controle em tempo hábil para um time de futebol de robôs, orientado ao uso de processamento paralelo em computadores multicore. O trabalho serve como um guia para compreensão e uso do modelo de programação proposto, apresentando ainda, por meio de experimentos, a sua eficiência dentro do contexto de processamento paralelo e a sua adequação para solução do problema de controle de futebol de robôs. / This work presents the development of a parallel programming model inspired by the general framework of connectionist systems as proposed by Rumelhart, McClelland and Hinton in their book entitled The Parallel Distributed Processing Perspective (MCCLELLAND, RUMELHART e HINTON, 1983). This model is intended to serve as the basis for the development of an autonomous system for real-time control of a robotic soccer team driven towards the usage of effective parallel processing on multicore computers. The work serves as a guide for understanding and using the proposed programming model, yet showing, through experiments, the efficiency within the context of parallel processing and its suitability for solving the robotic soccer control problem.

Neste trabalho é apresentado o desenvolvimento de um modelo de programação paralela inspirado na estrutura geral dos sistemas conexionistas como proposta por Rumelhart, McClelland e Hinton em seu livro intitulado The Parallel Distributed Processing Perspective (MCCLELLAND, RUMELHART e HINTON, 1983). Este modelo tem como objetivo servir como base para o desenvolvimento de um sistema autônomo de controle em tempo hábil para um time de futebol de robôs, orientado ao uso de processamento paralelo em computadores multicore. O trabalho serve como um guia para compreensão e uso do modelo de programação proposto, apresentando ainda, por meio de experimentos, a sua eficiência dentro do contexto de processamento paralelo e a sua adequação para solução do problema de controle de futebol de robôs. / This work presents the development of a parallel programming model inspired by the general framework of connectionist systems as proposed by Rumelhart, McClelland and Hinton in their book entitled The Parallel Distributed Processing Perspective (MCCLELLAND, RUMELHART e HINTON, 1983). This model is intended to serve as the basis for the development of an autonomous system for real-time control of a robotic soccer team driven towards the usage of effective parallel processing on multicore computers. The work serves as a guide for understanding and using the proposed programming model, yet showing, through experiments, the efficiency within the context of parallel processing and its suitability for solving the robotic soccer control problem.

Neste trabalho é apresentado o desenvolvimento de um modelo de programação paralela inspirado na estrutura geral dos sistemas conexionistas como proposta por Rumelhart, McClelland e Hinton em seu livro intitulado The Parallel Distributed Processing Perspective (MCCLELLAND, RUMELHART e HINTON, 1983). Este modelo tem como objetivo servir como base para o desenvolvimento de um sistema autônomo de controle em tempo hábil para um time de futebol de robôs, orientado ao uso de processamento paralelo em computadores multicore. O trabalho serve como um guia para compreensão e uso do modelo de programação proposto, apresentando ainda, por meio de experimentos, a sua eficiência dentro do contexto de processamento paralelo e a sua adequação para solução do problema de controle de futebol de robôs. / This work presents the development of a parallel programming model inspired by the general framework of connectionist systems as proposed by Rumelhart, McClelland and Hinton in their book entitled The Parallel Distributed Processing Perspective (MCCLELLAND, RUMELHART e HINTON, 1983). This model is intended to serve as the basis for the development of an autonomous system for real-time control of a robotic soccer team driven towards the usage of effective parallel processing on multicore computers. The work serves as a guide for understanding and using the proposed programming model, yet showing, through experiments, the efficiency within the context of parallel processing and its suitability for solving the robotic soccer control problem.

Made available in DSpace on 2014-06-12T15:58:23Z (GMT). No. of bitstreams: 2 arquivo4030_1.pdf: 9256625 bytes, checksum: 7f4c7776d120f9fef113904bc5e3e526 (MD5) license.txt: 1748 bytes, checksum: 8a4605be74aa9ea9d79846c1fba20a33 (MD5) Previous issue date: 2010 / Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco / A aprendizagem por reforço é usada em cenários nos quais não se dispõe de um resultado associado a cada estado nem a cada ação tomada por um agente inteligente. Essa forma de aprendizagem; portanto, mantém uma forte dependência da exploração dos espaços de estados e de ações que produz uma explosão de dados cujo armazenamento se torna um problema em muitas situações. Por outro lado, tem-se a mineração de dados como uma área da inteligência artificial que busca extrair informações ou padrões de grandes quantidades de dados, ou armazenados em um banco de dados ou trafegando em um fluxo contínuo de dados. A principal contribuição deste trabalho é mostrar como as técnicas de mineração de dados podem ser utilizadas para selecionar as variáveis e ações mais relevantes dos ambientes da aprendizagem por reforço. O objetivo desta seleção é reduzir a complexidade do problema e a quantidade de memória usada pelo agente, que podem acelerar a convergência da aprendizagem. A dificuldade em utilizar as técnicas de mineração de dados em ambientes da aprendizagem por reforço deve-se ao não armazenamento dos dados provenientes da exploração dos espaços de estados e de ações em um banco de dados. Este trabalho também contribui propondo um esquema de armazenamento para os estados visitados e as ações executadas pelo agente. Neste estudo, o método de seleção de atributos e de ações foi validado experimentalmente em um problema no qual a aprendizagem por reforço é a abordagem mais adequada; o drible no futebol de robôs RoboCup-2D. Este problema é composto de 23 variáveis contínuas e 113 ações disponíveis para o agente que consome cerca de 18MB de memória quando utilizado o algoritmo combinado com a técnica de tile-coding. Os resultados dos experimentos mostraram que a quantidade de variáveis do ambiente pode ser reduzida em até 56% e a quantidade de ações em até 85%, com uma redução do uso da memória de 95% e um aumento no desempenho de aproximadamente 10% de acordo com a distribuição da freqüência relativa de sucesso do agente. A abordagem proposta é simples de usar e eficiente

The RoboCup Sony Legged Robot League is an initiative to promote robotics technologies and artificial intelligence in the form of a soccer competition between four-legged robots. The Blekinge Institute of Technology, Royal Institute of Technology, the Universities of ¨ Orebro and Ume°a, competing in the RoboCup domain as Team Sweden , have been participants in the league for three years. To improve the chances of victory in the league, a way to communicate important data between robots is desired. This thesis explores methods for implementing this communication using only the built-in hardware of the robots, i.e. one speaker and two microphones. / david.erman@bth.se