This thesis deals with an echo state network and with acceleration of its learning by implementing the echo state network on a graphics processor. The theoretical part consists of the description of neural networks and some selected types of neural networks, on which is based the echo state network. After that, there are some other algorithms described used for time series analysis and last but not least, the tools that were used in the practical part of the thesis were briefly described. The practical part describes the creation of the accelerated version of the echo state network. After that, there is described the creation of input data sets of real financial indexes, on which the echo state network and the other algorithmns were then tested. By analyzing this accelerated version it was found that its learning speed did not reach the theoretical expectations. The accelerated version works slower, but with greater precision. By analyzing the results of the measurement of the other algorithmns it was found that the highest precision is achieved by solutions based on the neural network principle.

No description available.

The development of Advanced Driver Assistance Systems and autonomous driving is one of the main research fields in the area of vehicle development today. Initially the research in this area focused on analyzing and predicting driving maneuvers on highways. Nowadays, a vast amount of research focuses on urban areas as well. Driving maneuvers in urban areas are more complex and therefore more difficult to predict than driving maneuvers on highways. The goals of predicting and understanding driving maneuvers are to reduce accidents, to improve traffic density, and to develop reliable algorithms for autonomous driving. Driving behavior during different driving maneuvers such as turning at intersections, emergency braking or lane changes are analyzed. This thesis focuses on the driving behavior around lane changes and thus the prediction of lane changes in the urban area is applied with an Echo State Network. First, existing methods with a special focus on input variables and results were evaluated to derive input variables with regard to lane change and no lane change sequences. The data for this first analyses were obtained from a naturalistic driving study. Based on theses results the final set of variables (steering angle, turn signal and gazes to the left and right) was chosen for further computations. The parameters of the Echo State Network were then optimized using the data of the naturalistic driving study and the final set of variables. Finally, left and right lane changes were predicted. Furthermore, the Echo State Network was compared to a feedforward neural network. The Echo State Network could predict left and right lane changes more successful than the feedforward neural network. / Fahrerassistenzsysteme und Algorithmen zum autonomen Fahren stellen ein aktuelles Forschungsfeld im Bereich der Fahrzeugentwicklung dar. Am Anfang wurden vor allem Fahrmanöver auf der Autobahn analysiert und vorhergesagt, mittlerweile hat sich das Forschungsfeld auch auf den urbanen Verkehr ausgeweitet. Fahrmanöver im urbanen Raum sind komplexer als Fahrmanöver auf Autobahnen und daher schwieriger vorherzusagen. Ziele für die Vorhersage von Fahrmanövern sind die Reduzierung von Verkehrsunfällen, die Verbesserung des Verkehrsflusses und die Entwicklung von zuverlässigen Algorithmen für das autonome Fahren. Um diese Ziele zu erreichen, wird das Fahrverhalten bei unterschiedlichen Fahrmanövern analysiert, wie z.B. beim Abbiegevorgang an Kreuzungen, bei der Notbremsung oder beim Spurwechsel. In dieser Arbeit wird der Spurwechsel im urbanen Straßenverkehr mit einem Echo State Network vorhergesagt. Zuerst wurden existierende Methoden zur Spurwechselvorhersage bezogen auf die Eingaben und die Ergebnisse bewertet, um danach die spurwechselbezogenen Variableneigenschaften bezüglich Spurwechsel- und Nicht-Spurwechselsequenzen zu analysieren. Die Daten, die Basis für diese ersten Untersuchungen waren, stammen aus einer Realfahrstudie. Basierend auf diesen Resultaten wurden die finalen Variablen (Lenkwinkel, Blinker und Blickrichtung) für weitere Berechnungen ausgewählt. Mit den Daten aus der Realfahrstudie und den finalen Variablen wurden die Parameter des Echo State Networks optimiert und letztendlich wurden linke und rechte Spurwechsel vorhergesagt. Zusätzlich wurde das Echo State Network mit einem vorwärtsgerichteten neuronalen Netz verglichen. Das Echo State Network konnte linke und rechte Spurwechsel erfolgreicher vorhersagen als das vorwärtsgerichtete neuronale Netz.

Thesis is aimed at the possibility of utilization of extreme learning machines and echo state networks for time series forecasting with possibility of utilizing GPU acceleration. Such predictions are part of nearly everyone’s daily lives through utilization in weather forecasting, prediction of regular and stock market, power consumption predictions and many more. Thesis is meant to familiarize reader firstly with theoretical basis of extreme learning machines and echo state networks, taking advantage of randomly generating majority of neural networks parameters and avoiding iterative processes. Secondly thesis demonstrates use of programing tools, such as ND4J and CUDA toolkit, to create very own programs. Finally, prediction capability and convenience of GPU acceleration is tested.

Les fichiers sons qui accompagne mon document sont au format midi. Le programme que nous avons développés pour ce travail est en language Python. / Les réseaux de neurones à réservoir, dont le principe est de combiner un vaste réseau de neurones fixes avec un apprenant ne possédant aucune forme de mémoire, ont récemment connu un gain en popularité dans les communautés d’apprentissage machine, de traitement du signal et des neurosciences computationelles. Ces réseaux qui peuvent être classés en deux catégories : 1. les réseaux à états échoïques (ESN)[29] dont les activations des neurones sont des réels 2. les machines à états liquides (LSM)[43] dont les neurones possèdent des potentiels d’actions, ont été appliqués à différentes tâches [11][64][49][45][38] dont la génération de séquences mélodiques [30]. Dans le cadre de la présente recherche, nous proposons deux nouveaux modèles à base de réseaux de neurones à réservoir. Le premier est un modèle pour la reconnaissance de rythmes utilisant deux niveaux d’apprentissage, et avec lequel nous avons été en mesure d’obtenir des résultats satisfaisants tant au niveau de la reconnaissance que de la résistance au bruit. Le second modèle sert à l’apprentissage et à la génération de séquences périodiques. Ce modèle diffère du modèle génératif classique utilisé avec les ESN à la fois au niveau de ses entrées, puisqu’il possède une Horloge, ainsi qu’au niveau de l’algorithme d’apprentissage, puisqu’il utilise un algorithme que nous avons spécialement développé pour cette tache et qui se nomme "Orbite". La combinaison de ces deux éléments, nous a permis d’obtenir de bons résultats, pour la génération, le sur-apprentissage et l’extraction de données. Nous pensons également que ce modèle ouvre une fenêtre intéressante vers la réalisation d’un orchestre entièrement virtuel et nous proposons deux architectures possibles que pourrait avoir cet orchestre. Dans la dernière partie de ce travail nous présentons les outils que nous avons développés pour faciliter notre travail de recherche. / Reservoir computing, the combination of a recurrent neural network and one or more memoryless readout units, has seen recent growth in popularity in and machine learning, signal processing and computational neurosciences. Reservoir-based methods have been successfully applied to a wide range of time series problems [11][64][49][45][38] including music [30], and usually can be found in two flavours: Echo States Networks(ESN)[29], where the reservoir is composed of mean rates neurons, and Liquid Sates Machines (LSM),[43] where the reservoir is composed of spiking neurons. In this work, we propose two new models based upon the ESN architecture. The first one is a model for rhythm recognition that uses two levels of learning and with which we have been able to get satisfying results on both recognition and noise resistance. The second one is a model for learning and generating periodic sequences, with this model we introduced a new architecture for generative models based upon ESNs where the reservoir receives inputs from a clock, as well as a new learning algorithm that we called "Orbite". By combining these two elements within our model, we were able to get good results on generation, over-fitting and data extraction. We also believe that a combination of several instances of our model can serve as a basis for the elaboration of an entirely virtual orchestra, and we propose two architectures that this orchestra may have. In the last part of this work, we briefly present the tools that we have developed during our research.

<p>Deep ‘Analog Artificial Neural Networks’ (AANNs) perform complex classification problems with high accuracy. However, they rely on humongous amount of power to perform the calculations, veiling the accuracy benefits. The biological brain on the other hand is significantly more powerful than such networks and consumes orders of magnitude less power, indicating some conceptual mismatch. Given that the biological neurons are locally connected, communicate using energy efficient trains of spikes, and the behavior is non-deterministic, incorporating these effects in Artificial Neural Networks (ANNs) may drive us few steps towards a more realistic neural networks. </p> <p> </p> <p>Emerging devices can offer a plethora of benefits including power efficiency, faster operation, low area in a vast array of applications. For example, memristors and Magnetic Tunnel Junctions (MTJs) are suitable for high density, non-volatile Random Access Memories when compared with CMOS implementations. In this work, we analyze the possibility of harnessing the characteristics of such emerging devices, to achieve neuro-inspired solutions to intricate problems.</p> <p> </p> <p>We propose how the inherent stochasticity of nano-scale resistive devices can be utilized to realize the functionality of spiking neurons and synapses that can be incorporated in deep stochastic Spiking Neural Networks (SNN) for image classification problems. While ANNs mainly dwell in the aforementioned classification problem solving domain, they can be adapted for a variety of other applications. One such neuro-inspired solution is the Cellular Neural Network (CNN) based Boolean satisfiability solver. Boolean satisfiability (k-SAT) is an NP-complete (k≥3) problem that constitute one of the hardest classes of constraint satisfaction problems. We provide a proof of concept hardware based analog k-SAT solver that is built using MTJs. The inherent physics of MTJs, enhanced by device level modifications, is harnessed here to emulate the intricate dynamics of an analog, CNN based, satisfiability (SAT) solver. </p> <p> </p> <p>Furthermore, in the effort of reaching human level performance in terms of accuracy, increasing the complexity and size of ANNs is crucial. Efficient algorithms for evaluating neural network performance is of significant importance to improve the scalability of networks, in addition to designing hardware accelerators. We propose a scalable approach for evaluating Liquid State Machines: a bio-inspired computing model where the inputs are sparsely connected to a randomly interlinked reservoir (or liquid). It has been shown that biological neurons are more likely to be connected to other neurons in the close proximity, and tend to be disconnected as the neurons are spatially far apart. Inspired by this, we propose a group of locally connected neuron reservoirs, or an ensemble of liquids approach, for LSMs. We analyze how the segmentation of a single large liquid to create an ensemble of multiple smaller liquids affects the latency and accuracy of an LSM. In our analysis, we quantify the ability of the proposed ensemble approach to provide an improved representation of the input using the Separation Property (SP) and Approximation Property (AP). Our results illustrate that the ensemble approach enhances class discrimination (quantified as the ratio between the SP and AP), leading to improved accuracy in speech and image recognition tasks, when compared to a single large liquid. Furthermore, we obtain performance benefits in terms of improved inference time and reduced memory requirements, due to lower number of connections and the freedom to parallelize the liquid evaluation process.</p>

Les réseaux de neurones à réservoir, dont le principe est de combiner un vaste réseau de neurones fixes avec un apprenant ne possédant aucune forme de mémoire, ont récemment connu un gain en popularité dans les communautés d’apprentissage machine, de traitement du signal et des neurosciences computationelles. Ces réseaux qui peuvent être classés en deux catégories : 1. les réseaux à états échoïques (ESN)[29] dont les activations des neurones sont des réels 2. les machines à états liquides (LSM)[43] dont les neurones possèdent des potentiels d’actions, ont été appliqués à différentes tâches [11][64][49][45][38] dont la génération de séquences mélodiques [30]. Dans le cadre de la présente recherche, nous proposons deux nouveaux modèles à base de réseaux de neurones à réservoir. Le premier est un modèle pour la reconnaissance de rythmes utilisant deux niveaux d’apprentissage, et avec lequel nous avons été en mesure d’obtenir des résultats satisfaisants tant au niveau de la reconnaissance que de la résistance au bruit. Le second modèle sert à l’apprentissage et à la génération de séquences périodiques. Ce modèle diffère du modèle génératif classique utilisé avec les ESN à la fois au niveau de ses entrées, puisqu’il possède une Horloge, ainsi qu’au niveau de l’algorithme d’apprentissage, puisqu’il utilise un algorithme que nous avons spécialement développé pour cette tache et qui se nomme "Orbite". La combinaison de ces deux éléments, nous a permis d’obtenir de bons résultats, pour la génération, le sur-apprentissage et l’extraction de données. Nous pensons également que ce modèle ouvre une fenêtre intéressante vers la réalisation d’un orchestre entièrement virtuel et nous proposons deux architectures possibles que pourrait avoir cet orchestre. Dans la dernière partie de ce travail nous présentons les outils que nous avons développés pour faciliter notre travail de recherche. / Reservoir computing, the combination of a recurrent neural network and one or more memoryless readout units, has seen recent growth in popularity in and machine learning, signal processing and computational neurosciences. Reservoir-based methods have been successfully applied to a wide range of time series problems [11][64][49][45][38] including music [30], and usually can be found in two flavours: Echo States Networks(ESN)[29], where the reservoir is composed of mean rates neurons, and Liquid Sates Machines (LSM),[43] where the reservoir is composed of spiking neurons. In this work, we propose two new models based upon the ESN architecture. The first one is a model for rhythm recognition that uses two levels of learning and with which we have been able to get satisfying results on both recognition and noise resistance. The second one is a model for learning and generating periodic sequences, with this model we introduced a new architecture for generative models based upon ESNs where the reservoir receives inputs from a clock, as well as a new learning algorithm that we called "Orbite". By combining these two elements within our model, we were able to get good results on generation, over-fitting and data extraction. We also believe that a combination of several instances of our model can serve as a basis for the elaboration of an entirely virtual orchestra, and we propose two architectures that this orchestra may have. In the last part of this work, we briefly present the tools that we have developed during our research. / Les fichiers sons qui accompagne mon document sont au format midi. Le programme que nous avons développés pour ce travail est en language Python.