No description available.

abstract: Alternative computation based on neural systems on a nanoscale device are of increasing interest because of the massive parallelism and scalability they provide. Neural based computation systems also offer defect finding and self healing capabilities. Traditional Von Neumann based architectures (which separate the memory and computation units) inherently suffer from the Von Neumann bottleneck whereby the processor is limited by the number of instructions it fetches. The clock driven based Von Neumann computer survived because of technology scaling. However as transistor scaling is slowly coming to an end with channel lengths becoming a few nanometers in length, processor speeds are beginning to saturate. This lead to the development of multi-core systems which process data in parallel, with each core being based on the Von Neumann architecture. The human brain has always been a mystery to scientists. Modern day super computers are outperformed by the human brain in certain computations. The brain occupies far less space and consumes a fraction of the power a super computer does with certain processes such as pattern recognition. Neuromorphic computing aims to mimic biological neural systems on silicon to exploit the massive parallelism that neural systems offer. Neuromorphic systems are event driven systems rather than being clock driven. One of the issues faced by neuromorphic computing was the area occupied by these circuits. With recent developments in the field of nanotechnology, memristive devices on a nanoscale have been developed and show a promising solution. Memristor based synapses can be up to three times smaller than Complementary Metal Oxide Semiconductor (CMOS) based synapses. In this thesis, the Programmable Metallization Cell (a memristive device) is used to prove a learning algorithm known as Spike Time Dependant Plasticity (STDP). This learning algorithm is an extension to Hebb’s learning rule in which the synapses weight can be altered by the relative timing of spikes across it. The synaptic weight with the memristor will be its conductance, and CMOS oscillator based circuits will be used to produce spikes that can modulate the memristor conductance by firing with different phases differences. / Dissertation/Thesis / Masters Thesis Electrical Engineering 2015

O sistema olfatório é notável por sua capacidade de discriminar odores muito similares, mesmo que estejam misturados. Essa capacidade de discriminação é, em parte, devida a padrões de atividade espaço-temporais gerados nas células mitrais, as células principais do bulbo olfatório, durante a apresentação de um odor. Tais padrões dinâmicos decorrem de interações sinápticas recíprocas entre as células mitrais e interneurônios inibitórios do bulbo olfatório, por exemplo, as células granulares. Nesta tese, apresenta-se um modelo do bulbo olfatório baseado em modelos pulsados das células mitrais e granulares e avalia-se o seu desempenho como sistema reconhecedor de padrões usando-se bases de dados de padrões artificiais e reais. Os resultados dos testes mostram que o modelo possui a capacidade de separar padrões em diferentes classes. Essa capacidade pode ser explorada na construção de sistemas reconhecedores de padrões. Apresenta-se também a ferramenta denominada Nemos, desenvolvida para a implementação do modelo, que é uma plataforma para simulação de neurônios e redes de neurônios pulsados com interface gráfica amigável com o usuário. / The olfactory system is a remarkable system capable of discriminating very similar odorant mixtures. This is in part achieved via spatio-temporal activity patterns generated in mitral cells, the principal cells of the olfactory bulb, during odor presentation. Here, we present a spiking neural network model of the olfactory bulb and evaluate its performance as a pattern recognition system with datasets taken from both artificial and real pattern databases. Our results show that the dynamic activity patterns produced in the mitral cells of the olfactory bulb model by pattern attributes presented to it have a pattern separation capability. This capability can be explored in the construction of high-performance pattern recognition systems. Besides, we proposed Nemos a framework for simulation spiking neural networks through graphical user interface and has extensible models for neurons, synapses and networks.

L’hybridation est une technique qui consiste à interconnecter un réseau de neurones biologique et un réseau de neurones artificiel, utilisée dans la recherche en neuroscience et à des fins thérapeutiques. Durant ces trois années de doctorat, ce travail de thèse s’est focalisé sur l’hybridation dans un plan rapproché (communication directe bi-directionnelle entre l’artificiel et le vivant) et dans un plan plus élargies (interopérabilité des systèmes neuromorphiques). Au début des années 2000, cette technique a permis de connecter un système neuromorphique analogique avec le vivant. Ce travail est dans un premier temps, centré autour de la conception d’un réseau de neurones numérique, en vue d’hybridation, dans deux projets multi-disciplinaires en cours dans l’équipe AS2N de l’IMS, présentés dans ce document : HYRENE (ANR 2010-Blan-031601), ayant pour but le développement d’un système hybride de restauration de l’activité motrice dans le cas d’une lésion de la moelle épinière, BRAINBOW (European project FP7-ICT-2011-C), ayant pour objectif l’élaboration de neuro-prothèses innovantes capables de restaurer la communication autour de lésions cérébrales.Possédant une architecture configurable, un réseau de neurones numérique a été réalisé pour ces deux projets. Pour le premier projet, le réseau de neurones artificiel permet d’émuler l’activitéde CPGs (Central Pattern Generator), à l’origine de la locomotion dans le règne animale. Cette activité permet de déclencher une série de stimulations dans la moelle épinière lésée in vitro et de recréer ainsi la locomotion précédemment perdue. Dans le second projet, la topologie du réseau de neurones sera issue de l’analyse et le décryptage des signaux biologiques issues de groupes de neurones cultivés sur des électrodes, ainsi que de modélisations et simulations réalisées par nos partenaires. Le réseau de neurones sera alors capable de réparer le réseau de neurones lésé. Ces travaux de thèse présentent la démarche de conception des deux différents réseaux et des résultats préliminaires obtenus au sein des deux projets. Dans un second temps, ces travaux élargissent l’hybridation à l’interopérabilité des systèmes neuromorphiques. Au travers d’un protocole de communication utilisant Ethernet, il est possible d’interconnecter des réseaux de neurones électroniques, informatiques et biologiques. Dans un futur proche, il permettra d’augmenter la complexité et la taille des réseaux. / HYBRID experiments allow to connect a biological neural network with an artificial one,used in neuroscience research and therapeutic purposes. During these three yearsof PhD, this thesis focused on hybridization in a close-up view (bi-diretionnal direct communication between the artificial and the living) and in a broader view (interoperability ofneuromorphic systems). In the early 2000s, an analog neuromorphic system has been connected to a biological neural network. This work is firstly, about the design of a digital neural network, for hybrid experimentsin two multi-disciplinary projects underway in AS2N team of IMS presented in this document : HYRENE (ANR 2010-Blan-031601), aiming the development of a hybrid system for therestoration of motor activity in the case of a spinal cord lesion,BRAINBOW (European project FP7-ICT-2011-C), aiming the development of innovativeneuro-prostheses that can restore communication around cortical lesions. Having a configurable architecture, a digital neural network was designed for these twoprojects. For the first project, the artificial neural network emulates the activity of CPGs (Central Pattern Generator), causing the locomotion in the animal kingdom. This activity will trigger aseries of stimuli in the injured spinal cord textit in vitro and recreating locomotion previously lost. In the second project, the neural network topology will be determined by the analysis anddecryption of biological signals from groups of neurons grown on electrodes, as well as modeling and simulations performed by our partners. The neural network will be able to repair the injuredneural network. This work show the two different networks design approach and preliminary results obtained in the two projects.Secondly, this work hybridization to extend the interoperability of neuromorphic systems. Through a communication protocol using Ethernet, it is possible to interconnect electronic neuralnetworks, computer and biological. In the near future, it will increase the complexity and size of networks.

Lorsqu'on effectue une tâche, les circuits neuronaux doivent représenter et manipuler des stimuli continus à l'aide de potentiels d'action discrets. On suppose communément que les neurones représentent les quantités continues à l'aide de leur fréquence de décharge et ceci indépendamment les un des autres. Cependant, un tel codage indépendant est inefficace puisqu'il exige la génération d'un très grand nombre de potentiels d'action pour atteindre un certain niveau de précision. Dans ces travaux, on montre que les neurones d'un réseau récurrent peuvent apprendre - à l'aide d'une règle de plasticité locale - à coordonner leurs potentiels d'actions afin de représenter l'information avec une très haute précision tout en déchargeant de façon minimale. La règle d'apprentissage qui agit sur les connexions récurrentes, conduit à un codage efficace en imposant au niveau de chaque neurone un équilibre précis entre excitation et inhibition. Cet équilibre est un phénomène fréquemment observer dans le cerveau et c'est un principe central de notre théorie. On dérive également deux autres règles d'apprentissages biologiquement plausibles qui permettent respectivement au réseau de s'adapter aux statistiques de ses entrées et d'effectuer des transformations complexes et dynamiques sur elles. Finalement, dans ces réseaux, le stochasticité du temps de décharge d'un neurone n'est pas la signature d'un bruit mais au contraire de précision et d'efficacité. Le caractère aléatoire du temps de décharge résulte de la dégénérescence de la représentation. Ceci constitue donc une interprétation radicalement différente et nouvelle de l'irrégularité trouvée dans des trains de potentiels d'actions. / When performing a task, neural circuits must represent and manipulate continuous stimuli using discrete action potentials. It is commonly assumed that neurons represent continuous quantities with their firing rate and this independently from one another. However, such independent coding is very inefficient because it requires the generation of a large number of action potentials in order to achieve a certain level of accuracy. We show that neurons in a spiking recurrent network can learn - using a local plasticity rule - to coordinate their action potentials in order to represent information with high accuracy while discharging minimally. The learning rule that acts on recurrent connections leads to such an efficient coding by imposing a precise balance between excitation and inhibition at the level of each neuron. This balance is a frequently observed phenomenon in the brain and is central in our work. We also derive two biologically plausible learning rules that respectively allows the network to adapt to the statistics of its inputs and to perform complex and dynamic transformations on them. Finally, in these networks, the stochasticity of the spike timing is not a signature of noise but rather of precision and efficiency. In fact, the random nature of the spike times results from the degeneracy of the representation. This constitutes a new and a radically different interpretation of the irregularity found in spike trains.

At a cocktail party, one can choose to scan the room for conversations of interest, attend to a specific conversation partner, switch between conversation partners, or not attend to anything at all. The ability of the normal-functioning auditory system to flexibly listen in complex acoustic scenes plays a central role in solving the cocktail party problem (CPP). In contrast, certain demographics (e.g., individuals with hearing impairment or older adults) are unable to solve the CPP, leading to psychological ailments and reduced quality of life. Since the normal auditory system still outperforms machines in solving the CPP, an effective solution may be found by mimicking the normal-functioning auditory system. Spatial hearing likely plays an important role in CPP-processing in the auditory system. This thesis details the development of a biologically based approach to the CPP by modeling specific neural mechanisms underlying spatial tuning in the auditory cortex. First, we modeled bottom-up, stimulus-driven mechanisms using a multi-layer network model of the auditory system. To convert spike trains from the model output into audible waveforms, we designed a novel reconstruction method based on the estimation of time-frequency masks. We showed that our reconstruction method produced sounds with significantly higher intelligibility and quality than previous reconstruction methods. We also evaluated the algorithm's performance using a psychoacoustic study, and found that it provided the same amount of benefit to normal-hearing listeners as a current state-of-the-art acoustic beamforming algorithm. Finally, we modeled top-down, attention driven mechanisms that allowed the network to flexibly operate in different regimes, e.g., monitor the acoustic scene, attend to a specific target, and switch between attended targets. The model explains previous experimental observations, and proposes candidate neural mechanisms underlying flexible listening in cocktail-party scenarios. The strategies proposed here would benefit hearing-assistive devices for CPP processing (e.g., hearing aids), where users would benefit from switching between various modes of listening in different social situations. / 2022-05-19T00:00:00Z

<div>Neuromorphic computing has gained tremendous interest because of its ability to overcome the limitations of traditional signal processing algorithms in data intensive applications such as image recognition, video analytics, or language translation. The new computing paradigm is built with the goal of achieving high energy efficiency, comparable to biological systems.</div><div>To achieve such energy efficiency, there is a need to explore new neuro-mimetic devices, circuits, and architecture, along with new learning algorithms. To that effect, we propose two main approaches:</div><div><br></div><div>First, we explore an energy-efficient hardware implementation of a bio-plausible Spiking Neural Network (SNN). The key highlights of our proposed system for SNNs are 1) addressing connectivity issues arising from Network On Chip (NOC)-based SNNs, and 2) proposing stochastic CMOS binary SNNs using biased random number generator (BRNG). On-chip Power Line Communication (PLC) is proposed to address the connectivity issues in NOC-based SNNs. PLC can use the on-chip power lines augmented with low-overhead receiver and transmitter to communicate data between neurons that are spatially far apart. We also propose a CMOS '<i>stochastic-bit</i>' with on-chip stochastic Spike Timing Dependent Plasticity (sSTDP) based learning for memory-compressed binary SNNs. A chip was fabricated in 90 nm CMOS process to demonstrate memory-efficient reconfigurable on-chip learning using sSTDP training. </div><div><br></div><div>Second, we explored coupled oscillatory systems for distance computation and convolution operation. Recent research on nano-oscillators has shown the possibility of using coupled oscillator networks as a core computing primitive for analog/non-Boolean computations. Spin-torque oscillator (STO) can be an attractive candidate for such oscillators because it is CMOS compatible, highly integratable, scalable, and frequency/phase tunable. Based on these promising features, we propose a new coupled-oscillator based architecture for hybrid spintronic/CMOS hardware that computes multi-dimensional norm. The hybrid system composed of an array of four injection-locked STOs and a CMOS detector is experimentally demonstrated. Energy and scaling analysis shows that the proposed STO-based coupled oscillatory system has higher energy efficiency compared to the CMOS-based system, and an order of magnitude faster computation speed in distance computation for high dimensional input vectors.</div>

<p>Spiking Neural Networks (SNNs), widely known as the third generation of artificial neural networks, offer a promising solution to approaching the brains' processing capability for cognitive tasks. With more biologically realistic perspective on input processing, SNN performs neural computations using spikes in an event-driven manner. The asynchronous spike-based computing capability can be exploited to achieve improved energy efficiency in neuromorphic hardware. Furthermore, SNN, on account of spike-based processing, can be trained in an unsupervised manner using Spike Timing Dependent Plasticity (STDP). STDP-based learning rules modulate the strength of a multi-bit synapse based on the correlation between the spike times of the input and output neurons. In order to achieve plasticity with compressed synaptic memory, stochastic binary synapse is proposed where spike timing information is embedded in the synaptic switching probability. A bio-plausible probabilistic-STDP learning rule consistent with Hebbian learning theory is proposed to train a network of binary as well as quaternary synapses. In addition, hybrid probabilistic-STDP learning rule incorporating Hebbian and anti-Hebbian mechanisms is proposed to enhance the learnt representations of the stochastic SNN. The efficacy of the presented learning rules are demonstrated for feed-forward fully-connected and residual convolutional SNNs on the MNIST and the CIFAR-10 datasets.<br></p><p>STDP-based learning is limited to shallow SNNs (<5 layers) yielding lower than acceptable accuracy on complex datasets. This thesis proposes block-wise complexity-aware training algorithm, referred to as BlocTrain, for incrementally training deep SNNs with reduced memory requirements using spike-based backpropagation through time. The deep network is divided into blocks, where each block consists of few convolutional layers followed by an auxiliary classifier. The blocks are trained sequentially using local errors from the respective auxiliary classifiers. Also, the deeper blocks are trained only on the hard classes determined using the class-wise accuracy obtained from the classifier of previously trained blocks. Thus, BlocTrain improves the training time and computational efficiency with increasing block depth. In addition, higher computational efficiency is obtained during inference by exiting early for easy class instances and activating the deeper blocks only for hard class instances. The ability of BlocTrain to provide improved accuracy as well as higher training and inference efficiency compared to end-to-end approaches is demonstrated for deep SNNs (up to 11 layers) on the CIFAR-10 and the CIFAR-100 datasets.<br></p><p>Feed-forward SNNs are typically used for static image recognition while recurrent Liquid State Machines (LSMs) have been shown to encode time-varying speech data. Liquid-SNN, consisting of input neurons sparsely connected by plastic synapses to randomly interlinked reservoir of spiking neurons (or liquid), is proposed for unsupervised speech and image recognition. The strength of the synapses interconnecting the input and liquid are trained using STDP, which makes it possible to infer the class of a test pattern without a readout layer typical in standard LSMs. The Liquid-SNN suffers from scalability challenges due to the need to primarily increase the number of neurons to enhance the accuracy. SpiLinC, composed of an ensemble of multiple liquids, where each liquid is trained on a unique input segment, is proposed as a scalable model to achieve improved accuracy. SpiLinC recognizes a test pattern by combining the spiking activity of the individual liquids, each of which identifies unique input features. As a result, SpiLinC offers comparable accuracy to Liquid-SNN with added synaptic sparsity and faster training convergence, which is validated on the digit subset of TI46 speech corpus and the MNIST dataset.</p>

Cette dernière décennie a donné lieu à la réémergence des méthodes d'apprentissage machine basées sur les réseaux de neurones formels sous le nom d'apprentissage profond. Bien que ces méthodes aient permis des avancées majeures dans le domaine de l'apprentissage machine, plusieurs obstacles à la possibilité d'industrialiser ces méthodes persistent, notamment la nécessité de collecter et d'étiqueter une très grande quantité de données ainsi que la puissance de calcul nécessaire pour effectuer l'apprentissage et l'inférence avec ce type de réseau neuronal. Dans cette thèse, nous proposons d'étudier l'adéquation entre des algorithmes d'inférence et d'apprentissage issus des réseaux de neurones biologiques pour des architectures matérielles massivement parallèles. Nous montrons avec trois contributions que de telles adéquations permettent d'accélérer drastiquement les temps de calculs inhérents au réseaux de neurones. Dans notre premier axe, nous réalisons l'étude d'adéquation du moteur BCVision de Brainchip SAS pour les plate-formes GPU. Nous proposons également l'introduction d'une architecture hiérarchique basée sur des cellules complexes. Nous montrons que l'adéquation pour GPU accélère les traitements par un facteur sept, tandis que l'architecture hiérarchique atteint un facteur mille. La deuxième contribution présente trois algorithmes de propagation de décharges neuronales adaptés aux architectures parallèles. Nous réalisons une étude complète des modèles computationels de ces algorithmes, permettant de sélectionner ou de concevoir un système matériel adapté aux paramètres du réseau souhaité. Dans notre troisième axe nous présentons une méthode pour appliquer la règle Spike-Timing-Dependent-Plasticity à des données images afin d'apprendre de manière non-supervisée des représentations visuelles. Nous montrons que notre approche permet l'apprentissage d'une hiérarchie de représentations pertinente pour des problématiques de classification d'images, tout en nécessitant dix fois moins de données que les autres approches de la littérature. / The last decade has seen the re-emergence of machine learning methods based on formal neural networks under the name of deep learning. Although these methods have enabled a major breakthrough in machine learning, several obstacles to the possibility of industrializing these methods persist, notably the need to collect and label a very large amount of data as well as the computing power necessary to perform learning and inference with this type of neural network. In this thesis, we propose to study the adequacy between inference and learning algorithms derived from biological neural networks and massively parallel hardware architectures. We show with three contribution that such adequacy drastically accelerates computation times inherent to neural networks. In our first axis, we study the adequacy of the BCVision software engine developed by Brainchip SAS for GPU platforms. We also propose the introduction of a coarse-to-fine architecture based on complex cells. We show that GPU portage accelerates processing by a factor of seven, while the coarse-to-fine architecture reaches a factor of one thousand. The second contribution presents three algorithms for spike propagation adapted to parallel architectures. We study exhaustively the computational models of these algorithms, allowing the selection or design of the hardware system adapted to the parameters of the desired network. In our third axis we present a method to apply the Spike-Timing-Dependent-Plasticity rule to image data in order to learn visual representations in an unsupervised manner. We show that our approach allows the effective learning a hierarchy of representations relevant to image classification issues, while requiring ten times less data than other approaches in the literature.

Biological neural systems (BNS) in general and the central nervous system (CNS) specifically exhibit a strikingly efficient computational power along with an extreme flexible and adaptive basis for acquiring and integrating new knowledge. Acquiring more insights into the actual mechanisms of information processing within the BNS and their computational capabilities is a core objective of modern computer science, computational sciences and neuroscience. Among the main reasons of this tendency to understand the brain is to help in improving the quality of life of people suffer from loss (either partial or complete) of brain or spinal cord functions. Brain-computer-interfaces (BCI), neural prostheses and other similar approaches are potential solutions either to help these patients through therapy or to push the progress in rehabilitation. There is however a significant lack of knowledge regarding the basic information processing within the CNS. Without a better understanding of the fundamental operations or sequences leading to cognitive abilities, applications like BCI or neural prostheses will keep struggling to find a proper and systematic way to help patients in this regard. In order to have more insights into these basic information processing methods, this thesis presents an approach that makes a formal distinction between the essence of being intelligent (as for the brain) and the classical class of artificial intelligence, e.g. with expert systems. This approach investigates the underlying mechanisms allowing the CNS to be capable of performing a massive amount of computational tasks with a sustainable efficiency and flexibility. This is the essence of being intelligent, i.e. being able to learn, adapt and to invent. The approach used in the thesis at hands is based on the hypothesis that the brain or specifically a biological neural circuitry in the CNS is a dynamic system (network) that features emergent capabilities. These capabilities can be imported into spiking neural networks (SNN) by emulating the dynamic neural system. Emulating the dynamic system requires simulating both the inner workings of the system and the framework of performing the information processing tasks. Thus, this work comprises two main parts. The first part is concerned with introducing a proper and a novel dynamic synaptic model as a vital constitute of the inner workings of the dynamic neural system. This model represents a balanced integration between the needed biophysical details and being computationally inexpensive. Being a biophysical model is important to allow for the abilities of the target dynamic system to be inherited, and being simple is needed to allow for further implementation in large scale simulations and for hardware implementation in the future. Besides, the energy related aspects of synaptic dynamics are studied and linked to the behaviour of the networks seeking for stable states of activities. The second part of the thesis is consequently concerned with importing the processing framework of the dynamic system into the environment of SNN. This part of the study investigates the well established concept of binding by synchrony to solve the information binding problem and to proposes the concept of synchrony states within SNN. The concepts of computing with states are extended to investigate a computational model that is based on the finite-state machines and reservoir computing. Biological plausible validations of the introduced model and frameworks are performed. Results and discussions of these validations indicate that this study presents a significant advance on the way of empowering the knowledge about the mechanisms underpinning the computational power of CNS. Furthermore it shows a roadmap on how to adopt the biological computational capabilities in computation science in general and in biologically-inspired spiking neural networks in specific. Large scale simulations and the development of neuromorphic hardware are work-in-progress and future work. Among the applications of the introduced work are neural prostheses and bionic automation systems.