<p>This thesis explores the feasibility of Artificial Grammar (AG) recognition using spiking neural networks. A biologically inspired minicolumn model is designed as the base computational unit. Two network topographies are defined with different ideologies. Both networks consists of minicolumn models, referred to as nodes, connected with excitatory and inhibitory connections. The first network contains nodes for every bigram and trigram producible by the grammar’s finite state machine (FSM). The second network has only nodes required to identify unique internal states of the FSM. The networks produce predictable activity for tested input strings. Future work to improve the performance of the networks is discussed. The modeling framework developed can be used by neurophysiological research to implement network layouts and compare simulated performance characteristics to actual subject performance.</p>

Our senses are our mind's window to the outside world and determine how we perceive our environment.Sensory systems are complex multi-level systems that have to solve a multitude of tasks that allow us to understand our surroundings.However, questions on various levels and scales remain to be answered ranging from low-level neural responses to behavioral functions on the highest level.Modeling can connect different scales and contribute towards tackling these questions by giving insights into perceptual processes and interactions between processing stages.In this thesis, numerical simulations of spiking neural networks are used to deal with two essential functions that sensory systems have to solve: pattern recognition and prediction.The focus of this thesis lies on the question as to how neural network connectivity can be used in order to achieve these crucial functions.The guiding ideas of the models presented here are grounded in the probabilistic interpretation of neural signals, Hebbian learning principles and connectionist ideas.The main results are divided into four parts.The first part deals with the problem of pattern recognition in a multi-layer network inspired by the early mammalian olfactory system with biophysically detailed neural components.Learning based on Hebbian-Bayesian principles is used to organize the connectivity between and within areas and is demonstrated in behaviorally relevant tasks.Besides recognition of artificial odor patterns, phenomena like concentration invariance, noise robustness, pattern completion and pattern rivalry are investigated.It is demonstrated that learned recurrent cortical connections play a crucial role in achieving pattern recognition and completion.The second part is concerned with the prediction of moving stimuli in the visual system.The problem of motion-extrapolation is studied using different recurrent connectivity patterns.The main result shows that connectivity patterns taking the tuning properties of cells into account can be advantageous for solving the motion-extrapolation problem.The third part focuses on the predictive or anticipatory response to an approaching stimulus.Inspired by experimental observations, particle filtering and spiking neural network frameworks are used to address the question as to how stimulus information is transported within a motion sensitive network.In particular, the question if speed information is required to build up a trajectory dependent anticipatory response is studied by comparing different network connectivities.Our results suggest that in order to achieve a dependency of the anticipatory response to the trajectory length, a connectivity that uses both position and speed information seems necessary.The fourth part combines the self-organization ideas from the first part with motion perception as studied in the second and third parts.There, the learning principles used in the olfactory system model are applied to the problem of motion anticipation in visual perception.Similarly to the third part, different connectivities are studied with respect to their contribution to anticipate an approaching stimulus.The contribution of this thesis lies in the development and simulation of large-scale computational models of spiking neural networks solving prediction and pattern recognition tasks in biophysically plausible frameworks. / <p>QC 20150504</p>

Biological neural systems (BNS) in general and the central nervous system (CNS) specifically exhibit a strikingly efficient computational power along with an extreme flexible and adaptive basis for acquiring and integrating new knowledge. Acquiring more insights into the actual mechanisms of information processing within the BNS and their computational capabilities is a core objective of modern computer science, computational sciences and neuroscience. Among the main reasons of this tendency to understand the brain is to help in improving the quality of life of people suffer from loss (either partial or complete) of brain or spinal cord functions. Brain-computer-interfaces (BCI), neural prostheses and other similar approaches are potential solutions either to help these patients through therapy or to push the progress in rehabilitation. There is however a significant lack of knowledge regarding the basic information processing within the CNS. Without a better understanding of the fundamental operations or sequences leading to cognitive abilities, applications like BCI or neural prostheses will keep struggling to find a proper and systematic way to help patients in this regard. In order to have more insights into these basic information processing methods, this thesis presents an approach that makes a formal distinction between the essence of being intelligent (as for the brain) and the classical class of artificial intelligence, e.g. with expert systems. This approach investigates the underlying mechanisms allowing the CNS to be capable of performing a massive amount of computational tasks with a sustainable efficiency and flexibility. This is the essence of being intelligent, i.e. being able to learn, adapt and to invent. The approach used in the thesis at hands is based on the hypothesis that the brain or specifically a biological neural circuitry in the CNS is a dynamic system (network) that features emergent capabilities. These capabilities can be imported into spiking neural networks (SNN) by emulating the dynamic neural system. Emulating the dynamic system requires simulating both the inner workings of the system and the framework of performing the information processing tasks. Thus, this work comprises two main parts. The first part is concerned with introducing a proper and a novel dynamic synaptic model as a vital constitute of the inner workings of the dynamic neural system. This model represents a balanced integration between the needed biophysical details and being computationally inexpensive. Being a biophysical model is important to allow for the abilities of the target dynamic system to be inherited, and being simple is needed to allow for further implementation in large scale simulations and for hardware implementation in the future. Besides, the energy related aspects of synaptic dynamics are studied and linked to the behaviour of the networks seeking for stable states of activities. The second part of the thesis is consequently concerned with importing the processing framework of the dynamic system into the environment of SNN. This part of the study investigates the well established concept of binding by synchrony to solve the information binding problem and to proposes the concept of synchrony states within SNN. The concepts of computing with states are extended to investigate a computational model that is based on the finite-state machines and reservoir computing. Biological plausible validations of the introduced model and frameworks are performed. Results and discussions of these validations indicate that this study presents a significant advance on the way of empowering the knowledge about the mechanisms underpinning the computational power of CNS. Furthermore it shows a roadmap on how to adopt the biological computational capabilities in computation science in general and in biologically-inspired spiking neural networks in specific. Large scale simulations and the development of neuromorphic hardware are work-in-progress and future work. Among the applications of the introduced work are neural prostheses and bionic automation systems.

Locomotion is an integral part of a whole range of animal behaviours. The basic rhythm for locomotion in vertebrates has been shown to arise from local networks residing in the spinal cord and these networks are known as central pattern generators (CPG). However, during the locomotion, these centres are constantly interacting with the sensory feedback signals coming from muscles, joints and peripheral skin receptors in order to adapt the stepping or swimming to varying environmental conditions. Conceptual models of vertebrate locomotion have been constructed using mathematical models of locomotor subsystems based on the neurophysiological evidence obtained primarily in the cat and the salamander, an amphibian with a sprawling posture. Such models provide opportunity for studying the key elements in the transition from aquatic to terrestrial locomotion. Several aspects of locomotor control using the cat or the salamander as an animal model have been investigated employing computer simulations and here we use the same approach to address a number of questions or/and hypotheses related to rhythmic locomotion in quadrupeds. Some of the involved questions are, the role of mechanical linkage during deafferented walking, finding inherent stabilities/instabilities of muscle-joint interactions during normal walking and estimating phase dependent controlability of muscle action over joints. Also we investigate limb and body coordination for different gaits, use of side-stepping in front limbs for turning and the role of sensory feedback in gait generation and transitions in salamanders.      This thesis presents the basics of the biologically realistic models of cat and salamander locomotion and summarizes computational methods in modeling quadruped locomotor subsystems such as CPG, limb muscles and sensory pathways. In the case of cat hind limb, we conclude that the mechanical linkages between the legs play a major role in producing the alternating gait. In another experiment we use the model to identify open-loop linear transfer functions between muscle activations and joint angles while ongoing locomotion. We hypothesize that the musculo-skeletal system for locomotion in animals, at least in cats, operates under critically damped condition.      The 3D model of the salamander is successfully used to mimic locomotion on level ground and in water. We compare the walking gait with the trotting gait in simulations. We also found that for turning, the use of side-stepping alone or in combination with trunk bending is more effective than the use of trunk bending alone. The same model together with a more realistic CPG composed of spiking neurons was used to investigate the role of sensory feedback in gait generation and transition. We found that the proprioceptive sensory inputs are essential in obtaining the walking gait, whereas the trotting gait is more under central (CPG) influence compared to that of the peripheral or sensory feedback.      This thesis work sheds light on understanding the neural control mechanisms behind vertebrate locomotion. Additionally, both neuro-mechanical models can be used for further investigations in finding new control algorithms which give robust, adaptive, efficient and realistic stepping in each leg, which would be advantageous since it can be implemented on a controller of a quadruped-robotic device. / This work is Funded by Swedish International Development cooperation Agency (SIDA). QC 20111110

Ecological modelling problems have characteristics both featured in other modelling fields and specific ones, hence, methods developed and tested in other research areas may not be suitable for modelling ecological problems or may perform poorly when used on ecological data. This thesis identifies issues associated with the techniques typically used for solving ecological problems and develops new generic methods for decision support, especially suitable for ecological data modelling, which are characterised by: (1) adaptive learning, (2) knowledge discovery and (3) accurate prediction. These new methods have been successfully applied to challenging real world ecological problems. Despite the fact that the number of possible applications of computational intelligence methods in ecology is vast, this thesis primarily concentrates on two problems: (1) species establishment prediction and (2) environmental monitoring. Our review of recent papers suggests that multi-layer perceptron networks trained using the backpropagation algorithm are most widely used of all artificial neural networks for forecasting pest insect invasions. While the multi-layer perceptron networks are appropriate for modelling complex nonlinear relationships, they have rather limited exploratory capabilities and are difficult to adapt to dynamically changing data. In this thesis an approach that addresses these limitations is proposed. We found that environmental monitoring applications could benefit from having an intelligent taste recognition system possibly embedded in an autonomous robot. Hence, this thesis reviews the current knowledge on taste recognition and proposes a biologically inspired artificial model of taste recognition based on biologically plausible spiking neurons. The model is dynamic and is capable of learning new tastants as they become available. Furthermore, the model builds a knowledge base that can be extracted during or after the learning process in form of IF-THEN fuzzy rules. It also comprises a layer that simulates the influence of taste receptor cells on the activity of their adjacent cells. These features increase the biological relevance of the model compared to other current taste recognition models. The proposed model was implemented in software on a single personal computer and in hardware on an Altera FPGA chip. Both implementations were applied to two real-world taste datasets.In addition, for the first time the applicability of transductive reasoning for forecasting the establishment potential of pest insects into new locations was investigated. For this purpose four types of predictive models, built using inductive and transductive reasoning, were used for predicting the distributions of three pest insects. The models were evaluated in terms of their predictive accuracy and their ability to discover patterns in the modelling data. The results obtained indicate that evolving connectionist systems can be successfully used for building predictive distribution models and environmental monitoring systems. The features available in the proposed dynamic systems, such as on-line learning and knowledge discovery, are needed to improve our knowledge of the species distributions. This work laid down the foundation for a number of interesting future projects in the field of ecological modelling, robotics, pervasive computing and pattern recognition that can be undertaken separately or in sequence.

This thesis explores the feasibility of Artificial Grammar (AG) recognition using spiking neural networks. A biologically inspired minicolumn model is designed as the base computational unit. Two network topographies are defined with different ideologies. Both networks consists of minicolumn models, referred to as nodes, connected with excitatory and inhibitory connections. The first network contains nodes for every bigram and trigram producible by the grammar’s finite state machine (FSM). The second network has only nodes required to identify unique internal states of the FSM. The networks produce predictable activity for tested input strings. Future work to improve the performance of the networks is discussed. The modeling framework developed can be used by neurophysiological research to implement network layouts and compare simulated performance characteristics to actual subject performance.

Artificial Neural Networks (ANNs) appear increasingly and routinely to gain popularity today, as they are being used in several diverse research fields and many different contexts, which may range from biological simulations and experiments on artificial neuronal models to machine learning models intended for industrial and engineering applications. One example is the recent success of Deep Learning architectures (e.g., Deep Belief Networks [DBN]), which appear in the spotlight of machine learning research, as they are capable of delivering state-of-the-art results in many domains. While the performance of such ANN architectures is greatly affected by their scale, their capacity for scalability both for training and during execution is limited by the increased power consumption and communication overheads, implicitly posing a limiting factor on their real-time performance. The on-going work on the design and construction of spike-based neuromorphic platforms offers an alternative for running large-scale neural networks, such as DBNs, with significantly lower power consumption and lower latencies, but has to overcome the hardware limitations and model specialisations imposed by these type of circuits. SpiNNaker is a novel massively parallel fully programmable and scalable architecture designed to enable real-time spiking neural network (SNN) simulations. These properties render SpiNNaker quite an attractive neuromorphic exploration platform for running large-scale ANNs, however, it is necessary to investigate thoroughly both its power requirements as well as its communication latencies. This research focusses on around two main aspects. First, it aims at characterising the power requirements and communication latencies of the SpiNNaker platform while running large-scale SNN simulations. The results of this investigation lead to the derivation of a power estimation model for the SpiNNaker system, a reduction of the overall power requirements and the characterisation of the intra- and inter-chip spike latencies. Then it focuses on a full characterisation of spiking DBNs, by developing a set of case studies in order to determine the impact of (a) the hardware bit precision; (b) the input noise; (c) weight variation; and (d) combinations of these on the classification performance of spiking DBNs for the problem of handwritten digit recognition. The results demonstrate that spiking DBNs can be realised on limited precision hardware platforms without drastic performance loss, and thus offer an excellent compromise between accuracy and low-power, low-latency execution. These studies intend to provide important guidelines for informing current and future efforts around developing custom large-scale digital and mixed-signal spiking neural network platforms.

Reverse-engineering the brain is one of the US National Academy of Engineering’s “Grand Challenges.” The structure of the brain can be examined at many different levels, spanning many disciplines from low-level biology through psychology and computer science. This thesis focusses on real-time computation of large neural networks using the Izhikevich spiking neuron model. Neural computation has been described as “embarrassingly parallel” as each neuron can be thought of as an independent system, with behaviour described by a mathematical model. However, the real challenge lies in modelling neural communication. While the connectivity of neurons has some parallels with that of electrical systems, its high fan-out results in massive data processing and communication requirements when modelling neural communication, particularly for real-time computations. It is shown that memory bandwidth is the most significant constraint to the scale of real-time neural computation, followed by communication bandwidth, which leads to a decision to implement a neural computation system on a platform based on a network of Field Programmable Gate Arrays (FPGAs), using commercial off- the-shelf components with some custom supporting infrastructure. This brings implementation challenges, particularly lack of on-chip memory, but also many advantages, particularly high-speed transceivers. An algorithm to model neural communication that makes efficient use of memory and communication resources is developed and then used to implement a neural computation system on the multi- FPGA platform. Finding suitable benchmark neural networks for a massively parallel neural computation system proves to be a challenge. A synthetic benchmark that has biologically-plausible fan-out, spike frequency and spike volume is proposed and used to evaluate the system. It is shown to be capable of computing the activity of a network of 256k Izhikevich spiking neurons with a fan-out of 1k in real-time using a network of 4 FPGA boards. This compares favourably with previous work, with the added advantage of scalability to larger neural networks using more FPGAs. It is concluded that communication must be considered as a first-class design constraint when implementing massively parallel neural computation systems.

A Chip Multiprocessor for a Large-scale Neural SimulatorEustace PainkrasA thesis submitted to The University of Manchesterfor the degree of Doctor of Philosophy, 17 December 2012The modelling and simulation of large-scale spiking neural networks in biologicalreal-time places very high demands on computational processing capabilities andcommunications infrastructure. These demands are difficult to satisfy even with powerfulgeneral-purpose high-performance computers. Taking advantage of the remarkableprogress in semiconductor technologies it is now possible to design and buildan application-driven platform to support large-scale spiking neural network simulations.This research investigates the design and implementation of a power-efficientchip multiprocessor (CMP) which constitutes the basic building block of a spikingneural network modelling and simulation platform. The neural modelling requirementsof many processing elements, high-fanout communications and local memoryare addressed in the design and implementation of the low-level modules in the designhierarchy as well as in the CMP. By focusing on a power-efficient design, the energyconsumption and related cost of SpiNNaker, the massively-parallel computation engine,are kept low compared with other state-of-the-art hardware neural simulators.The SpiNNaker CMP is composed of many simple power-efficient processors withsmall local memories, asynchronous networks-on-chip and numerous bespoke modulesspecifically designed to serve the demands of neural computation with a globallyasynchronous, locally synchronous (GALS) architecture.The SpiNNaker CMP, realised as part of this research, fulfills the demands of neuralsimulation in a power-efficient and scalable manner, with added fault-tolerancefeatures. The CMPs have, to date, been incorporated into three versions of SpiNNakersystem PCBs with up to 48 chips onboard. All chips on the PCBs are performing successfully, during both functional testing and their targeted role of neural simulation.

The Internet of Things (IoT) computing paradigm has connected smart objects “things” and has brought new services at the proximity of the user. Edge Computing, a natural evolution of the traditional IoT, has been proposed to deal with the ever-increasing (i) number of IoT devices and (ii) the amount of data traffic that is produced by the IoT endpoints. EC promises to significantly reduce the unwanted latency that is imposed by the multi-hop communication delays and suggests that instead of uploading all the data to the remote cloud for further processing, it is beneficial to perform computation at the “edge” of the network, close to where the data is produced. However, bringing computation at the edge level has created numerous challenges as edge devices struggle to keep up with the growing application requirements (e.g. Neural Networks, or video-based analytics). In this thesis, we adopt the EC paradigm and we aim at addressing the open challenges. Our goal is to bridge the performance gap that is caused by the increased requirements of the IoT applications with respect to the IoT platform capabilities and provide latency- and energy-efficient computation at the edge level. Our first step is to study the performance of IoT applications that are based on Deep Neural Networks (DNNs). The exploding need to deploy DNN-based applications on resource-constrained edge devices has created several challenges, mainly due to the complex nature of DNNs. DNNs are becoming deeper and wider in order to fulfill users expectations for high accuracy, while they also become power hungry. For instance, executing a DNN on an edge device can drain the battery within minutes. Our solution to make DNNs more energy and inference friendly is to propose hardware-aware method that re-designs a given DNN architecture. Instead of proxy metrics, we measure the DNN performance on real edge devices and we capture their energy and inference time. Our method manages to find alternative DNN architectures that consume up to 78.82% less energy and are up to35.71% faster than the reference networks. In order to achieve end-to-end optimal performance, we also need to manage theedge device resources that will execute a DNN-based application. Due to their unique characteristics, we distinguish the edge devices into two categories: (i) a neuromorphic platform that is designed to execute Spiking Neural Networks (SNNs), and (ii) a general-purpose edge device that is suitable to host a DNN. For the first category, we train a traditional DNN and then we convert it to a spiking representation. We target the SpiNNaker neuromorphic platform and we develop a novel technique that efficiently configures the platform-dependent parameters, in order to achieve the highest possible SNN accuracy.Experimental results show that our technique is 2.5× faster than an exhaustive approach and can reach up to 0.8% higher accuracy compared to a CPU-based simulation method. Regarding the general-purpose edge devices, we show that a DNN-unaware platform can result in sub-optimal DNN performance in terms of power and inference time. Our approachconfigures the frequency of the device components (GPU, CPU, Memory) and manages to achieve average of 33.4% and up to 66.3% inference time improvements and an average of 42.8% and up to 61.5% power savings compared to the predefined configuration of an edge device. The last part of this thesis is the offloading optimization between the edge devicesand the gateway. The offloaded tasks create contention effects on gateway, which can lead to application slowdown. Our proposed solution configures (i) the number of application stages that are executed on each edge device, and (ii) the achieved utility in terms of Quality of Service (QoS) on each edge device. Our technique manages to (i) maximize theoverall QoS, and (ii) simultaneously satisfy network constraints (bandwidth) and user expectations (execution time). In case of multi-gateway deployments, we tackled the problem of unequal workload distribution. In particular, we propose a workload-aware management scheme that performs intra- and inter-gateway optimizations. The intra-gateway mechanism provides a balanced execution environment for the applications, and it achieves up to 95% performance deviation improvement, compared to un-optimized systems. The presented inter-gateway method manages to balance the workload among multiple gateways and is able to achieve a global performance threshold.