Reinforcement learning algorithms comprise a class of learning algorithms for neural networks. Reinforcement learning is distinguished from other classes by the type of problems that it is intended to solve. It is used for learning input-output mappings where the desired outputs are not known and only a scalar reinforcement value is available. Primary Reinforcement Learning (PRL) is a core component of the most actively researched form of reinforcement learning. The issues surrounding the convergence characteristics of PRL are considered in this thesis. There have been no convergence proofs for any kind of networks learning under PRL. A convergence theorem is proved in this thesis, showing that under some conditions, a particular reinforcement learning algorithm, the A[formula omitted] algorithm, will train a single-layer network correctly. The theorem is demonstrated with a series of simulations. A new PRL algorithm is proposed to deal with the training of multiple layer, binary output networks with continuous inputs. This is a more difficult learning problem than with binary inputs. The new algorithm is shown to be able to successfully train a network with multiple outputs when the environment conforms to the conditions of the convergence theorem for a single-layer network. / Applied Science, Faculty of / Electrical and Computer Engineering, Department of / Graduate

Despite considerable progress in mapping the fly’s connectome and elucidating the patterns of information flow in its brain, the complexity of the fly brain’s structure and the still-incomplete state of knowledge regarding its neural circuitry pose significant challenges beyond satisfying the computational resource requirements of current fly brain models that must be addressed to successfully reverse the information processing capabilities of the fly brain. These include the need to explicitly facilitate collaborative development of brain models by combining the efforts of multiple researchers, and the need to enable programmatic generation of brain models that effectively utilize the burgeoning amount of increasingly detailed publicly available fly connectome data. This thesis presents an open pipeline for modular construction of executable models of the fruit fly brain from incomplete biological brain data that addresses both of the above requirements. This pipeline consists of two major open-source components respectively called Neurokernel and NeuroArch. Neurokernel is a framework for collaborative construction of executable connectome-based fly brain models by integration of independently developed models of different functional units in the brain into a single emulation that can be executed upon multiple Graphics Processing Units (GPUs). Neurokernel enforces a programming model that enables functional unit models that comply with its interface requirements to communicate during execution regardless of their internal design. We demonstrate the power of this programming model by using it to integrate independently developed models of the fly retina and lamina into a single vision processing system. We also show how Neurokernel’s communication performance can scale over multiple GPUs, number of functional units in a brain emulation, and over the number of communication ports exposed by a functional unit model. Although the increasing amount of experimentally obtained biological data regarding the fruit fly brain affords brain modelers a potentially valuable resource for model development, the actual use of this data to construct executable neural circuit models is currently challenging because of the disparate nature of different data sources, the range of storage formats they use, and the limited query features of those formats complicates the process of inferring executable circuit designs from biological data. To overcome these limitations, we created a software package called NeuroArch that defines a data model for concurrent representation of both biological data and model structure and the relationships between them within a single graph database. Coupled with a powerful interface for querying both types of data within the database in a uniform high-level manner, this representation enables construction and dispatching of executable neural circuits to Neurokernel for execution and evaluation. We demonstrate the utility of the NeuroArch/Neurokernel pipeline by using the packages to generate an executable model of the central complex of the fruit fly brain from both published and hypothetical data regarding overlapping neuron arborizations in different regions of the central complex neuropils. We also show how the pipeline empowers circuit model designers to devise computational analogues to biological experiments such as parallel concurrent recording from multiple neurons and emulation of genetic mutations that alter the fly’s neural circuitry.