Neuroscience as a field is collecting more data than at any other time in history. The scale of this data allows us to ask fundamental questions about the mechanisms of brain function, the basis of behavior, and the development of disorders. Our ambitious goals as well as the abundance of data being recorded call for reproducible, reliable, and accessible systems to push the field forward. While we have made great strides in building reproducible and accessible machine learning (ML) systems for neuroscience, reliability remains a major issue. In this dissertation, we show that we can leverage existing data and domain expert knowledge to build more reliable ML systems to study animal behavior. First, we consider animal pose estimation, a crucial component in many scientific investigations. Typical transfer learning ML methods for behavioral tracking treat each video frame and object to be tracked independently. We improve on this by leveraging the rich spatial and temporal structures pervasive in behavioral videos. Our resulting weakly supervised models achieve significantly more robust tracking. Our tools allow us to achieve improved results when we have imperfect, limited data while requiring users to label fewer training frames and speeding up training. We can more accurately process raw video data and learn interpretable units of behavior. In turn, these improvements enhance performance on downstream applications. Next, we consider a ubiquitous approach to (attempt to) improve the reliability of ML methods, namely combining the predictions of multiple models, also known as deep ensembling. Ensembles of classical ML predictors, such as random forests, improve metrics such as accuracy by well-understood mechanisms such as improving diversity. However, in the case of deep ensembles, there is an open methodological question as to whether, given the choice between a deep ensemble and a single neural network with similar accuracy, one model is truly preferable over the other. Via careful experiments across a range of benchmark datasets and deep learning models, we demonstrate limitations to the purported benefits of deep ensembles. Our results challenge common assumptions regarding the effectiveness of deep ensembles and the “diversity” principles underpinning their success, especially with regards to important metrics for reliability, such as out-of-distribution (OOD) performance and effective robustness. We conduct additional studies of the effects of using deep ensembles when certain groups in the dataset are underrepresented (so-called “long tail” data), a setting whose importance in neuroscience applications is revealed by our aforementioned work. Altogether, our results demonstrate the essential importance of both holistic systems work and fundamental methodological work to understand the best ways to apply the benefits of modern machine learning to the unique challenges of neuroscience data analysis pipelines. To conclude the dissertation, we outline challenges and opportunities in building next-generation ML systems.

Thesis (MEng)--Stellenbosch University, 2014. / ENGLISH ABSTRACT: Muscular dystrophy, spinal cord injury, or amyotrophic lateral sclerosis (ALS) are injuries and disorders that disrupts the neuromuscular channels of the human body thus prohibiting the brain from controlling the body. Brain computer interface (BCI) allows individuals to bypass the neuromuscular channels and interact with the environment using the brain. The system relies on the user manipulating his neural activity in order to control an external device. Electroencephalography (EEG) is a cheap, non-invasive, real time acquisition device used in BCI applications to record neural activity. However, noise, known as artifacts, can contaminate the recording, thus distorting the true neural activity. Eye blinks are a common source of artifacts present in EEG recordings. Due to its large amplitude it greatly distorts the EEG data making it difficult to interpret data for BCI applications. This study proposes a new combination of techniques to detect and correct eye blink artifacts to improve the quality of EEG for BCI applications. Independent component analysis (ICA) is used to separate the EEG signals into independent source components. The source component containing eye blink artifacts are corrected by detecting each eye blink within the source component and using a trained wavelet neural network (WNN) to correct only a segment of the source component containing the eye blink artifact. Afterwards, the EEG is reconstructed without distorting or removing the source component. The results show a 91.1% detection rate and a 97.9% correction rate for all detected eye blinks. Furthermore for channels located over the frontal lobe, eye blink artifacts are corrected preserving the neural activity. The novel combination overall reduces EEG information lost, when compared to existing literature, and is a step towards improving EEG pre-processing in order to provide cleaner EEG data for BCI applications. / AFRIKAANSE OPSOMMING: Spierdistrofie, ’n rugmurgbesering, of amiotrofiese laterale sklerose (ALS) is beserings en steurnisse wat die neuromuskulêre kanale van die menslike liggaam ontwrig en dus verhoed dat die brein die liggaam beheer. ’n Breinrekenaarkoppelvlak laat toe dat die neuromuskulêre kanale omlei word en op die omgewing reageer deur die brein. Die BCI-stelsel vertrou op die gebruiker wat sy eie senuwee-aktiwiteit manipuleer om sodoende ’n eksterne toestel te beheer. Elektro-enkefalografie (EEG) is ’n goedkoop, nie-indringende, intydse dataverkrygingstoestel wat gebruik word in BCI toepassings. Nie net senuwee aktiwiteit nie, maar ook geraas , bekend as artefakte word opgeneem, wat dus die ware senuwee aktiwiteit versteur. Oogknip artefakte is een van die algemene artefakte wat teenwoordig is in EEG opnames. Die groot omvang van hierdie artefakte verwring die EEG data wat dit moeilik maak om die data te ontleed vir BCI toepassings. Die studie stel ’n nuwe kombinasie tegnieke voor wat oogknip artefakte waarneem en regstel om sodoende die kwaliteit van ’n EEG vir BCI toepassings te verbeter. Onafhanklike onderdeel analise (Independent component analysis (ICA)) word gebruik om die EEG seine te skei na onafhanklike bron-komponente. Die bronkomponent wat oogknip artefakte bevat word reggestel binne die komponent en gebruik ’n ervare/geoefende golfsenuwee-netwerk om slegs ’n deel van die komponent wat die oogknip artefak bevat reg te stel. Daarna word die EEG hervorm sonder verwringing of om die bron-komponent te verwyder. Die resultate toon ’n 91.1% opsporingskoers en ’n 97.9% regstellingskoers vir alle waarneembare oogknippe. Oogknip artefakte in kanale op die voorste lob word reggestel en behou die senuwee aktiwiteit wat die oorhoofse EEG kwaliteit vir BCI toepassings verhoog.

Electrophysiological recordings are a valuable tool for neuroscience in order to monitor the activity of multiple or even single neurons. Significant insights into the nervous system have been gained by analyses of resulting data; in particular, many findings were gained from spike trains whose correlations can give valuable indications about neural interplay. But detecting, specifying, and representing neural interactions is mathematically challenging. Further, recent advances of recording techniques led to an increase in volume of collected data, which often poses additional computational problems. These developments call for new, improved methods in order to extract crucial information. The matter of this thesis is twofold: It presents a novel method for the analysis of neural spike train data, as well as a generic framework in order to assess the new and related techniques. The new computational method, the Snap Shot Score, can be used to inspect spike trains with respect to temporal dependencies, which are visualised as an information flow network. These networks can specify the relationships in the data, indicate changes in dependencies, and point to causal interactions. The Snap Shot Score is demonstrated to reveal plausible networks both in a variety of simulations and for real data, which indicate its value for understanding neural dynamics. Additional to the Snap Shot Score, a neural simulation framework is suggested, which facilitates the assessment of neural network inference techniques in a highly automated fashion. Due to a new formal concept to rate learned networks, the framework can be used to test techniques under partial observability conditions. In the presence of hidden units quantification of results has been a tedious task that had to be done by hand, but which can now be automated. Thereby high throughput assessments become possible, which facilitate a comprehensive simulation-based characterisation of new methods.

Spontaneous activity in neuronal networks in vitro is common and has been well documented. However, alteration of spontaneous activity in such networks via conditioning electrical stimulation has received much less experimental attention. Two different patterns of electrical stimulation were used to enhance or depress the level of spontaneous activity in spinal cord cultures. High-frequency stimulation (HFS), a method routinely shown to increase the efficacy of synaptic transmission, was employed to augment spontaneous activity. Low-frequency stimulation (LFS), the technique often applied to depress synaptic efficacy, was employed to decrease spontaneous activity. In addition, LFS was used to reverse the effect of HFS on spontaneous activity. Likewise, HFS was applied to counter the effect of LFS. Because these networks were grown on multi-microelectrode plates (MMEPs), this allowed the simultaneous stimulation of any combination of the 64 electrodes in the array. Thus, the possible differences in response to single versus multi-electrode stimulation were also addressed. Finally, test-pulses were delivered before and after the conditioning stimulation on the same stimulation electrode(s) in order to assess the change in mean evoked action potentials (MEAPs). Dissociated spinal tissue from embryonic mice was allowed to mature into self-organized networks that exhibited spontaneous bursting activity after two weeks of incubation. Spontaneous activity was monitored from up to 14 recording channels simultaneously. Although uniform responses to stimulation across all recording electrodes were rarely observed, a large majority of the recording channels had similar responses. Spontaneous activity was increased in 52% of 89 HFS trials, whereas activity was decreased in 35% of 75 LFS trials. The duration of most of these increases was less than 5 minutes. When there were substantial and long-term (> 15 min) changes in spontaneous activity, the opposing stimulation pattern successfully reversed the effect of the previous stimulation. The percent change in MEAPs following conditioning stimulation suggested that synaptic modification had taken place in 75% of all test-pulse stimulation trials.

A "complex" system typically has a relatively large number of dynamically interacting components and tends to exhibit emergent behavior that cannot be explained by analyzing each component separately. A biological neural network is one example of such a system. A multi-agent model of such a network is developed to study the relationships between a network's structure and its spike train output. Using this model, inferences are made about the synaptic structure of networks through cluster analysis of spike train summary statistics A complexity measure for the network structure is also presented which has a one-to-one correspondence with the standard time series complexity measure sample entropy.

Hybrid neural-microelectronic systems, systems composed of biological neural networks and neuronal models, have great potential for the treatment of neural injury and disease. The utility of such systems will be ultimately determined by the ability of the engineered component to correctly replicate the function of biological neural networks. These models can take the form of mechanistic models, which reproduce neural function by describing the physiologic mechanisms that produce neural activity, and empirical models, which reproduce neural function through more simplified mathematical expressions. We present our research into the role of model complexity in creating robust and flexible behaviors in hybrid systems. Beginning with a complex mechanistic model of a leech heartbeat interneuron, we create a series of three systematically reduced models that incorporate both mechanistic and empirical components. We then evaluate the robustness of these models to parameter variation, and assess the flexibility of the models activities. The modeling studies are validated by incorporating both mechanistic and semi-empirical models in hybrid systems with a living leech heartbeat interneuron. Our results indicate that model complexity serves to increase both the robustness of the system and the ability of the system to produce flexible outputs.

The dynamics of biological neural networks are of great interest to neuroscientists and are frequently studied using conductance-based compartmental neuron models. For speed and ease of use, neuron models are often reduced in morphological complexity. This reduction may affect input processing and prevent the accurate reproduction of neural dynamics. However, such effects are not yet well understood. Therefore, for my first aim I analyzed the processing capabilities of 'branched' or 'unbranched' reduced models by collapsing the dendritic tree of a morphologically realistic 'full' globus pallidus neuron model while maintaining all other model parameters. Branched models maintained the original detailed branching structure of the full model while the unbranched models did not. I found that full model responses to somatic inputs were generally preserved by both types of reduced model but that branched reduced models were better able to maintain responses to dendritic inputs. However, inputs that caused dendritic sodium spikes, for instance, could not be accurately reproduced by any reduced model. Based on my analyses, I provide recommendations on how to construct reduced models and indicate suitable applications for different levels of reduction. In particular, I recommend that unbranched reduced models be used for fast searches of parameter space given somatic input output data. The intrinsic electrical properties of neurons depend on the modifiable behavior of their ion channels. Obtaining a quality match between recorded voltage traces and the output of a conductance based compartmental neuron model depends on accurate estimates of the kinetic parameters of the channels in the biological neuron. Indeed, mismatches in channel kinetics may be detectable as failures to match somatic neural recordings when tuning model conductance densities. In my first aim, I showed that this is a task for which unbranched reduced models are ideally suited. Therefore, for my second aim I optimized unbranched reduced model parameters to match three experimentally characterized globus pallidus neurons by performing two stages of automated searches. In the first stage, I set conductance densities free and found that even the best matches to experimental data exhibited unavoidable problems. I hypothesized that these mismatches were due to limitations in channel model kinetics. To test this hypothesis, I performed a second stage of searches with free channel kinetics and observed decreases in the mismatches from the first stage. Additionally, some kinetic parameters consistently shifted to new values in multiple cells, suggesting the possibility for tailored improvements to channel models. Given my results and the potential for cell specific modulation of channel kinetics, I recommend that experimental kinetic data be considered as a starting point rather than as a gold standard for the development of neuron models.

The work we present in this thesis is a series of studies of how heterogeneities in coupling affect the synchronization of coupled neural oscillators. We begin by examining how heterogeneity in coupling strength affects the equilibrium phase difference of a pair of coupled, spiking neurons when compared to the case of identical coupling. This study is performed using pairs of Hodgkin-Huxley and Wang-Buzsaki neurons. We find that heterogeneity in coupling strength breaks the symmetry of the bifurcation diagrams of equilibrium phase difference versus the synaptic rate constant for weakly coupled pairs of neurons. We observe important qualitative changes such as the loss of the ubiquitous in-phase and anti-phase solutions found when the coupling is identical and regions of parameter space where no phase locked solution exists. Another type of heterogeneity can be found by having different types of coupling between oscillators. Synaptic coupling between neurons can either be exciting or inhibiting. We examine the synchronization dynamics when a pair of neurons is coupled with one excitatory and one inhibitory synapse. We also use coupled pairs of Hodgkin-Huxley neurons and Wang-Buzsaki neurons for this work. We then explore the existance of 1:n coupled states for a coupled pair of theta neurons. We do this in order to reproduce an observed effect called quantal slowing. Quantal slowing is the phenomena where jumping between different $1:n$ coupled states is observed instead of gradual changes in period as a parameter in the system is varied. All of these topics fall under the general heading of coupled, non-linear oscillators and specifically weakly coupled, neural oscillators. The audience for this thesis is most likely going to be a mixed crowd as the research reported herein is interdisciplinary. Choosing the content for the introduction proved far more challenging than expected. It might be impossible to write a maximally useful introductory portion of a thesis when it could be read by a physicist, mathematician, engineer or biologist. Undoubtedly readers will find some portion of this introduction elementary. At the risk of boring some or all of my readers we decided it was best to proceed so that enough of the mathematical (biological) background is explained in the introduction so that a biologist (mathematician) is able to appreciate the motivations for the research and the results presented. We begin with a introduction in nonlinear dynamics explaining the mathematical tools we use to characterize the excitability of individual neurons, as well as oscillations and synchrony in neural networks. The next part of the introductory material is an overview of the biology of neurons. We then describe the neuron models used in this work and finally describe the techniques we employ to study coupled neurons.

The extent to which cortical maps may reorganize in adult humans is a significant and topical debate in visual neuroscience. Though there are conflicting findings, evidence from humans and animals indicates that the topography of the visual cortex may change after retinal deafferentation. Remarkably, this reorganization seems to be possible in adults, whose brains are less amenable to plastic change. If adult visual reorganization is legitimate, an understanding of its causes and consequences could be profound considering the millions suffering from age-related visual disorders. This dissertation explores whether visual training may yield a reorganization of sensory maps in the adult visual cortex. It describes research in which patients, diagnosed with age-related macular degeneration (AMD), underwent visual rehabilitation therapy. Functional brain scans and behavioral tests were conducted pre and post training. These interventions generated valuable knowledge regarding whether "reorganized" activity is a true rewiring of feed forward cortical processes or an artifact of attentional feedback. The rehabilitation training produced demonstrable differences in activation patterns along the primary visual cortex (V1), but sparse improvement in the behavioral tests. In contrast, there was significant improvement in fixation tests which assessed oculomotor control. These results suggest that the nature of reorganized activity has more to do with attentional mechanisms than feed forward reorganization. Future investigations could benefit from examining the brain sites that govern visual attention in the frontal and parietal cortices. These areas may have more to do with visual adaptation in AMD patients than V1.

The hippocampus is necessary for normal memory in rodents, birds, monkeys, and people. Damage to the hippocampus can result in the inability to learn new facts, defined by the relationship among stimuli. In rodents, spatial learning involves learning about the relationships among stimuli, and exemplifies the kind of learning the requires the hippocampus. Therefore, understanding the neural mechanisms underlying spatial learning may elucidate basic memory processes. Many hippocampal neurons fire when behaving rats, cats, or monkeys are in circumscribed regions (place fields) of an environment. The neurons, called place cells, fire in relation to distal stimuli, but can persist in signaling location when the stimuli are removed or lights are turned off (memory fields). In this thesis, computational models of spatial information processing simulated many of the defining properties of hippocampal place cells, including memory fields. Furthermore, the models suggested a neurally plausible mechanism of goal directed spatial navigation which involved the encoding of distances in the connections between place cells. To navigate using memory fields, the models required an excitatory, distributed, and plastic association system among place cells. Such properties are well characterized in area CA3 of the hippocampus. In this thesis, a new electrophysiological study provides evidence that a second system in the dentate gyrus has similar properties. Thus, two circuits in the hippocampus meet the requirements of the models. Some predictions of the models were then tested in a single-unit recording experiment in behaving rats. Place fields were more likely to occur in information rich areas of the environment, and removal of single cues altered place fields in a way consistent with the distance encoding mechanism suggested by the models. It was concluded that a distance encoding theory of rat spatial navigation has much descriptive and predictive utility, but most of its predic