The deep cerebellar nuclei (DCN) function as output gates for a large majority of the Purkinje cells of the cerebellar cortex and thereby determine how the cerebellum influences the rest of the brain and body. In my PhD programme I have investigated how the DCN process two kinds of input patterns received from Purkinje cells: irregularity of spike intervals and pauses in Purkinje cell activity resulting from the recognition of patterns received at the synapses with the upstream parallel fibres (PFs). To that objective I have created a network system of biophysically realistic Purkinje cell and DCN neuron models that enables the exploration of a wide range of network structure and cell physiology parameters. With this system I have performed simulations that show how the DCN neuron changes the information modality of its input, consisting of varying regularity in Purkinje cell spike intervals, to varying spike rates in its output to the nervous system outside of the cerebellum. This was confirmed in simulations where I exchanged the artificial Purkinje cell trains for those received from experimental collaborators. In pattern recognition simulations I have found that the morphological arrangement present in the cerebellum, where multiple Purkinje cells connect to each DCN neuron, has the effect of amplifying pattern recognition already performed in the Purkinje cells. Using the metric of signal-to-noise ratio I show that PF patterns previously encountered and stored in PF - Purkinje cell synapses are most clearly distinguished from those novel to the system by a 10-20 ms shortened burst firing of the DCN neuron. This result suggests that the effect on downstream targets of these excitatory projection neurons is a decreased excitation when a stored as opposed to novel pattern is received. My work has contributed to a better understanding of information processing in the cerebellum, with implications for human motor control as well as the increasingly recognised non-motor functions of the cerebellum.

Electrophysiological recordings are a valuable tool for neuroscience in order to monitor the activity of multiple or even single neurons. Significant insights into the nervous system have been gained by analyses of resulting data; in particular, many findings were gained from spike trains whose correlations can give valuable indications about neural interplay. But detecting, specifying, and representing neural interactions is mathematically challenging. Further, recent advances of recording techniques led to an increase in volume of collected data, which often poses additional computational problems. These developments call for new, improved methods in order to extract crucial information. The matter of this thesis is twofold: It presents a novel method for the analysis of neural spike train data, as well as a generic framework in order to assess the new and related techniques. The new computational method, the Snap Shot Score, can be used to inspect spike trains with respect to temporal dependencies, which are visualised as an information flow network. These networks can specify the relationships in the data, indicate changes in dependencies, and point to causal interactions. The Snap Shot Score is demonstrated to reveal plausible networks both in a variety of simulations and for real data, which indicate its value for understanding neural dynamics. Additional to the Snap Shot Score, a neural simulation framework is suggested, which facilitates the assessment of neural network inference techniques in a highly automated fashion. Due to a new formal concept to rate learned networks, the framework can be used to test techniques under partial observability conditions. In the presence of hidden units quantification of results has been a tedious task that had to be done by hand, but which can now be automated. Thereby high throughput assessments become possible, which facilitate a comprehensive simulation-based characterisation of new methods.

No description available.

This (cumulative) dissertation is concerned with mechanisms and models of information processing and transmission by individual neurons and small neural assemblies. In this document, I first provide historical context for these ideas and highlight similarities and differences to related concepts from machine learning and neuromorphic engineering. With this background, I then discuss the four main themes of my work, namely dendritic filtering and delays, homeostatic plasticity and adaptation, rate-coding with spiking neurons, and spike-timing based alternatives to rate-coding. The content of this discussion is in large part derived from several of my own publications included in Appendix C, but it has been extended and revised to provide a more accessible and broad explanation of the main ideas, as well as to show their inherent connections. I conclude that fundamental differences remain between our understanding of information processing and transmission in machine learning on the one hand and theoretical neuroscience on the other, which should provide a strong incentive for further interdisciplinary work on the domain boundaries between neuroscience, machine learning and neuromorphic engineering.

Recent approaches to IR include neural networks that generate query and document vector representations. The representations are used as the basis for document retrieval and are able to encode semantic features if trained on large datasets, an ability that sets them apart from classical IR approaches such as TF-IDF. However, the datasets necessary to train these networks are not available to the owners of most search services used today, since they are not used by enough users. Thus, methods for enabling the use of neural IR models in data-poor environments are of interest. In this work, a bag-of-trigrams neural IR architecture is used in a transfer learning procedure in an attempt to increase performance on a target dataset by pre-training on external datasets. The target dataset used is WikiQA, and the external datasets are Quora’s Question Pairs, Reuters’ RCV1 and SQuAD. When considering individual model performance, pre-training on Question Pairs and fine-tuning on WikiQA gives us the best individual models. However, when considering average performance, pre-training on the chosen external dataset result in lower performance on the target dataset, both when all datasets are used together and when they are used individually, with different average performance depending on the external dataset used. On average, pre-training on RCV1 and Question Pairs gives the lowest and highest average performance respectively, when considering only the pre-trained networks. Surprisingly, the performance of an untrained, randomly generated network is high, and beats the performance of all pre-trained networks on average. The best performing model on average is a neural IR model trained on the target dataset without prior pre-training. / Nya modeller inom informationssökning inkluderar neurala nät som genererar vektorrepresentationer för sökfrågor och dokument. Dessa vektorrepresentationer används tillsammans med ett likhetsmått för att avgöra relevansen för ett givet dokument med avseende på en sökfråga. Semantiska särdrag i sökfrågor och dokument kan kodas in i vektorrepresentationerna. Detta möjliggör informationssökning baserat på semantiska enheter, vilket ej är möjligt genom de klassiska metoderna inom informationssökning, som istället förlitar sig på den ömsesidiga förekomsten av nyckelord i sökfrågor och dokument. För att träna neurala sökmodeller krävs stora datamängder. De flesta av dagens söktjänster används i för liten utsträckning för att möjliggöra framställande av datamängder som är stora nog att träna en neural sökmodell. Därför är det önskvärt att hitta metoder som möjliggör användadet av neurala sökmodeller i domäner med små tillgängliga datamängder. I detta examensarbete har en neural sökmodell implementerats och använts i en metod avsedd att förbättra dess prestanda på en måldatamängd genom att förträna den på externa datamängder. Måldatamängden som används är WikiQA, och de externa datamängderna är Quoras Question Pairs, Reuters RCV1 samt SquAD. I experimenten erhålls de bästa enskilda modellerna genom att föträna på Question Pairs och finjustera på WikiQA. Den genomsnittliga prestandan över ett flertal tränade modeller påverkas negativt av vår metod. Detta äller både när samtliga externa datamänder används tillsammans, samt när de används enskilt, med varierande prestanda beroende på vilken datamängd som används. Att förträna på RCV1 och Question Pairs ger den största respektive minsta negativa påverkan på den genomsnittliga prestandan. Prestandan hos en slumpmässigt genererad, otränad modell är förvånansvärt hög, i genomsnitt högre än samtliga förtränade modeller, och i nivå med BM25. Den bästa genomsnittliga prestandan erhålls genom att träna på måldatamängden WikiQA utan tidigare förträning.

[Background]: The advent of bidirectional encoder representation from trans- formers (BERT) language models (Devlin et al., 2018) and MS Marco, a large scale human-annotated dataset for machine reading comprehension (Bajaj et al., 2016) that made publicly available, led the field of information retrieval (IR) to experience a revolution (Lin et al., 2020). The retrieval model based on BERT of Nogueira and Cho (2019), by the time they published their paper, became the top entry in the MS Marco passage-reranking leaderboard, surpassing the previous state of the art by 27% in MRR@10. However, training such neural IR models for different domains than MS Marco is still hard because neural approaches often require a vast amount of training data to perform effectively, which is not always available. To address the problem of the shortage of labelled data a new line of research emerged, training neural models with weak supervision. In weak supervision, given an unlabelled dataset labels are generated automatically using an existing model and then a machine learning model is trained upon the artificial “weak“ data. In case of weak supervision for IR, the training dataset comes in the form of a tuple (query, passage). Dehghani et al. (2017) in their work used the AOL query logs (Pass et al., 2006), which is a set of millions of real web queries, and BM25 to retrieve the relevant passages for each of the user queries. A drawback with this approach is that it is hard to obtain query logs for every single different domain. [Objective]: This thesis proposes an intuitive approach for addressing the shortage of data in domains with limited or no data at all through transfer learning in the context of IR. We leverage Wikipedia’s structure for creating a Wikipedia-based generic IR training dataset for zero-shot neural models. [Method]: We create the “pseudo-queries“ by concatenating the titles of Wikipedia’s articles along with each of their title sections and we consider the associated section’s passage as the relevant passage of the pseudo-queries. All of our experiments are evaluated on a standard collection: MS Marco, which is a large scale web collection. For our zero-shot experiments, our proposed model, called “Wiki“, is a BERT model trained on the artificial Wikipedia-based dataset and the baseline is a default BERT model without any additional training. In our second line of experiments, we explore the benefits gained by pre-fine- tuning on the Wikipedia-based IR dataset and further fine-tuning on in-domain data. Our proposed model, "Wiki+Ma", is a BERT model pre-fine-tuned in the Wikipedia-based dataset and further fine-tuned in MS Marco, while the baseline is a BERT model fine-tuned only in MS Marco. [Results]: Results regarding our first experiments show that our BERT model trained on the Wikipedia-based IR dataset, called "Wiki", achieves a performance of 0.197 in MRR@10, which is about +10 points more in comparison to a BERT model with default weights; in addition, results in the development set indicate that the “Wiki“ model performs better than BERT model trained on in-domain data when the data is between 10k-50k instances. Results regarding our second line of experiments show that pre-fine-tuning on the Wikipedia-based IR dataset benefits later fine-tuning steps on in-domain data in terms of stability. [Conclusion]: Our findings suggest that transfer learning for IR tasks by leveraging the generic knowledge incorporated in Wikipedia is possible, though more experimentation is needed to understand its limitations in comparison with the traditional approaches such as the BM25.