National and Kapodistrian University of Athens
Non-parametric Bayesian approaches to deep neural networks
Abstract
dc:descriptionDespite the recent successes in Machine Learning, there remain many open challenges. The goal of this thesis is to introduce two different design paradigms for both batch as well as sequential data. Each of the considered approaches comprises the construction of more flexible and efficient models, aspiring to increase the modeling capacity while addressing the complexity issues of the conventionally employed approaches. The thesis initially focuses on the batch scenario, where we consider Deep Neural architectures. Recent hardware advances, as well as the availability of large datasets, have enabled the construction of networks, that comprise a large number of neurons, connected via synaptic weights, in a layered fashion. When a multi-layer (deep) architecture is considered, the network is called a Deep Neural Network, and the corresponding discipline is referred to as Deep Learning. Application of DL approaches, has provided significant achievements in a wide range of domains. However, the currently employed deep architectures, suffer from over-parameterization; usually, the models entail millions of parameters, many of which are redundant. This leads to unnecessary computational complexity, limiting their scalability, while at the same time contributing to the well-known over-fitting tendencies of deep networks, undermining their generalization capabilities. Moreover, it is generally known that, even though the widely used non-linearities, such as the Rectified Linear Units, are a convenient mathematical tool, allowing the effective training of deep architectures, they do not come with a biologic plausibility. Indeed, there is an increasing body of evidence that neurons in a biological system that have similar functional properties are aggregated together, and local competition takes place among them. To this end, researchers have devoted significant effort to design more biologically plausible networks where instead of employing the common non-linearities, competition among linear units takes place. The resulting competition mechanism is called Winner-Takes-All and has provided significant results in the community. In this context, we revisit the current design paradigms of deep neural networks, aiming to introduce a novel, principled approach for network pruning and compression based on biologically inspired Local Winner-Takes-All deep networks. To this end, we propose an inferential construction for explicitly inferring the utility of network components (connections/units) in the context of LWTA-based networks. We employ appropriate arguments from the solid nonparametric Bayesian framework, namely stick-breaking priors. The introduced mechanism constitutes a data-driven procedure, allowing the model structure to intelligently adapt to the complexity of the data, while simultaneously inferring the needed floating point precision to represent its weights. We tie all the components under the variational inference framework, and we derive efficient training and inference procedures for our model. We demonstrate the capacity of our approach in a supervised classification setting in a variety of benchmark architectures and datasets. The empirical evidence vouch for the efficacy of the approach, yielding predictive accuracy at least competitive with the state-of-the-art, while allowing pruning and compressing the network much better than the competition. Furthermore, our closer examination of the behavior of the employed LWTA mechanism suggests that the considered competition-based approach succeeds in encoding salient distinctive features through the winner selection mechanism.In the second approach, we consider sequential data, that still remain one of the most challenging tasks in the Machine Learning community. Traditionally, Hidden Markov Models and their variants, were used to model such data, constituting a flexible and powerful approach. However, through the recent advances in deep architectures, Hidden Markov Models have largely been replaced by their deep alternatives, namely Recurrent Neural Networks and their variants. Even though, the inherent structure of RNNs and the employed non-linearities therein, allow for updating the states in more complicated ways, the discrete bottlenecks and generative structure of HMMs, allows for more robust training and interpretability. However, the simplicity of the commonly employed first-order assumption of the Markovian dynamics, usually chosen due to the resulting low computational complexity in conventional HMMs, restricts their modeling capacity, rendering the models unusable in real-world scenarios. The proposed higher-order variants aleviate this restriction by allowing more complex temporal dynamics; yet, their inherent computational complexity obstructs them from successful employment to realistic tasks. Hidden Semi-Markov Models and their variants have been proposed as an alternative, where by explicitly modeling the lasting time of each state, the flexibility and capacity of the resulting models can be increased. Nevertheless, in both cases, the models fail to consider potential non-homogeneous temporal dynamics in the data. To this end, significant research effort has been devoted in creating models that comprise a variable order chain. Nonetheless, the considered approaches also exhibit several drawbacks, such as the inability to model continuous observations. This work draws inspiration from the aforementioned results, and attempts to offer a principled way of modeling complex sequential data and time-series in general. To this end, we introduce a variant of rigid HMM architectures that constitutes an hierarchical extension; we postulate an additional latent first-order Markov Chain, allowing the model to alter the effective temporal dynamics of the conventional observation emitting Markov Chain. In this way, the model can dynamically infer which past state more strongly affects the current time frame. To increase the modeling capacity and robustness of the considered approach, we employ arguments from the solid Bayesian framework, relying on Variational Inference for tractable approximations. We demonstrate the modeling capabilities of the resulting model, in terms of recognition accuracy in one of the most challenging tasks in the Computer Vision community, namely, Human Action Recognition. Thus, we employ four benchmark datasets and compare the model's performance to similar baseline methods such as HMMs and LTSMs. We additionally explore the potency of the introduced hierarchical extension when compared to state-of-the-art methods, while examining its ability to model data with missing values. The provided experimental results assert the increased modeling capacity of the considered model, providing competitive recognition accuracy in all of the considered datasets, and effectively handling data with missing values, while exhibiting the smallest decrease in recognition accuracy relative to the increasing portion of missing values. Finally, further investigation of the behavior of the postulated latent process has provided strong evidence that the inferred temporal patterns can successfully model data with complex temporal dependencies.
Degree
thesis:*- Grantor dc:publisher
- National and Kapodistrian University of Athens
- Year dc:date
- 2020
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Panousis, Konstantinos-Panagiotis
- Πανούσης, Κωνσταντίνος-Παναγιώτης
Subjects
dc:subject × 30- Μηχανική μάθηση
- Επιβλεπόμενη μάθηση
- Νευρωνικά δίκτυα
- Βαθιά νευρωνικά δίκτυα
- Τοπικός ανταγωνισμός
- Μη παραμετρική μπεϋζιανή στατιστική
- Κρυφά μαρκοβιανά μοντέλα
- Αναγνώριση δράσεων
- Συνελικτικά νευρωνικά δίκτυα
- Προσεγγιστικός συμπερασμός
- Neural networks
- Deep neural networks
- Convolutional neural networks
- Local competition
- Winner-takes-all
- Deep learning
- Supervised learning
- Unsupervised learning
- Bayesian statistics
- Bayesian nonparametric statistics
- Indian buffet process
- Variational inference
- Approximate inference
- Hidden Markov models
- Expectation-maximization
- Forward-backward
- Φυσικές Επιστήμες
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Natural Sciences
- Computer and Information Sciences
Rights
dc:rights- Statement dc:rights
-
- BY
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/47711