Back to search

Institutes outside Greece

Parallelised Bayesian optimisation for deep learning

Abstract

dc:description

Training of deep neural networks (DNN) is an indispensable process in machine learning. The training process of DNNs aims to optimise the parameter values of the network, often relies on the derivative of the log-likelihoods of the underlying parameter space. As such, it is highly probable that the optimisation process to find local optimum values instead of the global ones. In addition to this, conventional approaches used for this process, such as Markov chain Monte Carlo methods, not only offer suboptimal runtime performance, but also prevent effective parallelisation due to inherent dependencies in the process. In this thesis, we consider an alternative approach to Markov chain Monte Carlo (MCMC) methods, namely the Sequential Monte Carlo (SMC) sampler, which generalises particle filters. More specifically, the thesis focuses on improving the performance and accuracy of the SMC methods, particularly in the context of fully Bayesian learning. The Radial Basis Function (RBF) network is an example of such training process based on fully Bayesian learning. In this setting, the thesis proposes a new method totrain neural networks using the importance sampling and resampling. The initial comparison of the two methods reveal that the proposed methodology is worse in both terms of accuracy and performance. This lead the research to concentrate of the performance and accuracy improvements of the proposed approach. The performance analysis began with application of a new proposed, parallel and fully distributed resampling methodology, with improved time complexity than the original approach using two MapReduce frameworks, Hadoop and Spark. Results indicate that Spark is up to 25 times faster than Hadoop, while on Spark the new proposed methodology is up to 10 times faster than the original method. However, it is noticed that application of the same algorithm on Message Passing Interface (MPI) provide significantly better runtimes and is more suitable for the proposed algorithm. The accuracy analysis began with experiments illustrating that the basic Sequential Monte Carlo sampler provides worse accuracy than alternative or competitor MCMC algorithms. Three different strategies are applied on the basic Sequential Monte Carlo sampler providing better accuracy. The analysis is extended to include competitor algorithms. The exhaustive evaluation shows that the proposed approach offers superior performance and accuracy.

Degree

thesis:*
Grantor dc:publisher
Institutes outside Greece
Year dc:date
2018

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Kekempanos, Lykourgos
  • Κεκεμπάνος, Λυκούργος

Subjects

dc:subject × 26

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/50829

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Kekempanos, Lykourgos; Κεκεμπάνος, Λυκούργος. Parallelised Bayesian optimisation for deep learning. Institutes outside Greece, 2018. http://hdl.handle.net/10442/hedi/50829