Democritus University of Thrace (DUTH)
Training and performance improvement techniques for artificial neural networks
Abstract
dc:descriptionArtificial Neural Networks (ANN) have shown that ANN can provide viable solutions to a variety of problems. The emergence of deep learning and deeper and more complex architecture has increased their outreach and their effectiveness into multiple areas and application fields. This has become possible with the emergence of super-performing computing devices with great parallelization capabilities, i.e. graphics cards. Since these devices are extremely power consuming, a open problem in neural network research is the optimization of the neural network architecture for each task. However, their complexity and the lack of efficient interpretation of ANN architectures (commonly considered black box techniques), has adverse effects on the optimization of each ANN architecture. The commonly used solution for this solution is the manual determination of the network architecture through trial and error. In this thesis, the main aim is to address this problem by exploring a solution by three distinct perspectives that can, however, work cooperatively. The first investigated aspect is using a combination of smaller in series connected neural networks that achieve the same if not better performance of larger and more complex neural networks. The data are handled, based on evaluation of a probabilistic function that attempts to identify a portion of the unknown data that are most probably classified correctly and those that need to be reclassified from the beginning. The correctly identified data can be used during a second training process in order to boost the overall classification performance. Another investigated aspect is finding the optimal topology of a neural network, based on the given dataset automatically and during training. The neural network is viewed from an information-theoretic point-of-view and information theoretic criteria were established in order to estimate the optimal size of each layer of the network. The developed solution provided better utilization of the given resources, thus minimizing the computational complexity of the training process. Finally, a novel activation function was proposed with learn-able parameters, based on the ReLU family of functions, that achieved better training efficiency in most of the tested datasets and neural network topologies. The learn-able parameter of the network appeared to act as a regularizing factor for the network. This thesis offers some important contributions towards the automatic topology optimization of neural networks, as well as a form of semi-supervised training strategy for classification. Both proposed solutions appeared to boost classification performance or offer better resource utilization for the optimized network. Finally, the self-regularizing activation function with a trainable parameter offer an increase in classification performance, compared to the fixed activation function, with a minimal computational cost.
Degree
thesis:*- Grantor dc:publisher
- Democritus University of Thrace (DUTH)
- Year dc:date
- 2022
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Maniatopoulos, Andreas-Antonios
- Μανιατόπουλος, Ανδρέας-Αντώνιος
Subjects
dc:subject × 12- τοπολογια νευρωνικών δικτύων
- βελτιστοποίηση νευρωνικών δικτύων
- συναρτηση ενεργοποίησης
- Artificial neural networks topology
- artificial neural networks optimization
- activation function
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Επιστήμες Μηχανικού και Τεχνολογία
- Ρομποτική
- Electrical Engineering, Electronic Engineering, Information Engineering
- Engineering and Technology
- Robotics
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/52963