University of Patras
Νέες μέθοδοι εκπαίδευσης τεχνητών νευρωνικών δικτύων, βελτιστοποίησης και εφαρμογές
Abstract
dc:descriptionThis thesis investigates Optimization methods for Artificial Neural Network training and their applications. In the first chapters we discuss the basic neural network definitions, well known network architectures and training methods, as well as the theoretical background that supports the development of new efficient and effective training algorithms. In Chapter 3 a new generalized theoretical result is presented that underpins the development of first-order globally convergent batch training algorithms which employ local learning rates. This result allows us to equip the algorithms of this class with a strategy for adapting the direction of search to a descent one. In this way, a decrease of the batch-error measure at each training iteration is ensured, and convergence of the sequence of weight iterates to a local minimizer of the batch error function is obtained from remote initial weights. The effectiveness of the theoretical result is illustrated in application examples by comparing two well-known training algorithms with local learning rates to their globally convergent modifications. In Chapter 4, a mathematical framework for the convergence analysis of the well-known QuickProp method is described. Furthermore, we propose a modification of this method that exhibits improved convergence speed and stability, while at the same time, alleviates the use of heuristic learning parameters. Simulations are conducted to compare and evaluate the performance of the new modified QuickProp algorithm with various popular training algorithms. The results of the experiments indicate that the increased convergence rate achieved by the proposed algorithm, by no means affects its generalization capability and stability. In Chapter 5, evolutionary neural network training algorithms are presented. These algorithms are applied to train neural networks with weight values confined to a narrow band of integers. We also train the network using threshold activation functions. Furthermore, parallel evolutionary algorithms for integer weight neural network training are introduced. These algorithms have been designed keeping in mind that the resulting integer weights require less bits to be stored and the digital arithmetic operations between them are easier to be implemented in hardware. Another advantage of the proposed evolutionary strategies is that they are capable of continuing the training process previously computed pieces of information regarding the learning rate adaptation procedure. The proposed alogorithm has been implemented and tested in various problems with large data sets and networks. Additionally, we propose a hybrid Evolutionary algorithm for on-line training, capable of training and retraining a neural network when the task is nonstationary. Finally, in Chapter 8 we present deterministic nonmonotone learning strategies, i.e. deterministic training algorithms in which error function values are allowed to increase at some iterations. To this end, we argue that the current error function value must satisfy a nonmonotone criterion with respect to the maximum error function value of the M previous epochs, and we propose a subprocedure to dynamically compute M. the nonmonotone strategy can be incorporated in any batch training algorithm and provides fast, stable and reliable learning. Extensive experimental results in different classes of problems show that this approach improves the convergence speed and success percentage of first-order training algorithms, and alleviates the need for fine-tuning problem-depended heuristic parameters.
Degree
thesis:*- Grantor dc:publisher
- University of Patras
- Year dc:date
- 2002
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Πλαγιανάκος, Βασίλειος
- Plagianakos, Vassilis
Subjects
dc:subject × 21- Δίκτυα, Τεχνητά νευρωνικά
- Εκπαίδευση ανά πρότυπο εισόδου
- Εκπαίδευση ανά ομάδα προτύπων εισόδου
- Δίκτυα, Νευρωνικά με ακέραια βάρη
- Δίκτυα, Νευρωνικά με συναρτήσεις κατώφλια
- Μη μονότονη εκπαίδευση
- Παράλληλοι αλγόριθμοι εκπαίδευσης
- Ευρεία σύγκλιση αλγορίθμων εκπαίδευσης
- Networks, Artificial neural
- Training algorithms
- Batch training
- Online training
- Integer weight neural networks
- Threshold activations
- Nonmonotone training
- Parallel implementation
- Global convergence
- Φυσικές Επιστήμες
- Μαθηματικά
- Natural Sciences
- Mathematics
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/14281