National Technical University of Athens (NTUA)
Ανάπτυξη αλγορίθμων μάθησης για βελτίωση της εκπαίδευσης και της ερμηνείας των βαθιών νευρωνικών δικτύων
Abstract
dc:descriptionIn recent years the field of Machine Learning has been developed dramatically. With the progress and use of advanced hardware and computer systems, Machine Learning has given solutions in many scientific problems and is a vital part of some industries. Especially, Deep Learning and Deep Neural Networks are responsible for this great success. In this thesis we got involved, mainly, with Deep Neural Networks and their training process. We analyzed different learning algorithms and pinpointed problems that worsen the performance of neural networks. This dissertation proposes various algorithms and learning methods that intend to improve the training process and the general performance of Deep Neural Networks. Specifically, the first part revolves around the method of training a network with batches. We focused on the techniques of Online Batch Selection and proposed an algorithm that is based on Biased Sampling. The goal of the algorithm is to select samples with high loss values and add them in the training process more frequently. Emphasizing on the difficult samples the network is trained faster and has a better performance. To prove the usefulness of the proposed method, a series of experiments was inducted on different datasets. The results show that the algorithm improves the convergence speed and the best performance scores of the model. Apart from that, it improves the training time and the number of computations per iteration in comparison to other works in the literature. In the second part of the dissertation, we delved into the field of Imbalanced Datasets. This phenomenon is encountered often regarding real-world datasets and is a serious obstacle of the training process and the generalization of machine learning models. We described and analyzed various methods and techniques that are popular in the literature. Our work was centered around algorithm-based methods that tackle the problem of imbalance. We proposed a method of training neural networks, called NBSBS-R, that selects samples based on some criteria and adds a proper amount of noise. This way the network can learn the minority class just as well, while achieving better performance. An experimental framework is introduced that uses imbalanced datasets to test the new algorithm. The results showed an improvement in the generalization performance of the networks compared to other methods. Also, the experiments showed that the method is able to work together with other data-transformation techniques in order to build a better model overall. Another subject that was studied, was the field of optimization. There is a wide variety of optimizers that can be used to train a neural network, while each one of them has its own intricacies. We dived into adaptive optimizers and proposed an algorithm, called AdaLip, that constructs a learning rate per layer based on the Lipschitz constant. Various reasons were presented to show the need of the different approach of different layers and were supported experimentally. We tested our method on image classification datasets and the results showed imrovements in the convergence speed and the overall training performance. The proposed algorithm can work together with other optimizers and boost their performance scores. Finally, a theoretical proof of convergence of the new optimizer was presented. In the final part of the thesis, we delved into the field of interpreting neural networks. Interpretability is concerned with understanding neural networks and their predictions. Initially, we explored various interpretability techniques and compared their performances. The experiments were based on medical images for classifying the stages of retinopathy. This contributed to a deeper understanding of the model's functionality in relation to retinal images. We also showed that using interpretability techniques it becomes possible to tackle the problem of image segmantation. Furthermore, we delved deeper into the operation of interpretability methods, specifically those employing reference points. We demonstrated that the use of reference points entails many risks of inaccuracy in interpreting neural network predictions. Based on this limitation, we proposed a new layer aiming to improve these issues. The proposed Baseline-Aware Embedding layer was designed to incorporate the concept of a baseline or reference point through training. Thus, interpretability algorithms that operate with reference points can utilize this layer in network architectures to generate more accurate interpretations for various predictions. We demonstrated this experimentally on four tabular datasets, chosen for their diverse features and the common occurrence of reference point issues.
Degree
thesis:*- Grantor dc:publisher
- National Technical University of Athens (NTUA)
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Ioannou, Georgios
- Ιωάννου, Γεώργιος
Subjects
dc:subject × 30- Μηχανική μάθηση
- Βαθιά μηχανική μάθηση
- Δίκτυα, Νευρωνικά
- Συνελικτικό νευρωνικό δίκτυο
- Βελτιστοποίηση
- Αλγόριθμοι μάθησης
- Στοχαστικοί αλγόριθμοι
- Δειγματοληψία
- Ερμηνευσιμότητα
- Επεξηγησιμότητα
- Σημεία αναφοράς
- Διανυσματική αναπαράσταση
- Machine learning
- Deep learning
- Neural networks
- Convolution neural networks
- Optimization
- Learning algorithms
- Stochastic methods
- Sampling
- Interpretability
- Explainability
- Baselines
- Embedding
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Επιστήμες Μηχανικού και Τεχνολογία
- Υπολογιστές, Υλικό (hardware) και Αρχιτεκτονική
- Electrical Engineering, Electronic Engineering, Information Engineering
- Engineering and Technology
- Computer science, Hardware and Architecture
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/55442