University of the Aegean
Deep learning algorithms and architectures for natural language processing
Abstract
dc:descriptionThis dissertation explores the development of deep neural networks training algorithms and architectures which can be utilized in real world Natural Language Processing (NLP) tasks. To this end, we develop a novel constrained optimization framework that incorporates a-priori knowledge in the training process. Two different types of knowledge are explored: i) intrinsic knowledge about the characteristics of the network's loss function landscape, and ii) external knowledge originating from the nature of the learning task. In the case of intrinsic knowledge we develop a generic algorithm for training deep neural networks utilizing information from the second derivatives of the loss function (Hessian matrix). However the algorithm is Hessian free in the sense that it only requires the computation of a Hessian-vector product which can be computed exactly and very efficiently within any modern computational graph framework. We demonstrate the efficiency of the proposed algorithm in a variety of both NLP and computer vision tasks. In the latter case we explore a variety of external knowledge sources depending on the learning task at hand. We first explore methodologies in a recommender systems setting because in most cases the ratings matrix is extremely sparse and thus the step of calculating similarities between users often fails. Thus any additional knowledge apart from that provided by the ratings matrix can greatly enhance the quality of the recommendations. Initially we propose a method for generating recommendations in such problematic cases by expanding and traversing the users' similarity graph so as to make new link predictions between users that are not directly connected. We evaluate our proposal on a social rating network, and show that the infusion of this new information in the similarity graph is comparable to state of the art graph embedding techniques but with lower computational cost and directly comparable coverage. We also investigate the matrix factorization problem as a neural network, where the users and items are expressed as trainable embeddings. Within this formulation, the Scaled Exponential Linear Unit (SELU) exhibits performance invariance properties regarding the selection of the optimization algorithm and its corresponding hyperparameters. This is demonstrated in a number of experiments which involve various activation functions and optimization algorithms for training different neural network architectures on standard recommender systems benchmark datasets. Better results can be achieved in a social ratings network setting where additional knowledge is available from the social connections between users in addition to their rating behavior. This allows us to propose, within the constrained optimization framework, an efficient matrix factorization algorithm for recommender systems which improves on previously proposed related approaches in terms of convergence speed, recommendation accuracy and performance on cold start users. We then modify this approach to tackle NLP tasks by leveraging external knowledge about word similarities provided by semantic lexicons. We propose a powerful matrix factorization algorithm that extends GloVe and produces word embeddings enhanced by the semantic information. The proposed algorithm outperforms other related approaches that utilize semantic information either during training or as a post-processing step. Finally the efficiency of our constrained optimization approach for developing deep neural networks training algorithms and architectures for NLP tasks is demonstrated by the incorporation of the generalized Hebbian algorithm as external knowledge within the framework. This allows us to enhance the Transformer architecture with layers that enforce the diversification between different heads in the multi-head attention layer. Experiments on machine translation show that this approach can improve the performance of the baseline Transformer model.
Degree
thesis:*- Grantor dc:publisher
- University of the Aegean
- Year dc:date
- 2020
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Sakketou, Flοra
- Σακκέτου, Φλώρα
Subjects
dc:subject × 11Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/48299