This thesis documents the use of Natural Language Processing (NLP) in Computer Assisted Language Learning (CALL) and its contribution to the learning experience of students studying Arabic as a foreign language. The goal of this project is to build an Intelligent Computer Assisted Language Learning (ICALL) system that provides computational assistance to learners of Arabic by teaching grammar, producing homework and issuing students with immediate feedback. To produce this system we use the Parasite system, which produces morphological, syntactic and semantic analysis of textual input, and extend it to provide error detection and diagnosis. The methodology we adopt involves relaxing constraints on unification so that correct information contained in a badly formed sentence may still be used to obtain a coherent overall analysis. We look at a range of errors, drawn from experience with learners at various levels, covering word internal problems (addition of inappropriate affixes, failure to apply morphotactic rules properly) and problems with relations between words (local constraints on features, and word order problems). As feedback is an important factor in learning, we look into different types of feedback that can be used to evaluate which is the most appropriate for the aim of our system.

In this dissertation, we introduce a novel text representation method mainly used for text classification purpose. The presented representation method is initially based on a variety of closeness relationships between pairs of words in text passages within the entire corpus. This representation is then used as the basis for our multi-level lightweight ontological representation method (TOR-FUSE), in which documents are represented based on their contexts and the goal of the learning task. The method is unlike the traditional representation methods, in which all the documents are represented solely based on the constituent words of the documents, and are totally isolated from the goal that they are represented for. We believe choosing the correct granularity of representation features is an important aspect of text classification. Interpreting data in a more general dimensional space, with fewer dimensions, can convey more discriminative knowledge and decrease the level of learning perplexity. The multi-level model allows data interpretation in a more conceptual space, rather than only containing scattered words occurring in texts. It aims to perform the extraction of the knowledge tailored for the classification task by automatic creation of a lightweight ontological hierarchy of representations. In the last step, we will train a tailored ensemble learner over a stack of representations at different conceptual granularities. The final result is a mapping and a weighting of the targeted concept of the original learning task, over a stack of representations and granular conceptual elements of its different levels (hierarchical mapping instead of linear mapping over a vector). Finally the entire algorithm is applied to a variety of general text classification tasks, and the performance is evaluated in comparison with well-known algorithms.

News companies have a need to automate and make the editors process of writing about hot and new events more effective. Current technologies involve robotic programs that fills in values in templates and website listeners that notifies the editors when changes are made so that the editor can read up on the source change at the actual website. Editors can provide news faster and better if directly provided with abstracts of the external sources. This study applies deep learning algorithms to automatically formulate abstracts and tag sources with appropriate tags based on the context. The study is a full stack solution, which manages both the editors need for speed and the training, testing and validation of the algorithms. Decision Tree, Random Forest, Multi Layer Perceptron and phrase document vectors are used to evaluate the categorisation and Recurrent Neural Networks is used to paraphrase unformatted texts. In the evaluation a comparison between different models trained by the algorithms with a variation of parameters are done based on the F-score. The results shows that the F-scores are increasing the more document the training has and decreasing the more categories the algorithm needs to consider. The Multi-Layer Perceptron perform best followed by Random Forest and finally Decision Tree. The document length matters, when larger documents are considered during training the score is increasing considerably. A user survey about the paraphrase algorithms shows the paraphrase result is insufficient to satisfy editors need. It confirms a need for more memory to conduct longer experiments.

When classifying texts using a linear classifier, the texts are commonly represented as feature vectors. Previous methods to represent features as vectors have been unable to capture the context of individual words in the texts, in theory leading to a poor representation of natural language. Bidirectional Encoder Representations from Transformers (BERT), uses a multi-headed self-attention mechanism to create deep bidirectional feature representations, able to model the whole context of all words in a sequence. A BERT model uses a transfer learning approach, where it is pre-trained on a large amount of data and can be further fine-tuned for several down-stream tasks. This thesis uses one multilingual, and two dedicated Swedish BERT models, for the task of classifying Swedish texts as of either easy-to-read or standard complexity in their respective domains. The performance on the text classification task using the different models is then compared both with feature representation methods used in earlier studies, as well as with the other BERT models. The results show that all models performed better on the classification task than the previous methods of feature representation. Furthermore, the dedicated Swedish models show better performance than the multilingual model, with the Swedish model pre-trained on more diverse data outperforming the other.

Notre travail est présenté en trois parties indépendantes. Tout d'abord, nous proposons trois heuristiques d'apprentissage actif pour les réseaux de neurones profonds : Nous mettons à l'échelle le `query by committee' , qui agrège la décision de sélectionner ou non une donnée par le vote d'un comité. Pour se faire nous formons le comité à l'aide de différents masques de dropout. Un autre travail se base sur la distance des exemples à la marge. Nous proposons d'utiliser les exemples adversaires comme une approximation de la dite distance. Nous démontrons également des bornes de convergence de notre méthode dans le cas de réseaux linéaires. L’usage des exemples adversaires ouvrent des perspectives de transférabilité d’apprentissage actif d’une architecture à une autre. Puis, nous avons formulé une heuristique d'apprentissage actif qui s'adapte tant au CNNs qu'aux RNNs. Notre méthode sélectionne les données qui minimisent l'énergie libre variationnelle. Dans un second temps, nous nous sommes concentrés sur la distance de Wasserstein. Nous projetons les distributions dans un espace où la distance euclidienne mimique la distance de Wasserstein. Pour se faire nous utilisons une architecture siamoise. Également, nous démontrons les propriétés sous-modulaires des prototypes de Wasserstein et comment les appliquer à l'apprentissage actif. Enfin, nous proposons de nouveaux outils de visualisation pour expliquer les prédictions d'un CNN sur du langage naturel. Premièrement, nous détournons une stratégie d'apprentissage actif pour confronter la pertinence des phrases sélectionnées aux techniques de phraséologie les plus récentes. Deuxièmement, nous profitons des algorithmes de déconvolution des CNNs afin de présenter une nouvelle perspective sur l'analyse d'un texte. / Our work is presented in three separate parts which can be read independently. Firstly we propose three active learning heuristics that scale to deep neural networks: We scale query by committee, an ensemble active learning methods. We speed up the computation time by sampling a committee of deep networks by applying dropout on the trained model. Another direction was margin-based active learning. We propose to use an adversarial perturbation to measure the distance to the margin. We also establish theoretical bounds on the convergence of our Adversarial Active Learning strategy for linear classifiers. Some inherent properties of adversarial examples opens up promising opportunity to transfer active learning data from one network to another. We also derive an active learning heuristic that scales to both CNN and RNN by selecting the unlabeled data that minimize the variational free energy. Secondly, we focus our work on how to fasten the computation of Wasserstein distances. We propose to approximate Wasserstein distances using a Siamese architecture. From another point of view, we demonstrate the submodular properties of Wasserstein medoids and how to apply it in active learning. Eventually, we provide new visualization tools for explaining the predictions of CNN on a text. First, we hijack an active learning strategy to confront the relevance of the sentences selected with active learning to state-of-the-art phraseology techniques. These works help to understand the hierarchy of the linguistic knowledge acquired during the training of CNNs on NLP tasks. Secondly, we take advantage of deconvolution networks for image analysis to present a new perspective on text analysis to the linguistic community that we call Text Deconvolution Saliency.

By formulating the task of property-based molecular optimization into a neural machine translation problem, researchers have been able to apply the Transformer model from the field of natural language processing to generate molecules with desirable properties by making a small modification to a given starting molecule. These results verify the capability of Transformer models in capturing the connection between properties and structural changes in molecular pairs. However, the current research only proposes a Transformer model with fixed parameters that can produce limit amount of optimized molecules. Additionally, the trained Transformer model does not always successfully generate optimized output for every molecule and desirable property constraint given. In order to push the Transformer model into real applications where different sets of desirable property constraints in combination of variety of molecules might need to be optimized, these obstacles need to be overcome first. In this work, we present a framework using reinforcement learning as a fine-tuning method for the pre-trained Transformer to induce various output and leverage the prior knowledge of the model for a challenging data point. Our results show that, based on the definition of the scoring function, the Transformer model can generate much larger numbers of optimized molecules for a data point that is considered challenging to the pre-trained model. Meanwhile, we also showcase the relation between the sampling size and the efficiency of the framework in yielding desirable outputs to demonstrate the optimal configuration for future users. Furthermore, we have chemists to inspect the generated molecules and find that the reinforcement learning fine-tuning causes the catastrophic forgetting problem that leads our model into generating unstable molecules. Through maintaining the prior knowledge or applying rule-based scoring component, we demonstrate two strategies that can successfully reduce the effect of catastrophic forgetting as a reference for future research.

No description available.

Twitter has quickly become one of the most popular social media sites. It has 313 million monthly active users, and 500 million tweets are published daily. With the massive number of tweets, Twitter users share information about a location along with the temporal awareness. In this work, I focus on tweets where author of the tweets exclusively mentions a location in the tweet. Natural language processing systems can leverage wide range of information from the tweets to build applications like recommender systems that predict the location of the author. This kind of system can be used to increase the visibility of the targeted audience and can also provide recommendations interesting places to visit, hotels to stay, restaurants to eat, targeted on-line advertising, and co-traveler matching based on the temporal information extracted from a tweet. In this work I determine if the author of the tweet is present in the mentioned location of the tweet. I also determine if the author is present in the location before tweeting, while tweeting, or after tweeting. I introduce 5 temporal tags (before the tweet but > 24 hours; before the tweet but < 24 hours; during the tweet is posted; after the tweet is posted but < 24 hours; and after the tweet is posted but > 24 hours). The major contributions of this paper are: (1) creation of a corpus of 1062 tweets containing 1200 location named entities, containing annotations whether author of a tweet is or is not located in the location he tweets about with respect to 5 temporal tags; (2) detailed corpus analysis including real annotation examples and label distributions per temporal tag; (3) detailed inter-annotator agreements, including Cohen's kappa, Krippendorff's alpha and confusion matrices per temporal tag; (4) label distributions and analysis; and (5) supervised learning experiments, along with the results.

The following diploma thesis deals with dictionary data processing, especially those in XML based formats. At first, the reader is acquainted with linguistic and lexicographical terms used in this work. Then particular lexicographical data format types and specific formats are introduced. Their advantages and disadvantages are discussed as well. According to previously set criteria, the LMF format has been chosen for design and implementation of Python application, which focuses especially on intelligent merging of more dictionaries into one. After passing all unit tests, this application has been used for processing LMF dictionaries, located on the faculty server of the research group for natural language processing. Finally, the advantages and disadvantages of this application are discussed and ways of further usage and extension are suggested.

The goal of information extraction is to retrieve relational data from texts written in natural human language. Applications of such obtained information is wide - from text summarization, through ontology creation up to answering questions by QA systems. This work describes design and implementation of a system working in computer cluster which transforms a dump of Wikipedia articles to a set of extracted information that is stored in distributed RDF database with a possibility to query it using created user interface.