[pt] Rotular as classes gramaticais ao longo de uma sentença - part-ofspeech tagging - é uma das primeiras tarefas de processamento de linguagem natural, fornecendo atributos importantes para realizar tarefas de alta complexidade. A representação de texto a nível de palavra tem sido amplamente adotada, tanto através de uma codificação esparsa convencional, e.g. bagofwords; quanto por uma representação distribuída, como os sofisticados modelos de word-embedding usados para descrever informações sintáticas e semânticas. Um problema importante desse tipo de codificação é a carência de aspectos morfológicos. Além disso, os sistemas atuais apresentam uma precisão por token em torno de 97 por cento. Contudo, quando avaliados por sentença, apresentam um resultado mais modesto com uma taxa de acerto em torno de 55−57 por cento. Neste trabalho, nós demonstramos como utilizar n-grams para derivar automaticamente atributos esparsos e morfológicos para processamento de texto. Essa representação permite que redes neurais realizem a tarefa de POS-Tagging a partir de uma representação a nível de caractere. Além disso, introduzimos uma estratégia de regularização capaz de selecionar atributos específicos para cada neurônio. A utilização de regularização embutida em nossos modelos produz duas variantes. A primeira compartilha os n-grams selecionados globalmente entre todos os neurônios de uma camada; enquanto que a segunda opera uma seleção individual para cada neurônio, de forma que cada neurônio é sensível apenas aos n-grams que mais o estimulam. Utilizando a abordagem apresentada, nós geramos uma alta quantidade de características que representam afeições morfossintáticas relevantes baseadas a nível de caractere. Nosso POS tagger atinge a acurácia de 96, 67 por cento no corpus Mac-Morpho para o Português. / [en] Part-of-speech tagging is one of the primary stages in natural language processing, providing useful features for performing higher complexity tasks. Word level representations have been largely adopted, either through a conventional sparse codification, such as bag-of-words, or through a distributed representation, like the sophisticated word embedded models used to describe syntactic and semantic information. A central issue on these codifications is the lack of morphological aspects. In addition, recent taggers present per-token accuracies around 97 percent. However, when using a persentence metric, the good taggers show modest accuracies, scoring around 55-57 percent. In this work, we demonstrate how to use n-grams to automatically derive morphological sparse features for text processing. This representation allows neural networks to perform POS tagging from a character-level input. Additionally, we introduce a regularization strategy capable of selecting specific features for each layer unit. As a result, regarding n-grams selection, using the embedded regularization in our models produces two variants. The first one shares globally selected features among all layer units, whereas the second operates individual selections for each layer unit, so that each unit is sensible only to the n-grams that better stimulate it. Using the proposed approach, we generate a high number of features which represent relevant morphosyntactic affection based on a character-level input. Our POS tagger achieves the accuracy of 96.67 percent in the Mac-Morpho corpus for Portuguese.

No description available.

Cross-lingual transfer learning has been the driving force of low-resource natural language processing in recent years, relying on massively multilingual language models with hopes of solving the data scarcity issue for languages with a limited digital presence. However, this "one-size-fits-all" approach is not equally applicable to all low-resource languages, suggesting limitations of such models in cross-lingual transfer. Besides, known similarities and phylogenetic relationships between source and target languages are often overlooked. In this work, the emphasis is placed on Faroese, a low-resource North Germanic language with several closely related resource-rich sibling languages. The cross-lingual transfer potential from these strong Scandinavian source candidates, as well as from additional genetically related, geographically proximate, and syntactically similar source languages is studied in single-source and multi-source experiments, in terms of Faroese syntactic parsing and part-of-speech tagging. In addition, the effect of task-specific fine-tuning on monolingual, linguistically informed smaller multilingual, and massively multilingual pre-trained language models is explored. The results suggest Icelandic as a strong source candidate, however, only when fine-tuning a monolingual model. With multilingual models, task-specific fine-tuning in Norwegian and Swedish seems even more beneficial. Although they do not surpass fully Scandinavian fine-tuning, models trained on genetically related and syntactically similar languages produce good results. Additionally, the findings indicate that multilingual models outperform models pre-trained on a single language, and that even better results can be achieved using a smaller, linguistically informed model, compared to a massively multilingual one.

[pt] Part-of-speech (POS) tagging é o processo de categorizar cada palavra de uma sentença com sua devida classe morfossintática (verbo, substantivo, adjetivo e etc). POS tagging é considerada uma atividade fundamental no processo de construção de aplicações de processamento de linguagem natural (PLN), muitas dessas aplicações, em algum ponto, demandam esse tipo de informação. Nesse trabalho, construímos um POS tagger para o Português Contemporâneo e o Português Histórico, baseado em uma arquitetura de rede neural recorrente. Tradicionalmente a construção dessas ferramentas requer muitas features específicas do domínio da linguagem e dados externos ao conjunto de treino, mas nosso POS tagger não usa esses requisitos. Treinamos uma rede Bidirectional Long short-term memory (BLSTM), que se beneficia das representações de word embeddings e character embeddings das palavras, para atividade de classificação morfossintática. Testamos nosso POS tagger em três corpora diferentes: a versão original do corpus MacMorpho, a versão revisada do corpus Mac-Morpho e no corpus Tycho Brahe. Nós obtemos um desempenho ligeiramente melhor que os sistemas estado da arte nos três corpora: 97.83 por cento de acurácia para o Mac-Morpho original, 97.65 por cento de acurácia para o Mac-Morpho revisado e 97.35 por cento de acurácia para Tycho Brahe. Conseguimos, também, uma melhora nos três corpora para a medida de acurácia fora do vocabulário, uma acurácia especial calculada somente sobre as palavras desconhecidas do conjunto de treino. Realizamos ainda um estudo comparativo para verificar qual dentre os mais populares algoritmos de criação de word embedding (Word2Vec, FastText, Wang2Vec e Glove), é mais adequado para a atividade POS tagging em Português. O modelo de Wang2Vec mostrou um desempenho superior. / [en] Part-of-speech (POS) tagging is a process of labeling each word in a sentence with a morphosyntactic class (verb, noun, adjective and etc). POS tagging is a fundamental part of the linguistic pipeline, most natural language processing (NLP) applications demand, at some step, part-of-speech information. In this work, we constructed a POS tagger for Contemporary Portuguese and Historical Portuguese, using a recurrent neural network architecture. Traditionally the development of these tools requires many handcraft features and external data, our POS tagger does not use these elements. We trained a Bidirectional Long short-term memory (BLSTM) network that benefits from the word embeddings and character embeddings representations of the words, for morphosyntactic classification. We tested our POS tagger on three different corpora: the original version of the Mac-Morpho corpus, the revised version of the Mac-Morpho corpus, and the Tycho Brahe corpus. We produce state-of-the-art POS taggers for the three corpora: 97.83 percent accuracy on the original Mac-Morpho corpus, 97.65 percent accuracy on the revised Mac-Morpho and 97.35 percent accuracy on the Tycho Brahe corpus. We also achieved an improvement in the three corpora in out-of-vocabulary accuracy, that is the accuracy on words not seen in training sentences. We also performed a comparative study to test which different types of word embeddings (Word2Vec, FastText, Wang2Vec, and Glove) is more suitable for Portuguese POS tagging. The Wang2Vec model showed higher performance.

Contemporary approaches to natural language processing are predominantly based on statistical machine learning from large amounts of text, which has been manually annotated with the linguistic structure of interest. However, such complete supervision is currently only available for the world's major languages, in a limited number of domains and for a limited range of tasks. As an alternative, this dissertation considers methods for linguistic structure prediction that can make use of incomplete and cross-lingual supervision, with the prospect of making linguistic processing tools more widely available at a lower cost. An overarching theme of this work is the use of structured discriminative latent variable models for learning with indirect and ambiguous supervision; as instantiated, these models admit rich model features while retaining efficient learning and inference properties. The first contribution to this end is a latent-variable model for fine-grained sentiment analysis with coarse-grained indirect supervision. The second is a model for cross-lingual word-cluster induction and the application thereof to cross-lingual model transfer. The third is a method for adapting multi-source discriminative cross-lingual transfer models to target languages, by means of typologically informed selective parameter sharing. The fourth is an ambiguity-aware self- and ensemble-training algorithm, which is applied to target language adaptation and relexicalization of delexicalized cross-lingual transfer parsers. The fifth is a set of sequence-labeling models that combine constraints at the level of tokens and types, and an instantiation of these models for part-of-speech tagging with incomplete cross-lingual and crowdsourced supervision. In addition to these contributions, comprehensive overviews are provided of structured prediction with no or incomplete supervision, as well as of learning in the multilingual and cross-lingual settings. Through careful empirical evaluation, it is established that the proposed methods can be used to create substantially more accurate tools for linguistic processing, compared to both unsupervised methods and to recently proposed cross-lingual methods. The empirical support for this claim is particularly strong in the latter case; our models for syntactic dependency parsing and part-of-speech tagging achieve the hitherto best published results for a wide number of target languages, in the setting where no annotated training data is available in the target language.

A family of Natural Language Processing (NLP) tasks such as part-of- speech (PoS) tagging, Named Entity Recognition (NER), and Multiword Expression (MWE) identification all involve assigning labels to sequences of words in text (sequence labeling). Most modern machine learning approaches to sequence labeling utilize word embeddings, learned representations of text, in which words with similar meanings have similar representations. Quite recently, contextualized word embeddings have garnered much attention because, unlike pretrained context- insensitive embeddings such as word2vec, they are able to capture word meaning in context. In this thesis, I evaluate the performance of different embedding setups (context-sensitive, context-insensitive word, as well as task-specific word, character, lemma, and PoS) on the three abovementioned sequence labeling tasks using a deep learning model (BiLSTM) and Portuguese datasets. v

No description available.

Contemporary approaches to natural language processing are predominantly based on statistical machine learning from large amounts of text, which has been manually annotated with the linguistic structure of interest. However, such complete supervision is currently only available for the world's major languages, in a limited number of domains and for a limited range of tasks. As an alternative, this dissertation considers methods for linguistic structure prediction that can make use of incomplete and cross-lingual supervision, with the prospect of making linguistic processing tools more widely available at a lower cost. An overarching theme of this work is the use of structured discriminative latent variable models for learning with indirect and ambiguous supervision; as instantiated, these models admit rich model features while retaining efficient learning and inference properties. The first contribution to this end is a latent-variable model for fine-grained sentiment analysis with coarse-grained indirect supervision. The second is a model for cross-lingual word-cluster induction and the application thereof to cross-lingual model transfer. The third is a method for adapting multi-source discriminative cross-lingual transfer models to target languages, by means of typologically informed selective parameter sharing. The fourth is an ambiguity-aware self- and ensemble-training algorithm, which is applied to target language adaptation and relexicalization of delexicalized cross-lingual transfer parsers. The fifth is a set of sequence-labeling models that combine constraints at the level of tokens and types, and an instantiation of these models for part-of-speech tagging with incomplete cross-lingual and crowdsourced supervision. In addition to these contributions, comprehensive overviews are provided of structured prediction with no or incomplete supervision, as well as of learning in the multilingual and cross-lingual settings. Through careful empirical evaluation, it is established that the proposed methods can be used to create substantially more accurate tools for linguistic processing, compared to both unsupervised methods and to recently proposed cross-lingual methods. The empirical support for this claim is particularly strong in the latter case; our models for syntactic dependency parsing and part-of-speech tagging achieve the hitherto best published results for a wide number of target languages, in the setting where no annotated training data is available in the target language.

No description available.

La plupart des méthodes de traitement automatique des langues (TAL) peuvent être formalisées comme des problèmes de prédiction, dans lesquels on cherche à choisir automatiquement l'hypothèse la plus plausible parmi un très grand nombre de candidats. Malgré de nombreux travaux qui ont permis de mieux prendre en compte la structure de l'ensemble des hypothèses, la taille de l'espace de recherche est généralement trop grande pour permettre son exploration exhaustive. Dans ce travail, nous nous intéressons à l'importance du design de l'espace de recherche et étudions l'utilisation de contraintes pour en réduire la taille et la complexité. Nous nous appuyons sur l'étude de trois problèmes linguistiques — l'analyse morpho-syntaxique, le transfert cross-lingue et le problème du réordonnancement en traduction — pour mettre en lumière les risques, les avantages et les enjeux du choix de l'espace de recherche dans les problèmes de TAL.Par exemple, lorsque l'on dispose d'informations a priori sur les sorties possibles d'un problème d'apprentissage structuré, il semble naturel de les inclure dans le processus de modélisation pour réduire l'espace de recherche et ainsi permettre une accélération des traitements lors de la phase d'apprentissage. Une étude de cas sur les modèles exponentiels pour l'analyse morpho-syntaxique montre paradoxalement que cela peut conduire à d'importantes dégradations des résultats, et cela même quand les contraintes associées sont pertinentes. Parallèlement, nous considérons l'utilisation de ce type de contraintes pour généraliser le problème de l'apprentissage supervisé au cas où l'on ne dispose que d'informations partielles et incomplètes lors de l'apprentissage, qui apparaît par exemple lors du transfert cross-lingue d'annotations. Nous étudions deux méthodes d'apprentissage faiblement supervisé, que nous formalisons dans le cadre de l'apprentissage ambigu, appliquées à l'analyse morpho-syntaxiques de langues peu dotées en ressources linguistiques.Enfin, nous nous intéressons au design de l'espace de recherche en traduction automatique. Les divergences dans l'ordre des mots lors du processus de traduction posent un problème combinatoire difficile. En effet, il n'est pas possible de considérer l'ensemble factoriel de tous les réordonnancements possibles, et des contraintes sur les permutations s'avèrent nécessaires. Nous comparons différents jeux de contraintes et explorons l'importance de l'espace de réordonnancement dans les performances globales d'un système de traduction. Si un meilleur design permet d'obtenir de meilleurs résultats, nous montrons cependant que la marge d'amélioration se situe principalement dans l'évaluation des réordonnancements plutôt que dans la qualité de l'espace de recherche. / Most natural language processing tasks are modeled as prediction problems where one aims at finding the best scoring hypothesis from a very large pool of possible outputs. Even if algorithms are designed to leverage some kind of structure, the output space is often too large to be searched exaustively. This work aims at understanding the importance of the search space and the possible use of constraints to reduce it in size and complexity. We report in this thesis three case studies which highlight the risk and benefits of manipulating the seach space in learning and inference.When information about the possible outputs of a sequence labeling task is available, it may seem appropriate to include this knowledge into the system, so as to facilitate and speed-up learning and inference. A case study on type constraints for CRFs however shows that using such constraints at training time is likely to drastically reduce performance, even when these constraints are both correct and useful at decoding.On the other side, we also consider possible relaxations of the supervision space, as in the case of learning with latent variables, or when only partial supervision is available, which we cast as ambiguous learning. Such weakly supervised methods, together with cross-lingual transfer and dictionary crawling techniques, allow us to develop natural language processing tools for under-resourced languages. Word order differences between languages pose several combinatorial challenges to machine translation and the constraints on word reorderings have a great impact on the set of potential translations that is explored during search. We study reordering constraints that allow to restrict the factorial space of permutations and explore the impact of the reordering search space design on machine translation performance. However, we show that even though it might be desirable to design better reordering spaces, model and search errors seem yet to be the most important issues.