Global ETD Search

31	Large-Context Question Answering with Cross-Lingual Transfer Sagen, Markus January 2021 (has links) Models based around the transformer architecture have become one of the most prominent for solving a multitude of natural language processing (NLP)tasks since its introduction in 2017. However, much research related to the transformer model has focused primarily on achieving high performance and many problems remain unsolved. Two of the most prominent currently are the lack of high performing non-English pre-trained models, and the limited number of words most trained models can incorporate for their context. Solving these problems would make NLP models more suitable for real-world applications, improving information retrieval, reading comprehension, and more. All previous research has focused on incorporating long-context for English language models. This thesis investigates the cross-lingual transferability between languages when only training for long-context in English. Training long-context models in English only could make long-context in low-resource languages, such as Swedish, more accessible since it is hard to find such data in most languages and costly to train for each language. This could become an efficient method for creating long-context models in other languages without the need for such data in all languages or pre-training from scratch. We extend the models’ context using the training scheme of the Longformer architecture and fine-tune on a question-answering task in several languages. Our evaluation could not satisfactorily confirm nor deny if transferring long-term context is possible for low-resource languages. We believe that using datasets that require long-context reasoning, such as a multilingual TriviaQAdataset, could demonstrate our hypothesis’s validity. Long-Context Multilingual Model Longformer XLM-R Longformer Long-term Context Extending Context Extend Context Large-Context Long-Context Large Context Long Context Cross-Lingual Multi-Lingual Cross Lingual Multi Lingual QA Question-Answering Question Answering Transformer model Machine Learning Transfer Learning SQuAD Memory Transfer Learning Long-Context Long Context Efficient Monolingual Multilingual QA model Language Model Huggingface BERT RoBERTa XLM-R mBERT Multilingual BERT Efficient Transformers Reformer Linformer Performer Transformer-XL Wikitext-103 TriviaQA HotpotQA WikiHopQA VINNOVA Peltarion AI LM MLM Deep Learning Natural Language Processing NLP Attention Transformers Transfer Learning Datasets Computer and Information Sciences Data- och informationsvetenskap

Search results

Large-Context Question Answering with Cross-Lingual Transfer