Institutional Repository of Vilnius University
Giliaisiais neuroniniais tinklais grįstų kalbos modelių charakteristikos analizė mažos imties duomenų rinkiniuose /
Abstract
dc:descriptionThis work analyzes transformer language model performance under low sample size datasets. Multiple autoencoder transformer models: BERT, RoBERTa, ELECTRA with their base and large configurations are applied and trained on selected datasets. Multilingual XLM-R and mBERT models are evaluated under English and Lithuanian low-resource contexts. The results are contrasted with equivalent experiments performed for long-sequence BigBird and XLNet as well as character-level CANINE transformer models. CoNLL-2003, SWAG, SQuAD 2.0, AG News and WikiANN benchmarks are used to compare the aforementioned transformer architectures. The comparisons are performed under multiple different low sample dataset sizes. The selection of transformer architectures and benchmarks are motivated by the analysis of relevant literature. It is shown that mBERT achieves the best results for low-resource Lithuanian language problems and that ELECTRA achieves the best results under most monolingual English conditions. The results also show a tendency of larger models to perform well under low sample size conditions.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2022
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Šyvis, Julius,
Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- lit
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD146242454&prefLang=en_US