Back to search

Institutional Repository of Vilnius University

Giliaisiais neuroniniais tinklais grįstų kalbos modelių charakteristikos analizė mažos imties duomenų rinkiniuose /

Abstract

dc:description

This work analyzes transformer language model performance under low sample size datasets. Multiple autoencoder transformer models: BERT, RoBERTa, ELECTRA with their base and large configurations are applied and trained on selected datasets. Multilingual XLM-R and mBERT models are evaluated under English and Lithuanian low-resource contexts. The results are contrasted with equivalent experiments performed for long-sequence BigBird and XLNet as well as character-level CANINE transformer models. CoNLL-2003, SWAG, SQuAD 2.0, AG News and WikiANN benchmarks are used to compare the aforementioned transformer architectures. The comparisons are performed under multiple different low sample dataset sizes. The selection of transformer architectures and benchmarks are motivated by the analysis of relevant literature. It is shown that mBERT achieves the best results for low-resource Lithuanian language problems and that ELECTRA achieves the best results under most monolingual English conditions. The results also show a tendency of larger models to perform well under low sample size conditions.

Degree

thesis:*
Grantor dc:publisher
Institutional Repository of Vilnius University
Year dc:date
2022

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Šyvis, Julius,

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
Language dc:language
lit

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Vilnius University
Base URL
epublications.vu.lt/oai
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Šyvis, Julius,. Giliaisiais neuroniniais tinklais grįstų kalbos modelių charakteristikos analizė mažos imties duomenų rinkiniuose /. Institutional Repository of Vilnius University, 2022. https://repository.vu.lt/VU:ELABAETD146242454&prefLang=en_US