Institutional Repository of Vilnius University
Didžiųjų kalbos modelių šališkumo ir etiškumo testavimas lietuvių kalba /
Abstract
dc:descriptionThis bachelor's thesis aims to adapt large language model testing for ethics and bias methodology to the Lithuanian language. After conducting a literature review, the concepts of ethics and bias were defined, and the aspect of social bias was selected for the study. Legal aspects related to ethics and social bias in artificial intelligence systems were also reviewed. Afterwards, in this thesis, metrics and benchmarks for evaluating large language model bias and ethics are analyzed with the goal of selecting the most suitable methodology for this research. Based on the described methodologies for investigating social bias, the “StereoSet” benchmark was chosen. A methodology for evaluating ethics and bias in Lithuanian was developed, including the preparation of the dataset and evaluation metrics. LLM-as-a-judge was used to test bias and ethics. During the experiment both open-source and closed-source large language models were tested. Based on the results of the experiment conducted during this thesis, a Lithuanian dataset adapted for testing ethics and bias was prepared. The “DeepEval” framework metrics for Bias, Fairness, and Coherence, along with the “StereoSet” benchmark metrics, were checked and adapted for evaluation in Lithuanian. The results of the experiment confirmed the initial hypothesis of the thesis - that the methodology developed by foreign researchers is suitable for testing large language models in Lithuanian. However, further improvements are needed in future research.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Sirvidas, Rokas,
Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- lit
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD239032286&prefLang=en_US