Institutional Repository of Vilnius University
Atvirųjų svorių didžiųjų kalbos modelių pagreitinimas realaus laiko sąveikai /
Abstract
dc:descriptionThe aim of this thesis is to experimentally investigate how quantization, speculative decoding, generation parameters, and context length affect the response latency of open-weight large language models in real-time interaction scenarios while also evaluating changes in response quality. The thesis analyzes acceleration methods for open-weight large language models, with particular attention to model-level and decoding-level optimizations. The theoretical part reviews scientific literature on quantization, mixed quantization schemes, importance-aware quantization, speculative decoding, the influence of generation parameters, and the impact of context length on inference speed. In the practical part, a research methodology was developed to compare different model configurations under identical conditions using latency and quality metrics. A single 8B instruction-tuned Llama-family model was selected for the experiments, and all measurements were carried out using the llama.cpp toolchain. In the first stage, standard quantization variants F16, Q4_K_S, Q4_K_M, Q5_K_M, and IQ4_NL were compared. In the second stage, the impact of imatrix on inference speed and perplexity was evaluated. In the third stage, two mixed quantization schemes were constructed based on tensor statistics and sensitivity analysis. In the fourth stage, speculative decoding with a smaller draft model was applied to the most promising configurations. The results showed that standard low-precision quantization provided the largest initial inference speedup, while speculative decoding produced the largest additional improvement in decoding speed. Meanwhile, imatrix and mixed quantization schemes were more useful for controlling the trade-off between speed and quality than for achieving the highest raw speed. The results of the study indicate that the most practically effective acceleration of open-weight LLMs is achieved by combining model-level and decoding-level optimization methods.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Kozlovskis, Dominykas,
Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- lit
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD308338488&prefLang=en_US