Back to search

Institutional Repository of Vilnius University

Research of methods for software execution vulnerability analysis using machine learning algorithms /

Abstract

dc:description

This work explores machine learning methods to analyze system calls and detect malicious applications more accurately and reliably. The main focus of this study is to develop a malware detection system architecture, including creating an isolated sandbox environment for program execution and system call collection. Based on the extracted logs, the dataset for training and testing machine learning models is created. Characteristics like buffer sizes and access to IP addresses or file paths are analyzed to classify benign and malicious behaviours. Several models are developed, including k-Nearest Neighbors, Support Vector Machines, Deep Learning, Random Forest, Naïve Bayes and LightGBM. Each model was evaluated based on its precision, recall, accuracy and F1 score. The Support Vector Machine algorithm demonstrated the best performance, followed by Random Forest and LightGBM models. During the feature importance analysis, the return values of system calls, signal handling, memory access patterns and interactions with the file system were key indicators of malicious activity. The misclassification analysis showed the overlap in the behaviour of malware and benign programs. That led to challenges in classifying non-malicious software with similar system call patterns to malware, leading to false positive results. This showcases the need to enhance the dataset diversity. This work demonstrates that system call analysis can improve malware detection by identifying its behavioural patterns. The results highlight the need to address such challenges as data imbalance and feature overlap between benign and malicious programs, which can lead to false positive or negative values in the predictions. To further enhance the accuracy of detection, the dataset should include a broader range of malware types and the real-time monitoring of system calls can be implemented. The deployment of models could further improve their practical use.

Degree

thesis:*
Grantor dc:publisher
Institutional Repository of Vilnius University
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tovstohan, Daria,

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
Language dc:language
eng

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Vilnius University
Base URL
epublications.vu.lt/oai
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Tovstohan, Daria,. Research of methods for software execution vulnerability analysis using machine learning algorithms /. Institutional Repository of Vilnius University, 2025. https://repository.vu.lt/VU:ELABAETD229556010&prefLang=en_US