Institutional Repository of Vilnius University
Research of methods for software execution vulnerability analysis using machine learning algorithms /
Abstract
dc:descriptionThis work explores machine learning methods to analyze system calls and detect malicious applications more accurately and reliably. The main focus of this study is to develop a malware detection system architecture, including creating an isolated sandbox environment for program execution and system call collection. Based on the extracted logs, the dataset for training and testing machine learning models is created. Characteristics like buffer sizes and access to IP addresses or file paths are analyzed to classify benign and malicious behaviours. Several models are developed, including k-Nearest Neighbors, Support Vector Machines, Deep Learning, Random Forest, Naïve Bayes and LightGBM. Each model was evaluated based on its precision, recall, accuracy and F1 score. The Support Vector Machine algorithm demonstrated the best performance, followed by Random Forest and LightGBM models. During the feature importance analysis, the return values of system calls, signal handling, memory access patterns and interactions with the file system were key indicators of malicious activity. The misclassification analysis showed the overlap in the behaviour of malware and benign programs. That led to challenges in classifying non-malicious software with similar system call patterns to malware, leading to false positive results. This showcases the need to enhance the dataset diversity. This work demonstrates that system call analysis can improve malware detection by identifying its behavioural patterns. The results highlight the need to address such challenges as data imbalance and feature overlap between benign and malicious programs, which can lead to false positive or negative values in the predictions. To further enhance the accuracy of detection, the dataset should include a broader range of malware types and the real-time monitoring of system calls can be implemented. The deployment of models could further improve their practical use.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tovstohan, Daria,
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- eng
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD229556010&prefLang=en_US