Back to search

Institutional Repository of Vilnius University

Malware detection through network analysis using feature engineering and machine learning research /

Abstract

dc:description

The main goal of this study is to find unique C2-based malware network flow features that can facilitate its detection purely from network capture analysis using machine learning classification algorithms. Different families of C2-based malware are used in this research (e.g. RATs, information stealers, backdoors). The main details that are important are how they these malware families are installed on the system and the patterns they use when communicating with remote command-and-control (C2) server. Dozens of scientific papers and sources were studied to see which state-of-the-art algorithms are successfully used for network anomaly detection and how feature engineering facilitates this process. Apart from that, numerous cybersecurity reports and malware analysis blogs were examined to understand the unique characteristics of different malware families and what to look for in big network traffic samples to find the malicious indicators. \newline Practical implementation part of this research involves: - Finding unique network features for analysed C2-based malware samples. - Developing a programme to perform feature engineering and create a CSV dataset from the PCAP samples. - Training and testing different ML models on created datasets. The research concludes with machine learning models' malicious packet classification outcomes, presented in the Results Analysis section and supported with a variety of performance metrics. A comparison of different algorithms and their parameter tuning is provided, discussing which of them performed best for specific datasets and whether the theoretically found features accurately reflected reality and enhanced models' performance. In conclusion, Random Forest and Extreme Gradient Boosting classifiers demonstrated the highest performance when all different malware datasets were combined and achieved 99% accuracy and F1-Score in multi-class classification. This high performance was achieved because of uniquely engineered features, which were derived from a thorough investigated malware analysis.

Degree

thesis:*
Grantor dc:publisher
Institutional Repository of Vilnius University
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Martijan, Martin,

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
Language dc:language
eng

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Vilnius University
Base URL
epublications.vu.lt/oai
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Martijan, Martin,. Malware detection through network analysis using feature engineering and machine learning research /. Institutional Repository of Vilnius University, 2025. https://repository.vu.lt/VU:ELABAETD229555524&prefLang=en_US