Back to search

Institutional Repository of Vilnius University

Adversarial machine learning for malware obfuscation /

Abstract

dc:description

Machine learning-based malware detectors have significantly improved the ability to identify malicious software. However, the attackers have responded by using machine learning to obfuscate malware and evade detection. As such, for each new detection model that is deployed, more advanced adversarial malware generators eventually emerge to defeat it. This continuous cat-and-mouse game requires researchers to study adversarial obfuscation methods. Gaining a better understanding of how different methods improve evasion probability allows researchers to strengthen malware defenses and anticipate the attackers' strategies. To provide a realistic baseline, this thesis uses the EMBER dataset and a Gradient Boosted Decision Tree classifier as the target black-box detector. The GBDT model from the EMBER framework was chosen for its strong detection performance. Two different adversarial malware obfuscation architectures were evaluated: MalGAN and malware_rl. First, MalGAN is the implementation of a generative adversarial network. It is designed to generate obfuscated malware examples. It comprises three parts: a generator, a discriminator, and a black-box detector. The main idea is for the generator and the discriminator to go back and forth, with occasional black-box queries. This allows the generator to learn how to fool the black-box detector. On the other hand, the malware_rl framework is a reinforcement learning algorithm, which uses an agent to interact with the black-box model. The agent takes discrete actions to interact with the environment, including the detector, and receives outputs and rewards. These two approaches were chosen because they represent the foundations of their respective architectures. This allows for a faithful comparison of two different architectures. This thesis shows that MalGAN and the RL-based approach achieved high evasion rates. Recommendations are made to combine these two architectures into a single architecture by leveraging their strengths.

Degree

thesis:*
Grantor dc:publisher
Institutional Repository of Vilnius University
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Kudirka, Justas,

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
Language dc:language
eng

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Vilnius University
Base URL
epublications.vu.lt/oai
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Kudirka, Justas,. Adversarial machine learning for malware obfuscation /. Institutional Repository of Vilnius University, 2026. https://repository.vu.lt/VU:ELABAETD277678634&prefLang=en_US