Institutional Repository of Vilnius University
Bypassing defensive system detection using obfuscation techniques research /
Abstract
dc:descriptionThroughout the years, antivirus and endpoint detection have improved beyond simple static detection scanning by incorporating complex behavior analysis and Artificial Intelligence algorithms to assist in identifying threats. This thesis covers the theoretical foundations of obfuscation through PE File manipulation, an advanced Process Hollowing injection method, and the establishment of persistence for C2 malware in the Windows Registry. We propose, and conduct experiments on a practical implementation of a Crypter, that combines these methods, alongside payload fragmentation, encryption and shuffling techniques to disguise malicious payloads inside benign-looking executables. The combination of these layers with injection and persistence techniques allows the software to challenge the detection of various modern antivirus solutions. The Crypter's performance was evaluated in both local and online environments to measure how effective the current security industry is at separating fragmented malicious files from benign data. The experiments were conducted on modern operating systems, such as Windows 11 and Windows 10, as well as older systems such as Windows 7 and Windows 8. The results from conducted experiments show a high contrast in security between legacy and modern systems, with the old systems being completely oblivious to the malware. Results conducted in modern systems show a highly sophisticated level of detection, utilizing static and behavioral detection, as well as artificial intelligence models, making it nearly impossible to penetrate the defenses of these systems. However, malware with no Injection and Persistence logic did manage to escape detection by Windows 11 Defender, making it possible to smuggle malware into the OS by obfuscating it with the explored methods. In order to compare these results with other third-party antivirus providers, tests are conducted on publicly available software, such as Kaspersky, MalwareBytes, Avast and TotalAV. The results show, that some third-party antivirus software alternatives do a better job than Windows systems, however results vary. Lastly, the third target of experiments are publicly available free online scanner tools, such as Kaspersky Threat Intelligence, Hybrid Analysis, as well as VirusTotal. The results show that these tools are quite accurate at detecting malicious code, depending on the vendor. While testing these tools several times, it was detected that they adapt to the malicious logic shown in previously uploaded executables and flag it.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Čepulis, Nerijus,
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- eng
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD277690619&prefLang=en_US