University of Victoria (Canada)
On the (in)security of behavioral-based dynamic anti-malware techniques
Abstract
dc:description.abstractThe Internet has become the primary vector for the delivery of malicious code in cyber attacks, and malware has rapidly become a pervasive critical threat. Anti- malware products offer effective protection from malware threats for servers and endpoint devices using a variety of techniques. Advanced enterprise-level anti-malware products rely on state-of-art behavioral-based detection algorithms, in addition to traditional signature-based mechanisms. These dynamic detection techniques have been around for more than a decade and in response hackers have developed methods to evade them. However, currently known bypass methods require intensive manual labor. Moreover, this manual work has to be repeated whenever a parameter of the environment (such as the payload, operating system, Antivirus version, etc) changes, making these methods impractical. This may lead to the belief that dynamic techniques provide a good deterrence, and hence good protection. In this thesis we evaluate dynamic techniques. Specifically, we build tools to implement generic unhooking and funneling, and using these tools we show how dynamic techniques can be bypassed with considerably less effort than by fully manual methods. We also extend the repertoire of existing bypass methods and introduce a new malicious function call technique which exploits detection techniques that monitor a limited collection of critical system functions, as well as a method for bypassing guard-page protections. We demonstrate the effectiveness of all our techniques by conducting attacks against two enterprise antivirus products. Our results lead us to conclude that that dynamic techniques do not provide sufficient protection.
Degree
thesis:*- Level thesis:degree_level
- Master of Science M.Sc.
- Department thesis:degree_department
- Department of Computer Science
- Year dc:date.issued
- 2017
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ersan, Erkan
- Advisors dc:contributor.supervisor
-
- Malka, Lior
- Kapron, Bruce M. (Bruce Michael)
Subjects
dc:subject × 28- Malware
- Malware Detection
- Behavioral-based Dynamic Anti-Malware Techniques
- Computer Security
- Anti-malware
- Antivirus
- Cyber Attacks
- Behavioral-based Detection
- EMET
- Hooking
- Unhooking
- Guard Pages
- Bypassing Techniques
- Evasion Techniques
- Evasion
- funneling
- Control Flow Integrity
- CFI
- Web-based malware
- ROP
- Return-Oriented Programming
- Shellcode
- Payload
- Windows
- Anti-malware Evaluation
- Antivirus Evaluation
- Anti-malware Effectiveness
- Antivirus Effectiveness
Rights
dc:rights- Statement dc:rights
-
- Available to the World Wide Web
- Language dc:language.iso
- en, English
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- http://hdl.handle.net/1828/7935