Back to search

University of Victoria (Canada)

On the (in)security of behavioral-based dynamic anti-malware techniques

Abstract

dc:description.abstract

The Internet has become the primary vector for the delivery of malicious code in cyber attacks, and malware has rapidly become a pervasive critical threat. Anti- malware products offer effective protection from malware threats for servers and endpoint devices using a variety of techniques. Advanced enterprise-level anti-malware products rely on state-of-art behavioral-based detection algorithms, in addition to traditional signature-based mechanisms. These dynamic detection techniques have been around for more than a decade and in response hackers have developed methods to evade them. However, currently known bypass methods require intensive manual labor. Moreover, this manual work has to be repeated whenever a parameter of the environment (such as the payload, operating system, Antivirus version, etc) changes, making these methods impractical. This may lead to the belief that dynamic techniques provide a good deterrence, and hence good protection. In this thesis we evaluate dynamic techniques. Specifically, we build tools to implement generic unhooking and funneling, and using these tools we show how dynamic techniques can be bypassed with considerably less effort than by fully manual methods. We also extend the repertoire of existing bypass methods and introduce a new malicious function call technique which exploits detection techniques that monitor a limited collection of critical system functions, as well as a method for bypassing guard-page protections. We demonstrate the effectiveness of all our techniques by conducting attacks against two enterprise antivirus products. Our results lead us to conclude that that dynamic techniques do not provide sufficient protection.

Degree

thesis:*
Level thesis:degree_level
Master of Science M.Sc.
Department thesis:degree_department
Department of Computer Science
Year dc:date.issued
2017

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ersan, Erkan
Advisors dc:contributor.supervisor
  • Malka, Lior
  • Kapron, Bruce M. (Bruce Michael)

Subjects

dc:subject × 28

Rights

dc:rights
Statement dc:rights
  • Available to the World Wide Web
Language dc:language.iso
en, English

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1828/7935

Chain of custody

source
Harvested from
University of Victoria (Canada)
Base URL
dspace.library.uvic.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Ersan, Erkan. On the (in)security of behavioral-based dynamic anti-malware techniques. Master of Science M.Sc. thesis, 2017. http://hdl.handle.net/1828/7935