Over the years, many defense techniques have been proposed by the security community. Even so, few have been adopted by the general public and deployed in production. This limited defense deployment and weak security has serious consequences, as large scale cyber-attacks are now a common occurrence in society. One major obstacle that stands in the way is practicality, the quality of being designed for actual use or having usefulness or convenience. For example, an exploit mitigation design may be considered not practical to deploy if it imposes high performance overhead, despite offering excellent and robust security guarantees. This is because achieving hallmarks of practical design, such as minimizing adverse side-effects like performance degradation or memory monopolization, is difficult in practice, especially when trying to provide a high level of security for users.
Secure and practical exploit mitigation design must successfully navigate several challenges. To illustrate, modern-day attacks, especially code re-use attacks, understand that rudimentary defenses such as Data Execution Prevention (DEP) and Address Space Layout Randomization (ASLR) will be deployed moving forward. These attacks have therefore evolved and diversified their angles of attack to become capable of leveraging a multitude of different code components. Accordingly, the security community has uncovered these threats and maintained progress in providing possible resolutions with new exploit mitigation designs. More specifically though, defenses have had to correspondingly extend their capabilities to protect more aspects of code, leading to defense techniques becoming increasingly complex. Trouble then arises as supporting such fine-grained defenses brings inherent disadvantages such as significant hardware resource utilization that could be otherwise used for useful work. This complexity has made performance, security, and scalability all competing ideals in practical system design. At the same time, other recent efforts have implemented mechanisms with negligible performance impact, but do so at the risk of weaker security guarantees.
This dissertation first formalizes the challenges in modern exploit mitigation design. To illustrate these challenges, this dissertation presents a survey from the perspective of both attacker and defender to provide an overview of this current security landscape. This includes defining an informal taxonomy of exploit mitigation strategies, explaining prominent attack vectors that are faced by security experts today, and identifying and defining code components that are generally abused by code re-use. This dissertation then presents two practical design solutions. Both defense system designs uphold goals of achieving realistic performance, providing strong security guarantees, being robust for modern application code-bases, and being able to scale across the system at large.
The first practical exploit mitigation design this dissertation presents is MARDU. MARDU is a novel re-randomization approach that utilizes on-demand randomization and the concept of code trampolines to support sharing of code transparently system-wide. To the best of my knowledge, MARDU is the first presented re-randomization technique capable of runtime code sharing for re-randomized code system-wide. Moreover, MARDU is one of the very few re-randomization mechanisms capable of performing seamless live thread migration to newly randomized code without pausing application execution. This dissertation describes the full design, implementation, and evaluation of MARDU to demonstrate its merits and show that careful design can uphold all practical design goals. For instance, scalability is a major challenge for randomization strategies, especially because traditional OS design expects code to be placed in known locations so that it can be reached by multiple processes, while randomization is purposefully trying to achieve the opposite, being completely unpredictable. This clash in expectations between system and defense design breaks a few very important assumptions for an application's runtime environment. This forces most randomization mechanisms to abandon the hope of upholding memory deduplication. MARDU resolves this challenge by applying trampolines to securely reach functions protected under secure memory. Even with this new calling convention in place, MARDU shows re-randomization degradation can be significantly reduced without sacrificing randomization entropy. Moreover, MARDU shows it is capable of defeating prominent code re-use variants with this practical design.
This dissertation then presents its second practical exploit mitigation solution, BASTION. BASTION is a fine-grained system call filtering mechanism aimed at significantly strengthening the security surrounding system calls. Like MARDU, BASTION upholds the principles of this dissertation and was implemented with practicality in mind. BASTION's design is based on empirical observation of what a legitimate system call invocation consists of. BASTION introduces System Call Integrity to enforce the correct and intended use of system calls within a program. In order to enforce this novel security policy, BASTION proposes three new specialized contexts for the effective enforcement of legitimate system call usage. Namely, these contexts enforce that: system calls are only invoked with the correct calling convention, system calls are reached through legitimate control-flow paths, and all system call arguments are free from attacker corruption. By enforcing System Call Integrity with the previously mentioned contexts, this dissertation adds further evidence that context-sensitive defense strategies are superior to context-insensitive ones. BASTION is able to prevent over 32 real-world and synthesized exploits in its security evaluation and incurs negligible performance overhead (0.60%-2.01%). BASTION demonstrates that narrow and specialized exploit mitigation designs can be effective in more than one front, to the point that BASTION not only revents code re-use, but is capable of defending against any attack class that requires the utilization of system calls. / Doctor of Philosophy / Limited security defense deployment and weak security has serious consequences, as large scale cyber-attacks are now a common occurrence. This may be surprising since many defense techniques have been proposed; yet in reality, few have become dopted by the general public. To elaborate, designing an ideal defense that is strong security-wise but does not use any computer resources is challenging. In practice, there is no free lunch, and therefore a design must consider how to best balance security with performance in an effort to be practical for users to deploy their defense. Common tradeoffs include adverse side-effects such as slowing down user applications or imposing significant memory usage. Therefore, practical and strong defense design is important to promote integration into the next generation of computer hardware and software. By sustaining practical design, the needed jump between a proof-of-concept and implementing it on commodity computer chips is substantially smaller. A practical defense should foremost guarantee strong levels of security and should not slow down a user's applications. Ideally, a practical defense is implemented to the point it seems invisible to the user and they don't even notice it. However, balancing practicality with strong security is hard to achieve in practice.
This dissertation first reviews the current security landscape - specifically two important attack strategies are examined. First, code re-use attacks, are exactly what they sound like; code re-use essentially reuse various bits and pieces of program code to create an attack. Second, system call abuse. System calls are essential functions that ordinarily allow a user program to talk with a computer's operating system; they enable operations such as a program asking for more memory or reading and writing files. When system calls are maliciously abused, they can cause a computer to use up all its free memory or even launch an attacker-written program. This dissertation goes over how these attacks work and correspondingly explains popular defense strategies that have been proposed by the security community so far.
This dissertation then presents two defense system solutions that demonstrate how a practical defense system could be made. To that end, the full design, implementation, and evaluation of each defense system, named MARDU and BASTION, is presented. This dissertation leverages attack insights as well as compiler techniques to achieve its goal. A compiler is an essential developer tool that converts human written code into a computer program. Moreover, compilers can be used to apply additional optimizations and security hardening techniques to make a program more secure. This dissertation's first defense solution, MARDU, is a runtime randomization defense. MARDU protects programs by randomizing the location of code chunks throughout execution so that attackers cannot find the code pieces they need to create an attack. Notably, MARDU is the first randomization defense that is able to be seamlessly deployed system-wide and is backwards compatible with programs not outfitted with MARDU. This dissertation's second defense solution, BASTION, is a defense system that strictly focuses on protection of system calls in a program. As mentioned earlier, system calls are security critical functions that allow a program to talk a computer operating system. BASTION protects the entire computer by ensuring that every time a system call is called by a user program, it was rightfully requested by the program and not maliciously by an attacker. BASTION verifies this request is legitimate by confirming that the current program state meets a certain set of criteria.
| Identifer | oai:union.ndltd.org:VTETD/oai:vtechworks.lib.vt.edu:10919/113108 |
| Date | 09 January 2023 |
| Creators | Jelesnianski, Christopher Stanislaw |
| Contributors | Electrical and Computer Engineering, Min, Chang Woo, Jang, Yeong Jin, Yao, Danfeng, Xiong, Wenjie, Zeng, Haibo |
| Publisher | Virginia Tech |
| Source Sets | Virginia Tech Theses and Dissertation |
| Language | English |
| Detected Language | English |
| Type | Dissertation |
| Format | ETD, application/pdf |
| Rights | Creative Commons Attribution 4.0 International, http://creativecommons.org/licenses/by/4.0/ |
Page generated in 0.0035 seconds