Memory corruption continues to plague modern software systems, as it has for decades. With the emergence of code-reuse attacks which take advantage of these vulnerabilities like Return- Oriented Programming (ROP) or non-control data attacks like Data-Oriented programming (DOP), defenses against these are growing thin. These attacks, and more advanced variations of them, are becoming more difficult to detect and to mitigate. In this arms race, it is critical to not only develop mitigation techniques, but also ways we can effectively deploy those techniques. In this work, we present rave - a framework which takes common design features of defenses against memory corruption and code-reuse and puts them in a real-world setting. Rave consists of two components: librave, the library responsible for static binary analysis and instrumentation, and CRIU-rave, an extended version of the battle-tested process migration tool available for Linux. In our prototype of this framework, we have shown that these tools can be used to rewrite live applications, like NGINX, with enough randomization to disrupt memory corruption attacks. This work is supported in part by ONR under grant N00014-18-1-2022 and NAVSEA/NEEC/NSWC Dahlgren under grant N00174-20-1-0009. / Master of Science / Memory corruption attacks continue to be a concrete threat against modern computer systems. Malicious actors can take advantage of related vulnerabilities to carry out more advance, hard-to-detect attacks which give them control of the target or leak critical information. Many works have been developed to defend against these sophisticated attacks and their triggers (memory corruption), but many struggle to be adopted into the real-world for reasons such as instability or difficulty in deployment. In this work, we introduce rave, a framework which seeks to address issues of stability and deployment by designing a way for defenders to coordinate and apply mitigation techniques in a real-world setting.

<p>Software vulnerabilities in programs and malware deployments have been increasing almost every year since we started measuring them. Information about how to program securely, how malware shall be avoided and technological countermeasures for this are more available than ever. Still, the trend seems to favor the attacker. This thesis tries to visualize the effects of a selection of technological countermeasures that have been proposed by researchers. These countermeasures: non-executable memory, address randomization, system call interception and file integrity monitoring are described along with the attacks they are designed to defend against. The coverage of each countermeasure is then visualized with the help of attack trees. Attack trees are normally used for describing how systems can be attacked but here they instead serve the purpose of showing where in an attack a countermeasure takes effect. Using attack trees for this highlights a couple of important aspects of a security mechanism, such as how early in an attack it is effective and which variants of an attack it potentially defends against. This is done by the use of what we call defensive codes that describe how a defense mechanism counters a sub-goal in an attack. Unfortunately the whole process is not well formalized and depends on many uncertain factors.</p>

Software vulnerabilities in programs and malware deployments have been increasing almost every year since we started measuring them. Information about how to program securely, how malware shall be avoided and technological countermeasures for this are more available than ever. Still, the trend seems to favor the attacker. This thesis tries to visualize the effects of a selection of technological countermeasures that have been proposed by researchers. These countermeasures: non-executable memory, address randomization, system call interception and file integrity monitoring are described along with the attacks they are designed to defend against. The coverage of each countermeasure is then visualized with the help of attack trees. Attack trees are normally used for describing how systems can be attacked but here they instead serve the purpose of showing where in an attack a countermeasure takes effect. Using attack trees for this highlights a couple of important aspects of a security mechanism, such as how early in an attack it is effective and which variants of an attack it potentially defends against. This is done by the use of what we call defensive codes that describe how a defense mechanism counters a sub-goal in an attack. Unfortunately the whole process is not well formalized and depends on many uncertain factors.

Memory corruption is one of the oldest and most prominent problems in the field of computer security. In order to protect the vulnerabilities that arise from memory corruption, a mitigation technique called Control-flow Integrity (CFI) was developed. The Android Open Source Project utilizes a specific implementation of the CFI policy called forward-edge CFI in the compilation of the Android system. However, memory corruption vulnerabilities are still a problem for Android systems. This raises the question: Is forward-edge CFI really effective in mitigating memory corruption vulnerabilities? In this research, the efficacy of forward-edge CFI in terms of mitigating memory corruption vulnerabilities in Android systems is analyzed. This is done by analyzing nine Common Vulnerabilities and Exposures (CVE) in terms of how they can be exploited and whether forward-edge CFI could mitigate them. Additionally, the Android binaries containing the vulnerabilities are analyzed in an attempt to detect the presence of CFI instrumentation. CFI was detected in one of nine vulnerable Android binaries, implying that there exist memory corruption vulnerabilities that forward-edge CFI definitely can not protect. The analysis of nine CVEs showed that five CVEs could be mitigated by forward-edge CFI. These results indicate that forward-edge CFI could definitely mitigate a portion of the memory corruption vulnerabilities plaguing Android systems. However, in order to protect a greater portion of memory corruption vulnerabilities, forward-edge CFI should be combined with other mitigation techniques such as Shadow Stacks.

The Internet of Things is deploying large numbers of bare-metal systems that have no protection against memory corruption and control-flow hijacking attacks. These attacks have enabled unauthorized entry to hotel rooms, malicious control of unmanned aerial vehicles, and invasions of privacy. Using static and dynamic analysis these systems can utilize state-of-the-art testing techniques to identify and<br>prevent memory-corruption errors and employ defenses against memory corruption and control-flow hijacking attacks in bare-metal systems that match or exceed those currently employed on desktop systems. This is shown using three case studies.<br><br>(1) EPOXY which, automatically applies data execution prevention, diversity, stack defenses, and separating privileged code from unprivileged code using a novel<br>technique called privileged overlaying. These protections prevent code injection attacks, and reduce the number of privileged instruction to 0.06% verses an unprotected<br>application.<br><br>(2) Automatic Compartments for Embedded Systems (ACES), which automatically creates compartments that enforce data integrity and code isolation within bare-metal applications. ACES enables exploring policies to best meet security and performance requirements for individual applications. Results show ACES' can form 10s of compartments within a single thread and has a 15% runtime overhead on average.<br><br><div>(3) HALucinator breaks the requirement for specialized hardware to perform bare-metal system testing. This enables state-of-the-art testing techniques –e.g., coverage based fuzzing – to scale with the availability of commodity computers, leading to the discovery of exploitable vulnerabilities in bare-metal systems. <br></div><div><br></div><div>Combined, these case studies advance the security of embedded system several decades and provide essential protections for today’s connected devices.</div>

<p>Program anomaly detection — modeling normal program executions to detect deviations at runtime as cues for possible exploits — has become a popular approach for software security. To leverage high performance modeling and complete tracing, existing techniques however focus on subsets of applications, e.g., on system calls or calls to predefined libraries. Due to limited scope, it is insufficient to detect subtle control-oriented and data-oriented attacks that introduces new illegal call relationships at the application level. Also such techniques are hard to apply on devices that lack a clear separation between OS and the application layer. This dissertation advances the design and implementation of program anomaly detection techniques by providing application context for library and system calls making it powerful for detecting advanced attacks targeted at manipulating intra- and inter-procedural control-flow and decision variables. </p> <p><br></p> <p>This dissertation has two main parts. The first part describes a statically initialized generic calling context program anomaly detection technique LANCET based on Hidden Markov Modeling to provide security against control-oriented attacks at program runtime. It also establishes an efficient execution tracing mechanism facilitated through source code instrumentation of applications. The second part describes a program anomaly detection framework EDISON to provide security against data-oriented attacks using graph representation learning and language models for intra and inter-procedural behavioral modeling respectively.</p> <p><br> This dissertation makes three high-level contributions. First, the concise descriptions demonstrates the design, implementation and extensive evaluation of an aggregation-based anomaly detection technique using fine-grained generic calling context-sensitive modeling that allows for scaling the detection over entire applications. Second, the precise descriptions show the design, implementation, and extensive evaluation of a detection technique that maps runtime traces to the program’s control-flow graph and leverages graphical feature representation to learn dynamic program behavior. Finally, this dissertation provides details and experience for designing program anomaly detection frameworks from high-level concepts, design, to low-level implementation techniques.</p>