Computers were not built with security in mind. As such, security has and still often takes a back seat to performance. However, in an era where there is so much sensitive data being stored, with cloud storage and huge customer databases, much has to be done to keep this data safe from intruders. Control flow hijacking attacks, stemming from a basic code injection attack to return-into-libc and other code re-use attacks, are among the most dangerous attacks. Currently available solutions, like Data execution prevention that can prevent a user from executing writable pages to prevent code injection attacks, do not have an efficient solution for protecting against code re-use attacks, which can execute valid code in a malicious order. To protect against control flow hijacking attacks, this work proposes architecture to make Control Flow Integrity, a solution that proposes to validate control flow against pre-computed control flow graph, practical. Current implementations of Control Flow Integrity have problems with code modularity, performance, or scalability, so I propose Dynamic Bloom Cache, a blocked-Bloom-filter-based approach, to solve current implementation issues.

Embedded software today is pervasive: they can be found everywhere, from coffee makers and medical devices, to cars and aircraft. Embedded software today is also open and connected to the Internet, exposing them to external attacks that can cause its Control-Flow Integrity (CFI) to be violated. Control-Flow Integrity is an important safety property of software, which ensures that the behavior of the software is not inadvertently changed. The violation of CFI in software can cause unintended behaviors, and can even lead to catastrophic incidents in safety-critical systems. This dissertation develops a two-part approach for CFI: (i) prescribing source-code safetychecks, that prevent the root-causes of CFI, that programmers can insert themselves, and (ii) formally proving CFI for the machine-code of programs with source-code safety-checks. First, our prescribed safety-checks, when applied, prevent the root-causes of CFI, thereby enabling software to recover from CFI violations in a customizable way. In addition, our prescribed safety-checks are visible to programmers, empowering them to ensure that the behavior of their software is not inadvertently changed by the prescribed safety-checks. However, programmer-inserted safety-checks may be incomplete. Thus, current techniques for proving CFI, which assume that safety-checks are complete, may not work. Second, this dissertation develops a logic approach that automates formal proofs of CFI for the machine-code of software containing both source-code CFI safety-checks and system calls. We extend an existing trustworthy Hoare logic with new proof rules, proof tactics, and a novel proof-search algorithm, which exploit the principle of local reasoning for safety properties to automatically generate CFI proofs for the machine-code of programs compiled with our prescribed source-code safety-checks. To the best of our knowledge, our approach to CFI is the first to combine programmer-visible source-code enforcement mechanisms for CFI–enabling programmers to customize them and observe that their software is not inadvertently changed–with machine-code proofs of CFI that can be automated, and that does not require a trusted or verified compiler to ensure its proven properties hold in machine-code. We evaluate our CFI approach on realistic embedded software. We evaluate our approach on the MiBench and WCET benchmarks, implementations of common file utilities, and programs interfacing with hardware inputs and outputs on the Raspberry Pi single-board-computer. The variety of our target programs, and our ability to support useful features such as file and hardware inputs and outputs, demonstrate the wide applicability of our approach.

<div>Control-flow hijacking attacks allow adversaries to take over seemingly benign software, e.g., a web browser, and cause it to perform malicious actions, i.e., grant attackers a shell on</div><div>a system. Such control-flow hijacking attacks exploit a gap between high level language semantics and the machine language that they are compiled to. In particular, systems</div><div>software such as web browsers and servers are implemented in C/C++ which provide no runtime safety guarantees, leaving memory and type safety exclusively to programmers. Compilers are ideally situated to perform the required analysis and close the semantic gap between C/C++ and machine languages by adding instrumentation to enforce full or partial memory safety.</div><div><br></div><div><div>In unprotected C/C++, adversaries must be assumed to be able to control to the contents of any writeable memory location (arbitrary writes), and to read the contents of any readable memory location (arbitrary reads). Defenses against such attacks range from enforcing full memory safety to protecting only select information, normally code pointers to prevent control-flow hijacking attacks. We advance the state of the art for control-flow hijacking</div><div>defenses by improving the enforcement of full memory safety, as well as partial memory safety schemes for protecting code pointers.</div></div><div><br></div><div><div>We demonstrate a novel mechanism for enforcing full memory safety, which denies attackers both arbitrary reads and arbitrary writes at half the performance overhead of the</div><div>prior state of the art mechanism. Our mechanism relies on a novel metadata scheme for maintaining bounds information about memory objects. Further, we maintain the application</div><div>binary interface (ABI), support all C/C++ language features, and are mature enough to protect all of user space, and in particular libc.</div></div><div><br></div><div><div>Backwards control-flow transfers, i.e., returns, are a common target for attackers. In particular, return-oriented-programming (ROP) is a code-reuse attack technique built around corrupting return addresses. Shadow stacks prevent ROP attacks by providing partial memory safety for programs, namely integrity protecting the return address. We provide a full taxonomy of shadow stack designs, including two previously unexplored designs, and demonstrate that with compiler support shadow stacks can be deployed in practice. Further we examine the state of hardware support for integrity protected memory regions within a process’ address space. Control-Flow Integrity (CFI) is a popular technique for securing forward edges, e.g., indirect function calls, from being used for control-flow hijacking attacks. CFI is a form of partial memory safety that provides weak integrity for function pointers by restricting them to a statically determined set of values based on the program’s control-flow graph. We survey existing techniques, and quantify the protection they provide on a per callsite basis.</div><div>Building off this work, we propose a new security policy, Object Type Integrity, which provides full integrity protection for virtual table pointers on a per object basis for C++</div><div>polymorphic objects.</div></div>

Malicious attacks on computer systems attempt to obtain and maintain illicit control over the victim system. To obtain unauthorized access, they often exploit vulnerabilities in the victim system, and to maintain illicit control, they apply various hiding techniques to remain stealthy. In this dissertation, we discuss and present solutions for two classes of security problems: TOCTTOU (time-of-check-to-time-of-use) and K-Queue. TOCTTOU is a vulnerability that can be exploited to obtain unauthorized root access, and K-Queue is a hiding technique that can be used to maintain stealthy control of the victim kernel. The first security problem is TOCTTOU, a race condition in Unix-style file systems in which an attacker exploits a small timing gap between a file system call that checks a condition and a use kernel call that depends on the condition. Our contributions on TOCTTOU include: (1) A model that enumerates the complete set of potential TOCTTOU vulnerabilities; (2) A set of tools that detect TOCTTOU vulnerabilities in Linux applications such as vi, gedit, and rpm; (3) A theoretical as well as an experimental evaluation of security risks that shows that TOCTTOU vulnerabilities can no longer be considered "low risk" given the wide-scale deployment of multiprocessors; (4) An event-driven protection mechanism and its implementation that defend Linux applications against TOCTTOU attacks at low performance overhead. The second security problem addressed in this dissertation is kernel queue or K-Queue, which can be used by the attacker to achieve continual malicious function execution without persistently changing either kernel code or data, which prevents state-of-the-art kernel integrity monitors such as CFI and SBCFI from detecting them. Based on our successful defense against a concrete instance of K-Queue-driven attacks that use the soft timer mechanism, we design and implement a solution to the general class of K-Queue-driven attacks, including (1) a unified static analysis framework and toolset that can generate specifications of legitimate K-Queue requests and the checker code in an automated way; (2) a runtime reference monitor that validates K-Queue invariants and guards such invariants against tampering; and (3) a comprehensive experimental evaluation of our static analysis framework and K-Queue Checkers.

<div>System programming languages such as C and C++ are ubiquitously used for systems software such as browsers and servers due to their flexibility and high performance. However, this flexibility comes with a price of lack of memory and type safety.</div><div><br></div><div>Control-Flow Hijacking (CFH), by taking advantage of the inherent lack of memory and type safety, has become one of the most common attack vectors against C/C++ programs. In such attacks, an attacker attempts to divert the normal control flow of the program to an attacker-controlled location. The most prominent defense against these kind of attacks is Control-Flow Integrity (CFI), which restricts the attack surface by limiting the set of possible targets for each indirect control-flow transfer. However, current analyses for the CFI target sets are highly conservative. Due to the ambiguity and imprecision in the analyses, CFI restricts adversaries to an over-approximation of the possible targets of individual indirect call sites. State-of-the-art CFI approaches fail to protect against special attack classes such as over-writing variadic function arguments. Furthermore, mitigation of control-flow attacks is not explored to its full potential in the context of language boundaries in current literature. Hence, we need effective solution to improve the precision of the CFI approaches as well as strong protection mechanisms against commonly abused corner cases.</div><div><br></div><div>We leverage the effectiveness of dynamic analysis in deriving a new approach to efficiently mitigate control-flow hijacking attacks. We present Ancile, a novel mechanism to improve the precision of the CFI mechanism by debloating any extraneous targets from the indirect control-flow transfers. We replaced the traditional static analysis approach for target discovery with seed demonstrated fuzzing. We have evaluated the effectiveness of our proposed mechanism with standard SPEC CPU benchmarks and other popular C and C++ applications.</div><div><br></div><div>To ensure complete security of C and C++ programs, we need to shield commonly exploited corners of C/C++ such as variadic functions. We performed extensive case studies to show the prevalence of such functions and their exploits. We also developed a sanitizer, HexVASAN, to effectively type-check and prevent any attack via variadic functions. CFH attacks, by abusing the difference of managed languages and their underlying system languages, are very frequent in client and server side programs. In order to safe-guard the control-flows in language boundaries, we propose a new mechanism, FitJit, to enforce type integrity. Finally, to understand the effectiveness of the dynamic analysis, we present Artemis, a comprehensive study of binary analysis on real world applications.</div>

Memory corruption is one of the oldest and most prominent problems in the field of computer security. In order to protect the vulnerabilities that arise from memory corruption, a mitigation technique called Control-flow Integrity (CFI) was developed. The Android Open Source Project utilizes a specific implementation of the CFI policy called forward-edge CFI in the compilation of the Android system. However, memory corruption vulnerabilities are still a problem for Android systems. This raises the question: Is forward-edge CFI really effective in mitigating memory corruption vulnerabilities? In this research, the efficacy of forward-edge CFI in terms of mitigating memory corruption vulnerabilities in Android systems is analyzed. This is done by analyzing nine Common Vulnerabilities and Exposures (CVE) in terms of how they can be exploited and whether forward-edge CFI could mitigate them. Additionally, the Android binaries containing the vulnerabilities are analyzed in an attempt to detect the presence of CFI instrumentation. CFI was detected in one of nine vulnerable Android binaries, implying that there exist memory corruption vulnerabilities that forward-edge CFI definitely can not protect. The analysis of nine CVEs showed that five CVEs could be mitigated by forward-edge CFI. These results indicate that forward-edge CFI could definitely mitigate a portion of the memory corruption vulnerabilities plaguing Android systems. However, in order to protect a greater portion of memory corruption vulnerabilities, forward-edge CFI should be combined with other mitigation techniques such as Shadow Stacks.

The Internet has become the primary vector for the delivery of malicious code in cyber attacks, and malware has rapidly become a pervasive critical threat. Anti- malware products offer effective protection from malware threats for servers and endpoint devices using a variety of techniques. Advanced enterprise-level anti-malware products rely on state-of-art behavioral-based detection algorithms, in addition to traditional signature-based mechanisms. These dynamic detection techniques have been around for more than a decade and in response hackers have developed methods to evade them. However, currently known bypass methods require intensive manual labor. Moreover, this manual work has to be repeated whenever a parameter of the environment (such as the payload, operating system, Antivirus version, etc) changes, making these methods impractical. This may lead to the belief that dynamic techniques provide a good deterrence, and hence good protection. In this thesis we evaluate dynamic techniques. Specifically, we build tools to implement generic unhooking and funneling, and using these tools we show how dynamic techniques can be bypassed with considerably less effort than by fully manual methods. We also extend the repertoire of existing bypass methods and introduce a new malicious function call technique which exploits detection techniques that monitor a limited collection of critical system functions, as well as a method for bypassing guard-page protections. We demonstrate the effectiveness of all our techniques by conducting attacks against two enterprise antivirus products. Our results lead us to conclude that that dynamic techniques do not provide sufficient protection. / Graduate / 2018-02-07 / 0984 / erkanersan@gmail.com