Operating system security has become a growing concern these days. As the complexity of software layers increases, the vulnerabilities that can be exploited by adversaries increases. Rootkits are gaining much attention these days in cyber-security. Rootkits are installed by an adversary after he/she gains elevated access to the computer system. Rootkits are used to maintain a consistent undetectable presence in the computer system and help as a toolkit to hide all the malware activities from the system administrator and anti-malware tools. Current defense mechanism used to prevent such activities is to strengthen the OS kernel and fix the known vulnerabilities. Software tools are developed at the OS or virtual machine monitor (VMM) levels to monitor the integrity of the kernel and try to catch any suspicious activity after infection. Recognizing the failure of software techniques and attempting to solve the endless war between the anti-rootkit and rootkit camps, in this thesis, we propose an autonomic architecture called SHARK, or Secure Hardware support Against RootKits. This new hardware architecture provides system-level security against the stealth activities of rootkits without trusting the entire software stack. It enhances the relationship of the OS and hardware and rules out the possibility of any hidden activity even when the OS is completely compromised. SHARK proposes a novel hardware manager that provides secure association with every software context making use of hardware resources. It helps system administrators to obtain feedback directly from the hardware to reveal all running processes. This direct feedback makes it impossible for rootkits to conceal running software contexts from the system administrator. We emulated the proposed architecture SHARK by using Bochs hardware simulator and a modified Linux kernel version 2.6.16.33 for the proposed architectural extension. In our emulated environment, we installed several real rootkits to compromise the kernel and concealed malware processes. SHARK is shown to be very effective in defending against a variety of rootkits employing different software schemes. Also, we performed performance analysis using SIMICS simulations and the results show a negligible overhead, making the proposed solution very practical.

Transport Layer Security (TLS) is a vital component to the security ecosystem and the most popular security protocol used on the Internet today. Despite the strengths of the protocol, numerous vulnerabilities result from its improper use in practice. Some of these vulnerabilities arise from weaknesses in authentication, from the rigidity of the trusted authority system to the complexities of client certificates. Others result from the misuse of TLS by developers, who misuse complicated TLS libraries, improperly validate server certificates, employ outdated cipher suites, or deploy other features insecurely. To make matters worse, system administrators and users are powerless to fix these issues, and lack the ability to properly control how their own machines communicate securely online. In this dissertation we argue that the problems described are the result of an improper placement of security responsibilities. We show that by placing TLS services in the operating system, both new and existing applications can be automatically secured, developers can easily use TLS without intimate knowledge of security, and security settings can be controlled by administrators. This is demonstrated through three explorations that provide TLS features through the operating system. First, we describe and assess TrustBase, a service that repairs and strengthens certificate-based authentication for TLS connections. TrustBase uses traffic interception and a policy engine to provide administrators fine-tuned control over the trust decisions made by all applications on their systems. Second, we introduce and evaluate the Secure Socket API (SSA), which provides TLS as an operating system service through the native POSIX socket API. The SSA enables developers to use modern TLS securely, with as little as one line of code, and also allows custom tailoring of security settings by administrators. Finally, we further explore a modern approach to TLS client authentication, leveraging the operating system to provide a generic platform for strong authentication that supports easy deployment of client authentication features and protects user privacy. We conclude with a discussion of the reasons for the success of our efforts, and note avenues for future work that leverage the principles exhibited in this work, both in and beyond TLS.