In this thesis, we address the problem faced by modern operating systems due to the exploitation of Hardware-Assisted Full-Virtualization technology by attackers. Virtualization technology has been of growing importance these days. With the help of such a technology, multiple operating systems can be run on a single piece of hardware, with little or no modification to the operating system. Both Intel and AMD have contributed to x86 full-virtualization through their respective instruction set architectures. Hardware virtualization extensions can be found in almost all x86 processors these days. Hardware virtualization technologies have opened a whole new frontier for a new kind of attack. A system hacker can abuse hardware virualization technology to gain control over an operating system on-the-fly (i.e., without a system restart) by installing a thin Virtual Machine Monitor (VMM) below the native operating system. Such a VMM based malware is termed a Hardware-Assisted Virtual Machine (HVM) rootkit. We discuss the technique used by a rootkit named Blue Pill to subvert the Windows Vista operating system by exploiting the AMD-V (codenamed "Pacifica") virtualization extensions. HVM rootkits do not hook any operating system code or data regions; hence detecting the existence of such malware using conventional techniques becomes extremely difficult. This thesis discusses existing methods to detect such rootkits and their inefficiencies. In this work, we implement a proof-of-concept HVM rootkit using Intel-VT hardware virtualization technology and also discuss how such an attack can be defended against by using an autonomic architecture called SHARK, which was proposed by Vikas et al., in MICRO 2008.

Continuously changing system configurations and attack methods make information system risk management using traditional methods a formidable task. Traditional qualitative approaches usually lack sufficient measurable detail on which to base confident, cost-effective decisions. Traditional quantitative approaches are burdened with the requirement to collect an abundance of detailed asset value and historical incident data and to apply complex calculations to measure the data precisely in work environments where there are limited resources to collect and process it. To ensure that safeguards (controls) are implemented to protect against a majority of known threats, industry leaders are requiring information processing systems to comply with security standards. The National Institute of Standards and Technology (NIST) Federal Information Risk Management Framework (RMF) and the associated suite of guidance documents describe the minimum security requirements for non-national-security federal information and information systems as mandated by the Federal Information Security Management Act (FISMA), enacted into law on December 17, 2002, as Title III of the E-Government Act of 2002. This study proposes using the Pathfinder procedure to mathematically model an information system FISMA-required security control state and an actual information system security control state. A comparison of these two security control states using the proposed method will generate a quantitative measure of the status of compliance of the actual system with the FISMA-required standard. The quantitative measures generated should provide information sufficient to plan risk mitigation strategy, track system compliance to standard, and allow for the discussion of system compliance with the FISMA-required standard in terms easily understood by participants at various levels of an organization without requiring all to have detailed knowledge of the internals of the security standard or the targeted system. The ability to clearly articulate system compliance status and risk mitigation requirements is critical to gaining the support of upper-level management whose responsibility it is to allocate funds sufficient to support government security programs.