<p>Abstract：The kernel system of Windows is more thoroughly exposed to people. So, thekernel-level Rootkits techniques are now laid on greater emphasis. It is very importantto maintain the security of computers and to conduct an in-depth research on theoperational mechanism by using kernel-level Rootkits in hiding its traces. Since theinvolved core techniques are beginning to catch on nowadays, we should analyzesome new key techniques employed for application of Rootkits, discuss the specificmethods and propose a set of defense strategy for computer security.</p>

Abstract：The kernel system of Windows is more thoroughly exposed to people. So, thekernel-level Rootkits techniques are now laid on greater emphasis. It is very importantto maintain the security of computers and to conduct an in-depth research on theoperational mechanism by using kernel-level Rootkits in hiding its traces. Since theinvolved core techniques are beginning to catch on nowadays, we should analyzesome new key techniques employed for application of Rootkits, discuss the specificmethods and propose a set of defense strategy for computer security.

This paper describes information about current most widespread methods, which are used by rootkits. It contains basic information connected with development of rootkits, such as process registers, memory protection and native API of Windows operation system. The primary objective of this paper is to provide overview of techniques, such as hooking, code patching and direct kernel object modification, which are used by rootkits and present methods to detect them. These methods will be then implemented by detection and removal tools of rootkits based on these techniques.

The use of virtualized environments continues to grow for efficient utilization of the available compute resources. Hypervisors virtualize the underlying hardware resources and allow multiple Operating Systems to run simultaneously on the same infrastructure. Since the hypervisor is installed at a higher privilege level than the Operating Systems in the software stack it is vulnerable to rootkits that can modify the environment to gain control, crash the system and even steal sensitive information. Thus, runtime integrity measurement of the hypervisor is essential. The currently proposed solutions achieve the goal by relying either partially or entirely on the features of the hypervisor itself, causing them to lack stealth and leaving themselves vulnerable to attack. We have developed a performance sensitive methodology for identifying rootkits in hypervisors from System Management Mode (SMM) while using the features of SMI Transfer Monitor (STM). STM is a recent technology from Intel and it is a virtual machine manager at the firmware level. Our solution extends a research prototype called EPA-RIMM, developed by Delgado and Karavanic at Portland State University. Our solution extends the state of the art in that it stealthily performs measurements of hypervisor memory and critical data structures using firmware features, keeps performance perturbation to acceptable levels and leverages the security features provided by the STM. We describe our approach and include experimental results using a prototype we have developed for Xen hypervisor on Minnowboard Turbot, an open hardware platform.

Computer systems are subject to a range of attacks that can compromise their intended operations. Conventional wisdom states that once a system has been compromised, the only way to recover is to format and reinstall. In this work, we present methods to automatically recover or self-heal from a compromise. We term the system an intrusion recovery system. The design consists of a layered architecture in which the production system and intrusion recovery system run in separate isolated virtual machines. The intrusion recovery system monitors the integrity of the production system and repairs state if a compromise is detected. A method is introduced to track the dynamic control flow graph of the production system guest kernel. A prototype of the system was built and tested against a suite of rootkit attacks. The system was able to recover from all attacks at a cost of about a 30% performance penalty.

A Methodology for Detecting and Classifying Rootkit Exploits John G. Levine 164 Pages Directed by Dr. Henry L. Owen We propose a methodology to detect and classify rootkit exploits. The goal of this research is to provide system administrators, researchers, and security personnel with the information necessary in order to take the best possible recovery actions concerning systems that are compromised by rootkits. There is no such methodolgoy available at present to perform this function. This may also help to detect and fingerprint additional instances and prevent further security instances involving rootkits. A formal framework was developed in order to define rootkit exploits as an existing rootkit, a modification to an exisiting, or an entirely new rootkit. A methodology was then described in order to apply this framework against rootkits that are to be investigated. We then proposed some new methods to detect and characterize specific types of rootkit exploits. These methods consisted of identifying unique string signatures of binary executable files as well as examining the system call table within the system kernel. We established a Honeynet in order to aid in our research efforts and then applied our methodology to a previously unseen rootkit that was targeted against the Honeynet. By using our methodology we were able to uniquely characterize this rootkit and identify some unique signatures that could be used in the detection of this specific rootkit. We applied our methodolgy against nine additional rootkit exploits and were were able to identify unique characterstics for each of these rootkits. These charactersitics could also be used in the prevention and detection of these rootkits.

In this thesis, we address the problem faced by modern operating systems due to the exploitation of Hardware-Assisted Full-Virtualization technology by attackers. Virtualization technology has been of growing importance these days. With the help of such a technology, multiple operating systems can be run on a single piece of hardware, with little or no modification to the operating system. Both Intel and AMD have contributed to x86 full-virtualization through their respective instruction set architectures. Hardware virtualization extensions can be found in almost all x86 processors these days. Hardware virtualization technologies have opened a whole new frontier for a new kind of attack. A system hacker can abuse hardware virualization technology to gain control over an operating system on-the-fly (i.e., without a system restart) by installing a thin Virtual Machine Monitor (VMM) below the native operating system. Such a VMM based malware is termed a Hardware-Assisted Virtual Machine (HVM) rootkit. We discuss the technique used by a rootkit named Blue Pill to subvert the Windows Vista operating system by exploiting the AMD-V (codenamed "Pacifica") virtualization extensions. HVM rootkits do not hook any operating system code or data regions; hence detecting the existence of such malware using conventional techniques becomes extremely difficult. This thesis discusses existing methods to detect such rootkits and their inefficiencies. In this work, we implement a proof-of-concept HVM rootkit using Intel-VT hardware virtualization technology and also discuss how such an attack can be defended against by using an autonomic architecture called SHARK, which was proposed by Vikas et al., in MICRO 2008.

Operating system security has become a growing concern these days. As the complexity of software layers increases, the vulnerabilities that can be exploited by adversaries increases. Rootkits are gaining much attention these days in cyber-security. Rootkits are installed by an adversary after he/she gains elevated access to the computer system. Rootkits are used to maintain a consistent undetectable presence in the computer system and help as a toolkit to hide all the malware activities from the system administrator and anti-malware tools. Current defense mechanism used to prevent such activities is to strengthen the OS kernel and fix the known vulnerabilities. Software tools are developed at the OS or virtual machine monitor (VMM) levels to monitor the integrity of the kernel and try to catch any suspicious activity after infection. Recognizing the failure of software techniques and attempting to solve the endless war between the anti-rootkit and rootkit camps, in this thesis, we propose an autonomic architecture called SHARK, or Secure Hardware support Against RootKits. This new hardware architecture provides system-level security against the stealth activities of rootkits without trusting the entire software stack. It enhances the relationship of the OS and hardware and rules out the possibility of any hidden activity even when the OS is completely compromised. SHARK proposes a novel hardware manager that provides secure association with every software context making use of hardware resources. It helps system administrators to obtain feedback directly from the hardware to reveal all running processes. This direct feedback makes it impossible for rootkits to conceal running software contexts from the system administrator. We emulated the proposed architecture SHARK by using Bochs hardware simulator and a modified Linux kernel version 2.6.16.33 for the proposed architectural extension. In our emulated environment, we installed several real rootkits to compromise the kernel and concealed malware processes. SHARK is shown to be very effective in defending against a variety of rootkits employing different software schemes. Also, we performed performance analysis using SIMICS simulations and the results show a negligible overhead, making the proposed solution very practical.

Liveanalys är ett begrepp som i detta arbete innebär att man undersöker ett datorsystem under tiden det är igång. Detta kan göras av flera skäl, t.ex. när det är risk för att kryptering finns på systemet vilket kan aktiveras när det stängs ner. Annars är det vanligt om man vill undersöka nätverkskopplingar, aktiva processer eller andra företeelser som kan vara volatila, dvs. försvinner när systemet stängs ner. Detta arbete kommer att ha fokus på motåtgärder vid forensisk liveanalys och redogöra för olika metoder och strategier som kan användas för dessa motåtgärder. Vi har bland annat skrivit ett program som automatiskt stänger ner systemet när man sätter i ett USB-minne eller annan media. Dessa media är oftast de man har sina forensiska program på när man ska göra en liveanalys. Andra viktiga element i arbetet är användning av kryptering, tidstämplar och sabotagekod för att försvåra liveanalysen. Vår analys i ämnet visar att det är relativt enkelt att förhindra att en liveanalys kan utföras på ett tillförlitligt sätt. / Live Analysis is a concept that in this paper means analyzing a computer system while it is running. This can be done for several reasons, such as when there is a risk that the system has encryption which can be activated when the system shuts down. Otherwise, it is common if you want to examine network connections, active processes or other phenomena that can be volatile, i.e. disappear when the system shuts down. This work will focus on countermeasures to live forensic analysis and describe different methods and strategies that can be used for these countermeasures. For example, we wrote a program that automatically shuts down the system when you insert a USB memory stick or any other media. These are usually the media which you have your forensic programs on when you do a live analysis. Other important elements of the work are the use of encryption, timestamps and malicious code for challenging live analysis. Our analysis of the topic shows that it is relatively easy to prevent that a live analysis can be performed in a reliable way.

M.Sc. / Anyone who uses a computer for work or recreational purposes has come across one or all of the following problems directly or indirectly (knowingly or not): viruses, worms, trojans, rootkits and botnets. This is especially the case if the computer is connected to the Internet. Looking at the statistics in [1] we can see that although malware detection techniques are detecting and preventing malware, they do not guarantee a 100% detection and or prevention of malware. Furthermore the statistics in [2] show that malware infection rates are increasing around the world at an alarming rate. The statistics also show that there are a high number of new malware samples being discovered every month and that 31% of malware attacks resulted in data loss [3], with 10% of companies reporting the loss of sensitive business data [4][5]. The reason for not being able to achieve a 100% detection and / or prevention of malware is because malware authors make use of sophisticated techniques such as code obfuscation in order to prevent malware from being detected. This has resulted in the emergence of malware known as polymorphic and metamorphic malware. The aforementioned malware poses serious challenges for anti-malware software specifically signature based techniques. However a more serious threat that needs to be addressed is that of rootkits. Rootkits can execute at the same privilege level as the Operating System (OS) itself. At this level the rootkit can manipulate the OS such that it can distribute other malware, hide existing malware, steal information, hide itself, disable anti-malware software etc all without the knowledge of the user. It is clear from the statistics that anti-malware products are not working because infection rates continue to rise and companies and end users continue to fall victims of these attacks. Therefore this dissertation will address the problem that current anti-malware techniques are not working. The main objective of this dissertation is to create a framework called ATE (Anti-malware Technique Evaluator) that can be used to critically evaluate current commercial anti-malware products. The framework will achieve this by identifying the current vulnerabilities that exist in commercial anti-malware products and the operating system. The prior will be achieved by making use of two rootkits, the Evader rootkit and the Sabotager rootkit, which were specifically developed to support the anti-malware product evaluation. Finally an anti-malware architecture we called External Malware Scanner (EMS), will be proposed to address the identified vulnerabilities.