This thesis deals with the generic detection of bootkits which are relatively a new kind of malicious sofware falling into the category of rootkits. The definition of malicious software is presented along with several examples. Then the attention is paid to the rootkits in the context of Microsoft Windows operating systems. This section lists several techniques used by rootkits. After that, the ways of preventing and detecting rootkits are mentioned. Bootkits are known for infecting hard disks Master Boot Record (MBR). The structure of the MBR is described along with the example of hard disk partitioning. Afterwards, the processor instruction set is outlined and the disassembly of Windows 7 MBR is given. The rest of the thesis is devoted to a description of the course of operating system bootkit infection, bootkit prevention, analysis of infected MBR samples, and in particular to the design, implementation and testing of the generic MBR infection detector.

Τα rootkit επιτρέπουν στον επιτιθέμενο χρήστη να συνεχίσει να έχει πρόσβαση σε ήδη παραβιασμένο σύστημα για μεγάλο χρονικό διάστημα μετά την παραβίαση, χωρίς να γίνει αντιληπτός από τον νόμιμο διαχειριστή. Στην εργασία αυτή αναλύ- θηκαν οι διάφορες τεχνικές οι οποίες χρησιμοποιούνται από τους επιτιθέμενους, με έμφαση στα rootkits επιπέδου πυρήνα, και αναπτύχθηκε πρόγραμμα rootkit το οποίο κάνει χρήση προηγμένων τεχνικών οι οποίες του επιτρέπουν τη λειτουρ- γία ακόμη και στις νεότερες εκδόσεις του πυρήνα. Στη συνέχεια αναλύθηκαν οι τεχνικές ανίχνευσης και αναπτύχθηκε εφαρμογή ανίχνευσης δυο διαφορετικών κατηγοριών rootkit. Η εφαρμογή έχει επίσης τη δυνατότητα να εξουδετερώσει τα rootkit της πρώτης κατηγορίας έτσι ώστε να αποκατασταθεί το σύστημα. / -

We present a rootkit prevention system, namely DARK that tracks suspicious Linux loadable kernel modules (LKM) at a granular level by using on-demand emulation, a technique that dynamically switches a running system between virtualized and emulated execution. Combining the strengths of emulation and virtualization, DARK is able to thoroughly capture the activities of the target module in a guest operating system (OS), while maintaining reasonable run-time performance. To address integrity-violation and confidentiality-violation rootkits, we create a group of security policies that can detect all available Linux rootkits. It is shown that normal guest OS performance is unaffected. The performance is only decreased when rootkits attempt to run, while most rootkits are detected at installation. Next, we present a sandbox-based malware analysis system called Rkprofiler that dynamically monitors and analyzes the behavior of Windows kernel malware. Kernel malware samples run inside a virtual machine (VM) that is supported and managed by a PC emulator. Rkprofiler provides several capabilities that other malware analysis systems do not have. First, it can detect the execution of malicious kernel code regardless of how the monitored kernel malware is loaded into the kernel and whether it is packed or not. Second, it captures all function calls made by the kernel malware and constructs call graphs from the trace files. Third, a technique called aggressive memory tagging (AMT) is proposed to track the dynamic data objects that the kernel malware visits. Last, Rkprofiler records and reports the hardware access events of kernel malware (e.g., MSR register reads and writes). Our evaluation results show that Rkprofiler can quickly expose the security-sensitive activities of kernel malware and thus reduces the effort exerted in conducting tedious manual malware analysis.

Riding on recent advances in computing and networking, our society is now experiencing the evolution into the age of information. While the development of these technologies brings great value to our daily life, the lucrative reward from cyber-crimes has also attracted criminals. As computing continues to play an increasing role in the society, security has become a pressing issue. Failures in computing systems could result in loss of infrastructure or human life, as demonstrated in both academic research and production environment. With the continuing widespread of malicious software and new vulnerabilities revealing every day, protecting the heterogeneous computing systems across the Internet has become a daunting task. Our approach to this challenge consists of two directions. The first direction aims to gain a better understanding of the inner working of both attacks and defenses in the cyber environment. Meanwhile, our other direction is designing secure systems in adversarial environment. / Ph. D.

The debug register rootkit is a special type of rootkit that has existed for over a decade, and is told to be undetectable by any scanning tools. It exploits the debug registers in Intel’s IA-32 processor architecture. This paper investigates the debug register rootkit to find out why it is considered a threat, and which malware removal tools have implemented detection algorithms against this threat. By implementing and running a debug register rootkit against the most popular Linux tools, new conclusions about the protection of the Linux system can be reached. Recently, debug register rootkits were found on Windows as well. This project intends to bring knowledge about the problem and investigate if there are any threats. Our study has shown that still after 12 years, the most popular tools for the Linux operating system have not implemented any detection algorithms against this threat. The security industry may need to prepare for this threat in case it is spread further.

The debug register rootkit is a special type of rootkit that has existed for over a decade, and is told to be undetectable by any scanning tools. It exploits the debug registers in Intel’s IA-32 processor architecture. This paper investigates the debug register rootkit to find out why it is considered a threat, and which malware removal tools have implemented detection algorithms against this threat. By implementing and running a debug register rootkit against the most popular Linux tools, new conclusions about the protection of the Linux system can be reached. Recently, debug register rootkits were found on Windows as well. This project intends to bring knowledge about the problem and investigate if there are any threats. Our study has shown that still after 12 years, the most popular tools for the Linux operating system have not implemented any detection algorithms against this threat. The security industry may need to prepare for this threat in case it is spread further.

This paper describes information about current most widespread methods, which are used by rootkits. It contains basic information connected with development of rootkits, such as process registers, memory protection and native API of Windows operation system. The primary objective of this paper is to provide overview of techniques, such as hooking, code patching and direct kernel object modification, which are used by rootkits and present methods to detect them. These methods will be then implemented by detection and removal tools of rootkits based on these techniques.

Widespread adoption of virtualization has resulted in an increased interest in Virtual Machine (VM) introspection. To perform useful analysis of the introspected VMs, hypervisors must deal with the semantic gap between the low-level information available to them and the high-level OS abstractions they need. To bridge this gap, systems have proposed making assumptions derived from the operating system source code or symbol information. As a consequence, the resulting systems create a tight coupling between the hypervisor and the operating systems run by the introspected VMs. This coupling is undesirable because any change to the internals of the operating system can render the output of the introspection system meaningless. In particular, malicious software can evade detection by making modifications to the introspected OS that break these assumptions. Instead, in this thesis, we introduce Architectural Introspection, a new introspection approach that does not require information about the internals of the introspected VMs. Our approach restricts itself to leveraging constraints placed on the VM by the hardware and the external environment. To interact with both of these, the VM must use externally specified interfaces that are both stable and not linked with a specific version of an operating system. Therefore, systems that rely on architectural introspection are more versatile and more robust than previous approaches to VM introspection. To illustrate the increased versatility and robustness of architectural introspection, we describe two systems, Patagonix and P2, that can be used to detect rootkits and unpatched software, respectively. We also detail Attestation Contracts, a new approach to attestation that relies on architectural introspection to improve on existing attestation approaches. We show that because these systems do not make assumptions about the operating systems used by the introspected VMs, they can be used to monitor both Windows and Linux based VMs. We emphasize that this ability to decouple the hypervisor from the introspected VMs is particularly useful in the emerging cloud computing paradigm, where the virtualization infrastructure and the VMs are managed by different entities. Finally, we show that these approaches can be implemented with low overhead, making them practical for real world deployment.

Widespread adoption of virtualization has resulted in an increased interest in Virtual Machine (VM) introspection. To perform useful analysis of the introspected VMs, hypervisors must deal with the semantic gap between the low-level information available to them and the high-level OS abstractions they need. To bridge this gap, systems have proposed making assumptions derived from the operating system source code or symbol information. As a consequence, the resulting systems create a tight coupling between the hypervisor and the operating systems run by the introspected VMs. This coupling is undesirable because any change to the internals of the operating system can render the output of the introspection system meaningless. In particular, malicious software can evade detection by making modifications to the introspected OS that break these assumptions. Instead, in this thesis, we introduce Architectural Introspection, a new introspection approach that does not require information about the internals of the introspected VMs. Our approach restricts itself to leveraging constraints placed on the VM by the hardware and the external environment. To interact with both of these, the VM must use externally specified interfaces that are both stable and not linked with a specific version of an operating system. Therefore, systems that rely on architectural introspection are more versatile and more robust than previous approaches to VM introspection. To illustrate the increased versatility and robustness of architectural introspection, we describe two systems, Patagonix and P2, that can be used to detect rootkits and unpatched software, respectively. We also detail Attestation Contracts, a new approach to attestation that relies on architectural introspection to improve on existing attestation approaches. We show that because these systems do not make assumptions about the operating systems used by the introspected VMs, they can be used to monitor both Windows and Linux based VMs. We emphasize that this ability to decouple the hypervisor from the introspected VMs is particularly useful in the emerging cloud computing paradigm, where the virtualization infrastructure and the VMs are managed by different entities. Finally, we show that these approaches can be implemented with low overhead, making them practical for real world deployment.

This master's thesis deals with Linux kernel security from the attacker's point of view. It maps methods and techniques of disguising the computing resources used by today's IT pirates. The thesis presents a unique method of attack directed on the system call interface and implemented in the form of two tools (rootkits). The thesis consists of a theoretical and a practical part. Emphasis is placed especially on the practical part, which manifests the presented information in the form of experiments and shows its use in real life. Readers are systematically guided as far as the creation of a unique rootkit, which is capable of infiltrating the Linux kernel by a newly discovered method -- even without support of loadable modules. A part of the thesis focuses on the issue of detecting the discussed attacks and on effective defence against them.