The use of virtualized environments continues to grow for efficient utilization of the available compute resources. Hypervisors virtualize the underlying hardware resources and allow multiple Operating Systems to run simultaneously on the same infrastructure. Since the hypervisor is installed at a higher privilege level than the Operating Systems in the software stack it is vulnerable to rootkits that can modify the environment to gain control, crash the system and even steal sensitive information. Thus, runtime integrity measurement of the hypervisor is essential. The currently proposed solutions achieve the goal by relying either partially or entirely on the features of the hypervisor itself, causing them to lack stealth and leaving themselves vulnerable to attack. We have developed a performance sensitive methodology for identifying rootkits in hypervisors from System Management Mode (SMM) while using the features of SMI Transfer Monitor (STM). STM is a recent technology from Intel and it is a virtual machine manager at the firmware level. Our solution extends a research prototype called EPA-RIMM, developed by Delgado and Karavanic at Portland State University. Our solution extends the state of the art in that it stealthily performs measurements of hypervisor memory and critical data structures using firmware features, keeps performance perturbation to acceptable levels and leverages the security features provided by the STM. We describe our approach and include experimental results using a prototype we have developed for Xen hypervisor on Minnowboard Turbot, an open hardware platform.

M.Sc. / Anyone who uses a computer for work or recreational purposes has come across one or all of the following problems directly or indirectly (knowingly or not): viruses, worms, trojans, rootkits and botnets. This is especially the case if the computer is connected to the Internet. Looking at the statistics in [1] we can see that although malware detection techniques are detecting and preventing malware, they do not guarantee a 100% detection and or prevention of malware. Furthermore the statistics in [2] show that malware infection rates are increasing around the world at an alarming rate. The statistics also show that there are a high number of new malware samples being discovered every month and that 31% of malware attacks resulted in data loss [3], with 10% of companies reporting the loss of sensitive business data [4][5]. The reason for not being able to achieve a 100% detection and / or prevention of malware is because malware authors make use of sophisticated techniques such as code obfuscation in order to prevent malware from being detected. This has resulted in the emergence of malware known as polymorphic and metamorphic malware. The aforementioned malware poses serious challenges for anti-malware software specifically signature based techniques. However a more serious threat that needs to be addressed is that of rootkits. Rootkits can execute at the same privilege level as the Operating System (OS) itself. At this level the rootkit can manipulate the OS such that it can distribute other malware, hide existing malware, steal information, hide itself, disable anti-malware software etc all without the knowledge of the user. It is clear from the statistics that anti-malware products are not working because infection rates continue to rise and companies and end users continue to fall victims of these attacks. Therefore this dissertation will address the problem that current anti-malware techniques are not working. The main objective of this dissertation is to create a framework called ATE (Anti-malware Technique Evaluator) that can be used to critically evaluate current commercial anti-malware products. The framework will achieve this by identifying the current vulnerabilities that exist in commercial anti-malware products and the operating system. The prior will be achieved by making use of two rootkits, the Evader rootkit and the Sabotager rootkit, which were specifically developed to support the anti-malware product evaluation. Finally an anti-malware architecture we called External Malware Scanner (EMS), will be proposed to address the identified vulnerabilities.

The prevalence of Search Engine Poisoning in trending topics and popular search terms on the web within search engines is investigated. Search Engine Poisoning is the act of manipulating search engines in order to display search results from websites infected with malware. Research done between February and August 2012, using both manual and automated techniques, shows us how easily the criminal element manages to insert malicious content into web pages related to popular search terms within search engines. In order to provide the reader with a clear overview and understanding of the motives and the methods of the operators of Search Engine Poisoning campaigns, an in-depth review of automated and semi-automated web exploit kits is done, as well as looking into the motives for running these campaigns. Three high profile case studies are examined, and the various Search Engine Poisoning campaigns associated with these case studies are discussed in detail to the reader. From February to August 2012, data was collected from the top trending topics on Google’s search engine along with the top listed sites related to these topics, and then passed through various automated tools to discover if these results have been infiltrated by the operators of Search Engine Poisoning campaings, and the results of these automated scans are then discussed in detail. During the research period, manual searching for Search Engine Poisoning campaigns was also done, using high profile news events and popular search terms. These results are analysed in detail to determine the methods of attack, the purpose of the attack and the parties behind it