Detectar e analisar malwares é um processo essencial para aprimorar os sistemas de segurança. As soluções atuais apresentam limitações no processo de investigação e detecção de códigos maliciosos sofisticados. Mais do que utilizar técnicas para evadir sistemas de análise, malwares sofisticados requerem condições específicas no ambiente em que são executados para revelar seu comportamento malicioso. Com o surgimento das Redes Definidas por Software (SDN), notou-se uma oportunidade para aprimorar o processo de investigação de malware propondo uma arquitetura flexível apta a detectar variações comportamentais de maneira automática. Esta tese apresenta uma arquitetura especializada para analisar códigos maliciosos que permite controlar de maneira unificada o ambiente de análise, incluindo o sandbox e os elementos que o circundam. Dessa maneira, é possível gerenciar regras de contenção, configuração dinâmica de recursos, e manipular o tráfego de rede gerado pelos malwares. Para avaliar a arquitetura foi analisado um conjunto de malwares em dois cenários de avaliação. No primeiro cenário de avaliação, as funcionalidades descritas pela solução proposta revelaram novos eventos comportamentais em 100% dos malwares analisados. Já, no segundo cenários de avaliação, foi analisado um conjunto de malwares projetados para dispositivos IoT. Em consequência, foi possível bloquear ataques, monitorar a comunicação do malware com seu controlador de botnet, e manipular comandos de ataques. / Mechanisms to detect and analyze malicious software are essential to improve security systems. Current security mechanisms have limited success in detecting sophisticated malicious software. More than to evade analysis system, many malware require specific conditions to activate their actions in the target system. The flexibility of Software-Defined Networking (SDN) provides an opportunity to develop a malware analysis architecture that can detect behavioral deviations in an automated way. This thesis presents a specialized architecture to analyze malware by managing the analysis environment in a centralized way, including to control the sandbox and the elements that surrounds it. The proposed architecture enables to determine the network access policy, to handle the analysis environment resource configuration, and to manipulate the network connections performed by the malware. To evaluate our solution we have analyzed a set of malware in two evaluation scenarios. In the first evaluation scenario, we showed that the mechanisms proposed have increased the number of behavioral events in 100% of the malware analyzed. In the second evaluation scenario, we have analyzed malware designed for IoT devices. As a result, by using the MARS features, it was possible to block attacks, to manipulate attack commands, and to enable the malware communication with the respective botnet controller. The experimental results showed that our solution can improve the dynamic malware analysis process by providing this configuration flexibility to the analysis environment.

Detectar e analisar malwares é um processo essencial para aprimorar os sistemas de segurança. As soluções atuais apresentam limitações no processo de investigação e detecção de códigos maliciosos sofisticados. Mais do que utilizar técnicas para evadir sistemas de análise, malwares sofisticados requerem condições específicas no ambiente em que são executados para revelar seu comportamento malicioso. Com o surgimento das Redes Definidas por Software (SDN), notou-se uma oportunidade para aprimorar o processo de investigação de malware propondo uma arquitetura flexível apta a detectar variações comportamentais de maneira automática. Esta tese apresenta uma arquitetura especializada para analisar códigos maliciosos que permite controlar de maneira unificada o ambiente de análise, incluindo o sandbox e os elementos que o circundam. Dessa maneira, é possível gerenciar regras de contenção, configuração dinâmica de recursos, e manipular o tráfego de rede gerado pelos malwares. Para avaliar a arquitetura foi analisado um conjunto de malwares em dois cenários de avaliação. No primeiro cenário de avaliação, as funcionalidades descritas pela solução proposta revelaram novos eventos comportamentais em 100% dos malwares analisados. Já, no segundo cenários de avaliação, foi analisado um conjunto de malwares projetados para dispositivos IoT. Em consequência, foi possível bloquear ataques, monitorar a comunicação do malware com seu controlador de botnet, e manipular comandos de ataques. / Mechanisms to detect and analyze malicious software are essential to improve security systems. Current security mechanisms have limited success in detecting sophisticated malicious software. More than to evade analysis system, many malware require specific conditions to activate their actions in the target system. The flexibility of Software-Defined Networking (SDN) provides an opportunity to develop a malware analysis architecture that can detect behavioral deviations in an automated way. This thesis presents a specialized architecture to analyze malware by managing the analysis environment in a centralized way, including to control the sandbox and the elements that surrounds it. The proposed architecture enables to determine the network access policy, to handle the analysis environment resource configuration, and to manipulate the network connections performed by the malware. To evaluate our solution we have analyzed a set of malware in two evaluation scenarios. In the first evaluation scenario, we showed that the mechanisms proposed have increased the number of behavioral events in 100% of the malware analyzed. In the second evaluation scenario, we have analyzed malware designed for IoT devices. As a result, by using the MARS features, it was possible to block attacks, to manipulate attack commands, and to enable the malware communication with the respective botnet controller. The experimental results showed that our solution can improve the dynamic malware analysis process by providing this configuration flexibility to the analysis environment.

Malware analysis and detection continues to be one of the central battlefields for cybersecurity industry. For the desktop malware domain, we observed multiple significant ransomware attacks in the past several years, e.g., it was estimated that in 2017 the WannaCry ransomware attack affected more than 200,000 computers across 150 countries with hundreds of millions damages. Similarly, we witnessed the increased impacts of Android malware on global individuals due to the popular smartphone and IoT devices worldwide. In this dissertation, we describe similarity comparison based novel techniques that can be applied to achieve large scale desktop and Android malware analysis, and the practical implications of machine learning based approaches for malware detection. First, we propose a generic and effective solution for accurate and efficient binary similarity analysis of desktop malware. Binary similarity analysis is an essential technique for a variety of security analysis tasks, including malware detection and malware clustering. Even though various solutions have been developed, existing binary similarity analysis methods still suffer from limited efficiency, accuracy, and usability. In this work, we propose a novel graphical fuzzy hashing scheme for accurate and efficient binary similarity analysis. We first abstract control flow graphs (CFGs) of binary codes to extract blended n-gram graphical features of the CFGs, and then encode the graphical features into numeric vectors (called graph signatures) to measure similarity by comparing the graph signatures. We further leverage a fuzzy hashing technique to convert the numeric graph signatures into smaller fixed size fuzzy hash outputs for efficient comparisons. Our comprehensive evaluation demonstrates that our blended n-gram graphical feature based CFG comparison is more effective and efficient compared to existing CFG comparison techniques. Based on our CFG comparison method, we develop BingSim, a binary similarity analysis tool, and show that BingSim outperforms existing binary similarity analysis tools while conducting similarity analysis based malware detection and malware clustering. Second, we identify the challenges faced by overall similarity based Android malware clustering and design a specialized system for solving the problems. Clustering has been well studied for desktop malware analysis as an effective triage method. Conventional similarity-based clustering techniques, however, cannot be immediately applied to Android malware analysis due to the excessive use of third-party libraries in Android application development and the widespread use of repackaging in malware development. We design and implement an Android malware clustering system through iterative mining of malicious payloads and checking whether malware samples share the same version of malicious payloads. Our system utilizes a hierarchical clustering technique and an efficient bit-vector format to represent Android apps. Experimental results demonstrate that our clustering approach achieves precision of 0.90 and recall of 0.75 for the Android Genome mal- ware dataset, and average precision of 0.98 and recall of 0.96 with respect to manually verified ground-truth. Third, we study the fundamental issues faced by traditional machine learning (ML) based Android malware detection systems, and examine the role of ML for Android malware detection in practice, which leads to a revised evaluation strategy that evaluates an ML based malware detection system by checking their zero-day detection capabilities. Existing machine learning based Android malware research obtains the ground truth by consulting AV products, and uses the same label set for training and testing. However, there is a mismatch between how the ML system has been evaluated, and the true purpose of using ML system in practice. The goal of applying ML is not to reproduce or verify the same potentially imperfect knowledge, but rather to produce something that is better — closer to the ultimate ground truth about the apps’ maliciousness. Therefore, it will be more meaningful to check their zero-day detection capabilities than detection accuracy for known malware. This evaluation strategy is aligned with how an ML algorithm can potentially benefit malware detection in practice, by acknowledging that any ML classifier has to be trained on imperfect knowledge, and such knowledge evolves over time. Besides the traditional malware prediction approaches, we also examine the mislabel identification approaches. Through extensive experiments, we demonstrate that: (a) it is feasible to evaluate ML based Android malware detection systems with regard to their zero-day malware detection capabilities; (b) both malware prediction and mislabel identification approaches can be used to achieve verifiable zero-day malware detection, even when trained with an old and noisy ground truth dataset.

We present a rootkit prevention system, namely DARK that tracks suspicious Linux loadable kernel modules (LKM) at a granular level by using on-demand emulation, a technique that dynamically switches a running system between virtualized and emulated execution. Combining the strengths of emulation and virtualization, DARK is able to thoroughly capture the activities of the target module in a guest operating system (OS), while maintaining reasonable run-time performance. To address integrity-violation and confidentiality-violation rootkits, we create a group of security policies that can detect all available Linux rootkits. It is shown that normal guest OS performance is unaffected. The performance is only decreased when rootkits attempt to run, while most rootkits are detected at installation. Next, we present a sandbox-based malware analysis system called Rkprofiler that dynamically monitors and analyzes the behavior of Windows kernel malware. Kernel malware samples run inside a virtual machine (VM) that is supported and managed by a PC emulator. Rkprofiler provides several capabilities that other malware analysis systems do not have. First, it can detect the execution of malicious kernel code regardless of how the monitored kernel malware is loaded into the kernel and whether it is packed or not. Second, it captures all function calls made by the kernel malware and constructs call graphs from the trace files. Third, a technique called aggressive memory tagging (AMT) is proposed to track the dynamic data objects that the kernel malware visits. Last, Rkprofiler records and reports the hardware access events of kernel malware (e.g., MSR register reads and writes). Our evaluation results show that Rkprofiler can quickly expose the security-sensitive activities of kernel malware and thus reduces the effort exerted in conducting tedious manual malware analysis.

Today, host-based malware detection approaches such as antivirus programs are severely lagging in terms of defense against malware. Two important aspects that the overall effectiveness of malware detection depend on are the success of extracting information from malware using malware analysis to generate signatures, and then the success of utilizing these signatures on target hosts with appropriate system monitoring techniques. Today's malware employ a vast array of anti-analysis and anti-monitoring techniques to deter analysis and to neutralize antivirus programs, reducing the overall success of malware detection. In this dissertation, we present a set of practical approaches of robust and efficient malware analysis and system monitoring that can help make malware detection on hosts become more effective. First, we present a framework called Eureka, which efficiently deobfuscates single-pass and multi-pass packed binaries and restores obfuscated API calls, providing a basis for extracting comprehensive information from the malware using further static analysis. Second, we present the formal framework of transparent malware analysis and Ether, a dynamic malware analysis environment based on this framework that provides transparent fine-(single instruction) and coarse-(system call) granularity tracing. Third, we introduce an input-based obfuscation technique that hides trigger-based behavior from any input-oblivious analyzer. Fourth, we present an approach that automatically reverse-engineers the emulator and extracts the syntax and semantics of the bytecode language, which helps constructing control-flow graphs of the bytecode program and enables further analysis on the malicious code. Finally, we present Secure In-VM Monitoring, an approach of efficiently monitoring a target host while being robust against unknown malware that may attempt to neutralize security tools.

Metamorphism is a technique that mutates the binary code using different obfuscations. It is difficult to write a new metamorphic malware and in general malware writers reuse old malware. To hide detection the malware writers change the obfuscations (syntax) more than the behavior (semantic) of such a new malware. On this assumption and motivation, this thesis presents a new framework named MARD for Metamorphic Malware Analysis and Real-Time Detection. We also introduce a new intermediate language named MAIL (Malware Analysis Intermediate Language). Each MAIL statement is assigned a pattern that can be used to annotate a control flow graph for pattern matching to analyse and detect metamorphic malware. MARD uses MAIL to achieve platform independence, automation and optimizations for metamorphic malware analysis and detection. As part of the new framework, to build a behavioral signature and detect metamorphic malware in real-time, we propose two novel techniques, named ACFG (Annotated Control Flow Graph) and SWOD-CFWeight (Sliding Window of Difference and Control Flow Weight). Unlike other techniques, ACFG provides a faster matching of CFGs, without compromising detection accuracy; it can handle malware with smaller CFGs, and contains more information and hence provides more accuracy than a CFG. SWOD-CFWeight mitigates and addresses key issues in current techniques, related to the change of the frequencies of opcodes, such as the use of different compilers, compiler optimizations, operating systems and obfuscations. The size of SWOD can change, which gives anti-malware tool developers the ability to select appropriate parameter values to further optimize malware detection. CFWeight captures the control flow semantics of a program to an extent that helps detect metamorphic malware in real-time. Experimental evaluation of the two proposed techniques, using an existing dataset, achieved detection rates in the range 94% - 99.6% and false positive rates in the range 0.93% - 12.44%. Compared to ACFG, SWOD-CFWeight significantly improves the detection time, and is suitable to be used where the time for malware detection is more important as in real-time (practical) anti-malware applications. / Graduate / 0984 / alam_shahid@yahoo.com

Context: In today's society virus makers have a large set of obfuscation tools to avoid classic signature detection used by antivirus software. Therefore there is a need to identify new and obfuscated viruses in a better way. One option is to look at the behaviour of a program by executing the program in a virtual environment to determine if it is malicious or benign. This approach is called dynamic heuristic analysis. Objectives: In this study a new heuristic dynamic analysis tool for detecting unknown malware is proposed. The proposed implementation is evaluated against state-of-the-art in terms of accuracy. Methods: The proposed implementation uses Cuckoo sandbox to collect the behavior of a software and a decision tree to classify the software as either malicious or benign. In addition, the implementation contains several custom programs to handle the interaction between the components. Results: The experiment evaluating the implementation shows that an accuracy of 90% has been reached which is higher than 2 out of 3 state-of-the-art software. Conclusions: We conclude that an implementation using Cuckoo and decision tree works well for classifying malware and that the proposed implementation has a high accuracy that could be increased in the future by including more samples in the training set.

The first detected computer virus commenced in the 1970s. Since then, malware infections have grown exponentially along with rapid increases within the digital environment. Malware detection is a challenging task due to the relentless growth in complexity and volume. That is why the need for automated detection arises. Applying machine learning to malware detection is not a new trend, and researchers have been experimenting with since the 1990s. This thesis aims to evaluate classification algorithms to discover malicious Portable Executables by looking at their static features. Six machine learning models were built and tested based on 20,000 malicious and benign files. Random Forest scored the highest cross-validation score of 99.3% amongst the models with 15 features. Selecting the number of features was based on research of previous studies. This thesis confirms that it is possible to use machine learning for static malware detection. It can also help for future automated malware analysis research. / Det första datorviruset upptäcktes på 1970-talet. Sedan dess, har antalet attacker ökat i och med den skenande digitala utvecklingen. Att finna skadlig kod är en utmanade uppgift då de ökar i komplexitet och volym. Därför finns det ett behov att automatisera spårningen. Att använda maskininlärning för upptäckt av skadlig kod är inte en ny trend och forskare har experimenterat med det sedan år 1990. Syftet med denna avhandling är att utvärdera klassificeringsalgortimer för att upptäckta skadlig kod i Portable Executables genom att använda statiska prediktorer. Sex stycken maskininlärnings modeller skapades och testades baserat på 20.000 skadliga och legitima filer. Random Forest uppnådde det högsta korsvalderingsvärdet på 99.3% av dessa modeller med 15 prediktorer. Att använda 15 prediktorer var inspirerat av forskning av tidigare studier. Denna avhandling bevisar att det är möjligt att använda maskininlärning för statisk spårning av skadlig kod. Det kan också användas för framtida automatiserade forskningsstudier om skadlig kod.

Denna literaturöversikt går igenom två olika system inom IT-säkerhet och hur de fungerar tillsammans med maskinlärningstekniker till en relativt ytlig nivå.Syftet med denna rapport är att kunna sammanfatta dessa system och se hur de kan hjälpa med en systemadministratörs uppgifter, hur det kan användas för automatisera och vad för positiva och negativa förändringar det kan ha på en infrastruktur.Maskinlärning kan vara ett kraftigt verktyg för systemadministratörer för att lätta på arbetsmängden som kan förekomma inom en organisation, vilket är också varför det är viktigt att diskutera när och var man ska utplacera en lösning. Den här studien ska diskutera användningen av maskinlärning och när och var det kan användas. / This literature review discusses two different systems within IT-security and how they work within machine learning to a relatively surface-level degree.The purpose of this paper is to be able to summarize these systems and see how they can help a system administrator’s assignments. how it can be used for automation and the positives and negatives.Machine learning can be a powerful tool for system administrators to alleviate the workload which can exist within an organization, which is why it is important to discuss when and where to deploy a solution.

M. Tech. (Department of Information Technology, Faculty of Applied and Computer Sciences), Vaal University of Technology / Malware is today one of the biggest security threats to the Internet. Malware is any malicious software with the intent to perform malevolent activities on a targeted system. Viruses, worms, trojans, backdoors and adware are but a few examples that fall under the umbrella of malware. The purpose of this research is to investigate techniques that are used in order to effectively perform Malware analysis and detection on enterprise systems to reduce the damage of malware attacks on the operation of organizations. Malware analysis experiments were carried out using the two techniques of malware analysis, which are Dynamic and Static analysis, on two different malware samples. Portable executable and Microsoft word document files were the two samples that were analysed in an isolated sandbox lab environment. Static analysis is the process of examining and extracting information from malware code without executing the malware, while Dynamic analysis is the process of executing malware in order to observe and record its behaviour in a controlled environment. The results from the experiments disclosed the behaviour, encryption techniques, and other techniques employed by the malware samples. These malware analysis experiments were carried out in an isolated lab environment that was built for the purpose of this research. The results showed that Dynamic analysis is more effective than Static analysis. The study proposes the use of both techniques for comprehensive malware analysis and detection.