Current signature-based Anti-Virus (AV) detection approaches take, on average, two weeks from discovery to definition update release to AV users. In addition, these signatures get stale quickly: AV products miss between 25%-80% of new malicious software within a week of not updating. This thesis researches and develops a detection/classification mechanism for malicious software through statistical analysis of dynamic malware behavior. Several characteristics for each behavior type were stored and analyzed such as function DLL names, function parameters, exception thread ids, exception opcodes, pages accessed during faults, port numbers, connection types, and IP addresses. Behavioral data was collected via Norman Sandbox for storage and analysis. We proposed to find which statistical measures and metrics can be collected for use in the detection and classification of malware. We conclude that our logging and cataloging procedure is a potentially viable method in creating behavior-based malicious software detection and classification mechanisms.

Traditional way to detect malicious software is based on signature matching. However, signature matching only detects known malicious software. In order to detect unknown malicious software, it is necessary to analyze the software for its impact on the system when the software is executed. In one approach, the software code can be statically analyzed for any malicious patterns. Another approach is to execute the program and determine the nature of the program dynamically. Since the execution of malicious code may have negative impact on the system, the code must be executed in a controlled environment. For that purpose, we have developed a sandbox to protect the system. Potential malicious behavior is intercepted by hooking Win32 system calls. Using the developed sandbox, we detect unknown virus using dynamic instruction sequences mining techniques. By collecting runtime instruction sequences in basic blocks, we extract instruction sequence patterns based on instruction associations. We build classification models with these patterns. By applying this classification model, we predict the nature of an unknown program. We compare our approach with several other approaches such as simple heuristics, NGram and static instruction sequences. We have also developed a method to identify a family of malicious software utilizing the system call trace. We construct a structural system call diagram from captured dynamic system call traces. We generate smart system call signature using profile hidden Markov model (PHMM) based on modularized system call block. Smart system call signature weakly identifies a family of malicious software.

L'objectif de cette thèse est le développement de méthodes de compréhension des logiciels malveillants, afin d'aider l'analyste humain à mieux appréhender cette menace. La première réalisation de cette thèse est une analyse à grande échelle et en profondeur des protections de logiciels malveillants. Plus précisément, nous avons étudié des centaines d'exemplaires de logiciels malveillants, soigneusement sélectionnés pour leur dangerosité. En mesurant de façon automatique un ensemble de caractéristiques originales, nous avons pu alors montrer l'existence d'un modèle de protection particulièrement prévalent dans ces programmes, qui est basé sur l'auto modification du code et sur une limite stricte entre code de protection et code utile. Ensuite, nous avons développé une méthode d'identification d'implémentations cryptographiques adaptée aux programmes en langage machine protégés. Nous avons validé notre approche en identifiant de nombreuses implémentations d'algorithmes cryptographiques -- dont la majorité sont complètement invisibles pour les outils existants --, et ceci en particulier dans des protections singulièrement obscures de logiciels malveillants. Finalement, nous avons développé ce qui est, à notre connaissance, le premier environnement d'émulation de réseaux de machines infectées avec plusieurs milliers de machines. Grâce à cela, nous avons montré que l'exploitation d'une vulnérabilité du protocole pair-à-pair du réseau Waledac permet de prendre son contrôle / The main goal of this thesis is the development of malware analysis methods to help human analysts better comprehend the threat it represents. The first achievement in this thesis is the large-scale and in-depth analysis of malware protection techniques. In particular, we have studied hundreds of malware samples, carefully selected according to their threat level. By automatically measuring a set of original characteristics, we have been able to demonstrate the existence of a particularly prevalent model of protection in these programmes that is based on self-modifying code and on a strict delimitation between protection code and payload code. Then, we have developed an identification method for cryptographic implementations adapted to protected machine language programmes. We have validated our approach by identifying several implementations of cryptographic algorithms ---the majority unidentified by existing tools--- and this even in particularly obscure malware protection schemes. Finally, we have developed what is, to our knowledge, the first emulation environment for botnets involving several thousands of machines. Thanks to this, we were able to validate the viability of the use of a vulnerability in the peer-to-peer protocol in the Waledac botnet to take over this network

This thesis deals with the generic detection of bootkits which are relatively a new kind of malicious sofware falling into the category of rootkits. The definition of malicious software is presented along with several examples. Then the attention is paid to the rootkits in the context of Microsoft Windows operating systems. This section lists several techniques used by rootkits. After that, the ways of preventing and detecting rootkits are mentioned. Bootkits are known for infecting hard disks Master Boot Record (MBR). The structure of the MBR is described along with the example of hard disk partitioning. Afterwards, the processor instruction set is outlined and the disassembly of Windows 7 MBR is given. The rest of the thesis is devoted to a description of the course of operating system bootkit infection, bootkit prevention, analysis of infected MBR samples, and in particular to the design, implementation and testing of the generic MBR infection detector.

In the last decade, a lot of machine learning and data mining based approaches have been used in the areas of intrusion detection, malware detection and classification and also traffic analysis. In the area of malware analysis, static binary analysis techniques have become increasingly difficult with the code obfuscation methods and code packing employed when writing the malware. The behavior-based analysis techniques are being used in large malware analysis systems because of this reason. In prior art, a number of clustering and classification techniques have been used to classify the malwares into families and to also identify new malware families, from the behavior reports. In this thesis, we have analysed in detail about the use of Profile Hidden Markov models for the problem of malware classification and clustering. The advantage of building accurate models with limited examples is very helpful in early detection and modeling of malware families. The thesis also revisits the learning setting of an Intrusion Detection System that employs machine learning for identifying attacks and normal traffic. It substantiates the suitability of incremental learning setting(or stream based learning setting) for the problem of learning attack patterns in IDS, when large volume of data arrive in a stream. Related to the above problem, an elaborate survey of the IDS that use data mining and machine learning was done. Experimental evaluation and comparison show that in terms of speed and accuracy, the stream based algorithms perform very well as large volumes of data are presented for classification as attack or non-attack patterns. The possibilities for using stream algorithms in different problems in security is elucidated in conclusion.

The goal of this thesis is to create a virtual machine that emulates a running programs written in Java programing language, which would be suitable for malware analysis and detection. The emulator is able to detect arguments of exploitable methods from Java standard classes, the order of calling these exploitable methods and also execution the test application. Overall functionality was tested on appropriate examples in which held its own measurements. At the end of the paper we describe testing of the emulator, which also contains tables and graphs for better results visualization.