Malware is still one of the most prominent vectors through which computer networks and systems are compromised. A compromised computer system or network provides data and or processing resources to the world of cybercrime. With cybercrime projected to cost the world $6 trillion by 2021, malware is expected to continue being a growing challenge. Statistics around malware growth over the last decade support this theory as malware numbers enjoy almost an exponential increase over the period. Recent reports on the complexity of the malware show that the fight against malware as a means of building more resilient cyberspace is an evolving challenge. Compounding the problem is the lack of cyber security expertise to handle the expected rise in incidents. This thesis proposes advancing automation of the malware static analysis and detection to improve the decision-making confidence levels of a standard computer user in regards to a file’s malicious status. Therefore, this work introduces a framework that relies on two novel approaches to score the malicious intent of a file. The first approach attaches a probabilistic score to heuristic anomalies to calculate an overall file malicious score while the second approach uses fuzzy hashes and evidence combination theory for more efficient malware detection. The approaches’ resultant quantifiable scores measure the malicious intent of the file. The designed schemes were validated using a dataset of “clean” and “malicious” files. The results obtained show that the framework achieves true positive – false positive detection rate “trade-offs” for efficient malware detection.

Ce travail de thèse fait partie du projet 3D NeuroSecure. C'est un projet d'investissement d'avenir, qui vise à développer une solution de collaboration sécurisée pour l'innovation thérapeutique appliquant les traitements de haute performance (HPC) au monde biomédical. Cette solution donnera la possibilité pour les experts de différents domaines de naviguer intuitivement dans l'imagerie Big Data avec un accès via des terminaux mobile. La protection des données contre les fuites de données est primordiale. En tant que tel, l'environnement client et les communications avec le serveur doivent être sécurisé. Nous avons concentré notre travail sur le développement d'une solution antivirale sur le système d'exploitation Android. Nous avons promu la création de nouveaux algorithmes, méthodes et outils qui apportent des avantages par rapport à état de l'art, mais plus important encore, qui peuvent être utilisés efficacement dans un contexte de production. C'est pourquoi, ce qui est proposé ici est souvent un compromis entre ce qui peut théoriquement être fait et son applicabilité. Les choix algorithmiques et technologiques sont motivés par une relation entre efficacité et performance. Cette thèse contribue à l'état de l'art dans les domaines suivants:Analyse statique et dynamique d'applications Android, web crawling d'application.Tout d'abord, pour rechercher des fonctions malveillantes et des vulnérabilités, il faut concevoir les outils qui extraient des informations pertinentes des applications Android. C'est la base de toute analyse. En outre, tout algorithme de classification est toujours limité par la qualité discriminative des données sous-jacentes. Une partie importante de cette thèse est la la conception d'outils d'analyse statique et dynamique efficientes, telles qu'un module de reverse engineering, un outil d'analyse de communication, un système Android instrumenté.Algorithme d'initialisation, d'apprentissage et d'anti-saturation pour réseau de neurones.Les réseaux de neurones sont initialisés au hasard. Il est possible de contrôler la distribution aléatoire sous-jacente afin de réduire l'effet de saturation, le temps de l'entrainement et la capacité à atteindre le minimum global. Nous avons développé une procédure d’initialisation qui améliore les résultats par rapport à l'état del'art. Nous avons aussi adapté l'algorithme ADAM pour prendre en compte les interdépendances avec des techniques de régularisation, en particulier le Dropout. Enfin, nous utilisons techniques d'anti-saturation et nous montrons qu'elles sont nécessaires pour entraîner correctement un réseau neuronal.Un algorithme pour représenter les sous-séquences communes à un groupe de séquences.Nous proposons un nouvel algorithme pour construire l'AntichaineEnglobante des sous-séquences communes. Il est capable de traiter et de représenter toutes les sous-séquences d'un ensemble de séquences. C'estun outil qui permet de caractériser de manière systématique un groupe de séquence. Cet algorithme est une nouvelle voie de recherche verscréation automatique de règles de détection de famille de virus. / This thesis work is part of the 3D NeuroSecure project. It is an investment project, that aims to develop a secure collaborative solution for therapeutic innovation using high performance processing(HPC) technology to the biomedical world. This solution will give the opportunity for experts from different fields to navigate intuitivelyin the Big Data imaging with access via 3D light terminals. Biomedicaldata protection against data leaks is of foremost importance. As such,the client environnement and communications with the server must besecured. We focused our work on the development of antimalware solutionon the Android OS. We emphasizes the creation of new algorithms,methods and tools that carry advantages over the current state-of-the-art, but more importantly that can be used effectively ina production context. It is why, what is proposed here is often acompromise between what theoretically can be done and its applicability. Algorithmic and technological choices are motivated by arelation of efficiency and performance results. This thesis contributes to the state of the art in the following areas:Static and dynamic analysis of Android applications, application web crawling.First, to search for malicious activities and vulnerabilities, oneneeds to design the tools that extract pertinent information from Android applications. It is the basis of any analysis. Furthermore,any classifier or detector is always limited by the informative power of underlying data. An important part of this thesis is the designing of efficient static and dynamic analysis tools forapplications, such as an reverse engineering module, a networkcommunication analysis tool, an instrumented Android system, an application web crawlers etc.Neural Network initialization, training and anti-saturation techniques algorithm.Neural Networks are randomly initialized. It is possible to control the underlying random distribution in order to the reduce the saturation effect, the training time and the capacity to reach theglobal minimum. We developed an initialization procedure that enhances the results compared to the state-of-the-art. We also revisited ADAM algorithm to take into account interdependencies with regularization techniques, in particular Dropout. Last, we use anti-saturation techniques and we show that they are required tocorrectly train a neural network.An algorithm for collecting the common sequences in a sequence group.We propose a new algorithm for building the Embedding Antichain fromthe set of common subsequences. It is able to process and represent allcommon subsequences of a sequence set. It is a tool for solving the Systematic Characterization of Sequence Groups. This algorithm is a newpath of research toward the automatic creation of malware familydetection rules.

Malware is still one of the most prominent vectors through which computer networks and systems are compromised. A compromised computer system or network provides data and or processing resources to the world of cybercrime. With cybercrime projected to cost the world $6 trillion by 2021, malware is expected to continue being a growing challenge. Statistics around malware growth over the last decade support this theory as malware numbers enjoy almost an exponential increase over the period. Recent reports on the complexity of the malware show that the fight against malware as a means of building more resilient cyberspace is an evolving challenge. Compounding the problem is the lack of cyber security expertise to handle the expected rise in incidents. This thesis proposes advancing automation of the malware static analysis and detection to improve the decision-making confidence levels of a standard computer user in regards to a file’s malicious status. Therefore, this work introduces a framework that relies on two novel approaches to score the malicious intent of a file. The first approach attaches a probabilistic score to heuristic anomalies to calculate an overall file malicious score while the second approach uses fuzzy hashes and evidence combination theory for more efficient malware detection. The approaches’ resultant quantifiable scores measure the malicious intent of the file. The designed schemes were validated using a dataset of “clean” and “malicious” files. The results obtained show that the framework achieves true positive – false positive detection rate “trade-offs” for efficient malware detection.

Dynamic spectrum access (DSA) technique has been widely accepted as a crucial solution to mitigate the potential spectrum scarcity problem. Spectrum sharing between the government incumbents and commercial wireless broadband operators/users is one of the key forms of DSA. Two categories of spectrum management methods for shared use between incumbent users (IUs) and secondary users (SUs) have been proposed, i.e., the server-driven method and the sensing-based method. The server-driven method employs a central server to allocate spectrum resources while considering incumbent protection. The central server has access to the detailed IU operating information, and based on some accurate radio propagation model, it is able to allocate spectrum following a particular access enforcement method. Two types of access enforcement methods -- exclusion zone and protection zone -- have been adopted for server-driven DSA systems in the current literature. The sensing-based method is based on recent advances in cognitive radio (CR) technology. A CR can dynamically identify white spaces through various incumbent detection techniques and reconfigure its radio parameters in response to changes of spectrum availability. The focus of this dissertation is to address critical privacy and security issues in the existing DSA systems that may severely hinder the progress of DSA's deployment in the real world. Firstly, we identify serious threats to users' privacy in existing server-driven DSA designs and propose a privacy-preserving design named P²-SAS to address the issue. P²-SAS realizes the complex spectrum allocation process of protection-zone-based DSA in a privacy-preserving way through Homomorphic Encryption (HE), so that none of the IU or SU operation data would be exposed to any snooping party, including the central server itself. Secondly, we develop a privacy-preserving design named IP-SAS for the exclusion-zone- based server-driven DSA system. We extend the basic design that only considers semi- honest adversaries to include malicious adversaries in order to defend the more practical and complex attack scenarios that can happen in the real world. Thirdly, we redesign our privacy-preserving SAS systems entirely to remove the somewhat- trusted third party (TTP) named Key Distributor, which in essence provides a weak proxy re-encryption online service in P²-SAS and IP-SAS. Instead, in this new system, RE-SAS, we leverage a new crypto system that supports both a strong proxy re-encryption notion and MPC to realize privacy-preserving spectrum allocation. The advantages of RE-SAS are that it can prevent single point of vulnerability due to TTP and also increase SAS's service performance dramatically. Finally, we identify the potentially crucial threat of compromised CR devices to the ambient wireless infrastructures and propose a scalable and accurate zero-day malware detection system called GuardCR to enhance CR network security at the device level. GuardCR leverages a host-based anomaly detection technique driven by machine learning, which makes it autonomous in malicious behavior recognition. We boost the performance of GuardCR in terms of accuracy and efficiency by integrating proper domain knowledge of CR software. / Ph. D. / With the rapid development of wireless technologies in recent years, wireless spectrum which all the wireless communication signals travel over is becoming the bottleneck of the fast growing wireless market. The spectrum scarcity problem is largely due to the current spectrum allocation scheme. Some spectrum bands, like the cellular bands, are overly crowded, while some government-held spectrum bands are used inadequately. By allowing users from the crowded spectrum bands to dynamically access to those less frequently used spectrum bands, the spectrum scarcity problem can be significantly alleviated. However, there are two critical issues that hinder the application of dynamic spectrum access in the real world: privacy and security. For privacy, in order to determine when, where, and how the spectrum can be reused, users need to bear the risk of sharing their sensitive operation data. This is especially frustrating for governmental and military parties whose operation data is highly classified. We solve the privacy problem by designing a privacy-preserving dynamic spectrum access system. The system is based on secure multi-party computation, which keeps users’ input operation data private when performing spectrum allocation computation over those inputs. The system achieves 128-bit industry-level security strength, and it is also computation and memory efficient for real-world deployment. For security, dynamic spectrum access requires radio devices to contain many software components so that the radio devices can be dynamically programmed to access different spectrum bands. However, the software also exposes the radio devices to the risk of malware infection. We develop a malware detection system to capture the anomalous behaviors in radio software executions. By adopting advanced machine learning techniques, our system is even able to detect first-seen malware.

No description available.

Most of the attacks and fraudulent activities on the Internet are carried out by malware. In particular, botnets, as state-of-the-art malware, are now considered as the largest threat to Internet security. In this thesis, we focus on addressing the botnet detection problem in an enterprise-like network environment. We present a comprehensive correlation-based framework for multi-perspective botnet detection consisting of detection technologies demonstrated in four complementary systems: BotHunter, BotSniffer, BotMiner, and BotProbe. The common thread of these systems is correlation analysis, i.e., vertical correlation (dialog correlation), horizontal correlation, and cause-effect correlation. All these Bot* systems have been evaluated in live networks and/or real-world network traces. The evaluation results show that they can accurately detect real-world botnets for their desired detection purposes with a very low false positive rate. We find that correlation analysis techniques are of particular value for detecting advanced malware such as botnets. Dialog correlation can be effective as long as malware infections need multiple stages. Horizontal correlation can be effective as long as malware tends to be distributed and coordinated. In addition, active techniques can greatly complement passive approaches, if carefully used. We believe our experience and lessons are of great benefit to future malware detection.

The continuity in the financial and other related losses due to cyber-attacks prove the substantial growth of malware and their lethal proliferation techniques. Every successful malware attack highlights the weaknesses in the defence mechanisms responsible for securing the targeted computer or a network. The recent cyber-attacks reveal the presence of sophistication and intelligence in malware behaviour having the ability to conceal their code and operate within the system autonomously. The conventional detection mechanisms not only possess the scarcity in malware detection capabilities, they consume a large amount of resources while scanning for malicious entities in the system. Many recent reports have highlighted this issue along with the challenges faced by the alternate solutions and studies conducted in the same area. There is an unprecedented need of a resilient and autonomous solution that takes proactive approach against modern malware with stealth behaviour. This thesis proposes a multi-aspect solution comprising of an intelligent malware detection framework and an energy efficient hosting model. The malware detection framework is a combination of conventional and novel malware detection techniques. The proposed framework incorporates comprehensive feature heuristics of files generated by a bespoke static feature extraction tool. These comprehensive heuristics are used to train the machine learning algorithms; Support Vector Machine, Decision Tree, and Boosting to differentiate between clean and malicious files. Both these techniques; feature heuristics and machine learning are combined to form a two-factor detection mechanism. This thesis also presents a cloud-based energy efficient and scalable hosting model, which combines multiple infrastructure components of Amazon Web Services to host the malware detection framework. This hosting model presents a client-server architecture, where client is a lightweight service running on the host machine and server is based on the cloud. The proposed framework and the hosting model were evaluated individually and combined by specifically designed experiments using separate repositories of clean and malicious files. The experiments were designed to evaluate the malware detection capabilities and energy efficiency while operating within a system. The proposed malware detection framework and the hosting model showed significant improvement in malware detection while consuming quite low CPU resources during the operation.

基於屬性相似度將樣本進行分群的技術已經被廣泛應用在許多領域，如模式識別，特徵提取和惡意行為偵測。由於此技術的重要性，很多人已經將各種分群技術利用分散式框架進行再製，例如K-means搭配Hadoop在Apache Mahout平台上。由於K-means需要預先定義分群數量，而自組織映射圖（SOM）需要預先定義圖的大小，所以能夠自動將樣本依照樣本間的變化容差進行分群的GHSOM（增長層次自組織映射圖）就提供了一個很棒的非監督學習方法用來針對某些資訊不完整的資料。然而，GHSOM目前並不是一個分散式的演算法，這就限制了其在大數據資料的應用上。在本篇論文中，我們提出了一種新的分散式GHSOM演算法。我們使用Scala的Actor Model來實現GHSOM的分散式系統，我們將GHSOM演算法中的水平擴增以及垂直擴增交由Actor來處理並顯示出顯著的性能提升。為了評估我們所提出的方法，我們收集並分析了數千個惡意程式在現實生活中的執行行為，並通過在數百萬個樣本上進行非監督分群後推導出惡意程式行為的檢測規則來顯示其性能的改進、規則有效性以及實踐中的潛在用法。 / Clustering techniques that group samples based on their attribute similarity have been widely used in many fields such as pattern recognition, feature extraction and malicious behavior characterization. Due to its importance, various clustering techniques have been developed with distributed frameworks such as K-means with Hadoop in Apache Mahout for scalable computation. While K-means requires the number of clusters and self organizing maps (SOM) requires the map size to be given, the technique of GHSOM (growing hierarchical self organizing maps) that clusters samples dynamically to satisfy the requirement on tolerance of variation between samples, poses an attractive unsupervised learning solution for data that have limited information to decide the number of clusters in advance. However it is not scalable with sequential computation, which limits its applications on big data. In this paper, we present a novel distributed algorithm on GHSOM. We take advantage of parallel computation with scala actor model for GHSOM construction, distributing vertical and horizontal expansion tasks to actors and showing significant performance improvement. To evaluate the presented approach, we collect and analyze execution behaviors of thousands of malware in real life and derive detection rules with the presented unsupervised clustering on millions samples, showing its performance improvement, rule effectiveness and potential usage in practice.

Les flux d’information sont des transferts d’information entre les objets d’un environnement donné. À l’échelle du système, pour toute information appartenant à une application donnée, les flux impliquant cette information décrivent comment l’application propage ses données dans le système et l’ensemble de ces flux peut ainsi être considéré comme un profil comportemental de l’application. À cause du nombre croissant d’applications malveillantes, il est devenu nécessaire d’explorer des nouvelles techniques permettant de faciliter voir automatiser l’analyse et la détection de malware. Dans cette thèse, nous proposons ainsi une méthode pour caractériser et détecter les malware Android en nous basant sur les flux d’information qu’ils causent dans le système. Cette méthode repose sur deux autres contributions de la thèse : AndroBlare, la version Android d’un moniteur de flux d’information du nom de Blare, et les graphes de flux système, une structure de donnée représentant de manière compacte et humainement compréhensible les flux d’information observés. Nous avons évalué avec succès notre approche en construisant le profil de 4 malware différents et avons montré que ces profils permettaient de détecter l’exécution d’applications infectées par les malware dont on a un profil. / : Information flows are information exchanges between objects in a given environment. At system level, information flows involving data belonging to a given application describe how this application disseminates its data in the system and can be considered as behaviour based profile of the application. Because of the increasing number of Android malware, there is an urgent need to explore new approaches to analyse and detect Android malware. In this thesis, we thus propose an approach to characterize and detect Android malware based on information flows they cause in the system. This approach leverages two other contributions of the thesis which are AndroBlare, the Android version of an information flow monitor named Blare, and the system flow graph, a data structure to represent in a compact and human readable way the information flows observed by AndroBlare. We successfully evaluated our approach by building the profile of 4 different malware and showed that these profiles permitted to detect the execution of applications infected by malware for which we have computed a profile.

This master thesis deals with the design and implementation of a tool for malware detection using DNS traffic analysis. Text of the thesis is divided into theoretical and practical part. In theoretical part the reader will be acknowledged with the domain of malware and botnet detection. Consequently, various options and methods of malware detection will be described. Practical part of the thesis contains description of malware detection tool architecture as well as key aspects of its implementation. Moreover, the emphasis is being placed on testing and experiments. The result of the thesis is a tool, written in python, for malware detection using DNS traffic analysis, that uses a combination of several methods of detection.