<p>The system call based research for host intrusion detection systems (HIDSs) and Android malware detection systems (AMDSs) have been conducted over the past several years. Several HIDS and AMDS frameworks have been proposed using different intrusion and malware datasets. Security researchers have used several machine learning (ML) techniques to improve the classification performance with high accuracy and low false-alarm rate. However, the emphasis on real-world deployment of HIDS and AMDS for intrusion and malware detection is limited. To address this issue, we propose a system call traces processing framework with the ability to perform early detection of intrusions and malware. In the proposed framework, a limited number of system calls are analyzed which are invoked by the processes/applications during their early execution. To verify the efficiency, we perform the experiments on a publicly available intrusion dataset known as ADFA-LD dataset and a self-constructed dataset for Android environment. We analyze both the datasets with statistical methods, and process the selected traces with 2-4 gram model and Term Frequency–Inverse Document Frequency (TF-IDF) model during the extraction of features. We train six ML classifiers using the datasets including Decision Tree, Random Forest, Multi-layer Perceptron, K-nearest-neighbor, Multi-variable Naive Bayesian, and Support Vector Machine. The experimental results demonstrate that the performance of proposed HIDS and AMDS are similar to the approaches that used all the system calls invoked during the full execution of applications. We also develop a client-server architecture based Android app for our Android malware detection system.</p>

Android malware collusion is a new threat model that occurs when multiple Android apps communicate in order to execute an attack. This threat model threatens all Android users' private information and system resource security. Although recent research has made advances in collusion detection and classification, security analysts still do not have robust tools which allow them to definitively identify colluding Android applications. Specifically, in order to determine whether an alert produced by a tool scanning for Android collusion is a true-positive or a false-positive, the analyst must perform manual analysis of the suspected apps, which is both time consuming and prone to human errors. In this thesis, we present a new approach to definitive Android collusion detection and confirmation by rendering inter-component communications between a set of potentially collusive Android applications. Inter-component communications (abbreviated to ICCs), are a feature of the Android framework that allows components from different applications to communicate with one another. Our approach allows Android security analysts to inspect all ICCs within a set of suspicious Android applications and subsequently identify collusive attacks which utilize ICCs. Furthermore, our approach also visualizes all potentially collusive data-flows within each component within a set of apps. This allows analysts to inspect, step-by-step, the the data-flows that are currently used by collusive attacks, or the data-flows that could be used for future collusive attacks. Our tool effectively visualizes the malicious and benign ICCs in sets of proof-of-concept and real-world colluding applications. We conducted a user study which revealed that our approach allows for accurate and efficient identification of true- and false-positive collusive ICCs while still maintaining usability. / Master of Science

No description available.

No description available.

No description available.

Les flux d’information sont des transferts d’information entre les objets d’un environnement donné. À l’échelle du système, pour toute information appartenant à une application donnée, les flux impliquant cette information décrivent comment l’application propage ses données dans le système et l’ensemble de ces flux peut ainsi être considéré comme un profil comportemental de l’application. À cause du nombre croissant d’applications malveillantes, il est devenu nécessaire d’explorer des nouvelles techniques permettant de faciliter voir automatiser l’analyse et la détection de malware. Dans cette thèse, nous proposons ainsi une méthode pour caractériser et détecter les malware Android en nous basant sur les flux d’information qu’ils causent dans le système. Cette méthode repose sur deux autres contributions de la thèse : AndroBlare, la version Android d’un moniteur de flux d’information du nom de Blare, et les graphes de flux système, une structure de donnée représentant de manière compacte et humainement compréhensible les flux d’information observés. Nous avons évalué avec succès notre approche en construisant le profil de 4 malware différents et avons montré que ces profils permettaient de détecter l’exécution d’applications infectées par les malware dont on a un profil. / : Information flows are information exchanges between objects in a given environment. At system level, information flows involving data belonging to a given application describe how this application disseminates its data in the system and can be considered as behaviour based profile of the application. Because of the increasing number of Android malware, there is an urgent need to explore new approaches to analyse and detect Android malware. In this thesis, we thus propose an approach to characterize and detect Android malware based on information flows they cause in the system. This approach leverages two other contributions of the thesis which are AndroBlare, the Android version of an information flow monitor named Blare, and the system flow graph, a data structure to represent in a compact and human readable way the information flows observed by AndroBlare. We successfully evaluated our approach by building the profile of 4 different malware and showed that these profiles permitted to detect the execution of applications infected by malware for which we have computed a profile.

The rapid increase in mobile malware and deployment of over-privileged applications over the years has been of great concern to the security community. Encroaching on user’s privacy, mobile applications (apps) increasingly exploit various sensitive data on mobile devices. The information gathered by these applications is sufficient to uniquely and accurately profile users and can cause tremendous personal and financial damage. On Android specifically, the security and privacy holes in the operating system and framework code has created a whole new dynamic for malware and privacy exploitation. This research work seeks to develop novel analysis techniques that monitor Android applications for possible unwanted behaviors and then suggest various ways to deal with the privacy leaks associated with them. Current state-of-the-art static malware analysis techniques on Android-focused mainly on detecting known variants without factoring any kind of software obfuscation. The dynamic analysis systems, on the other hand, are heavily dependent on extending the Android OS and/or runtime virtual machine. These methodologies often tied the system to a single Android version and/or kernel making it very difficult to port to a new device. In privacy, accesses to the database system’s objects are not controlled by any security check beyond overly-broad read/write permissions. This flawed model exposes the database contents to abuse by privacy-agnostic apps and malware. This research addresses the problems above in three ways. First, we developed a novel static analysis technique that fingerprints known malware based on three-level similarity matching. It scores similarity as a function of normalized opcode sequences found in sensitive functional modules and application permission requests. Our system has an improved detection ratio over current research tools and top COTS anti-virus products while maintaining a high level of resiliency to both simple and complex obfuscation. Next, we augment the signature-related weaknesses of our static classifier with a hybrid analysis system which incorporates bytecode instrumentation and dynamic runtime monitoring to examine unknown malware samples. Using the concept of Aspect-oriented programming, this technique involves recompiling security checking code into an unknown binary for data flow analysis, resource abuse tracing, and analytics of other suspicious behaviors. Our system logs all the intercepted activities dynamically at runtime without the need for building custom kernels. Finally, we designed a user-level privacy policy enforcement system that gives users more control over their personal data saved in the SQLite database. Using bytecode weaving for query re-writing and enforcing access control, our system forces new policies at the schema, column, and entity levels of databases without rooting or voiding device warranty.

The number of mobile applications (i.e., apps) is rapidly growing, as the mobile computing becomes an integral part of the modern user experience. Malicious apps have infiltrated open marketplaces for mobile platforms. These malicious apps can exfiltrate user's private data, abuse of system resources, or disrupting regular services. Despite the recent advances on mobile security, the problem of detecting vulnerable and malicious mobile apps with high detection accuracy remains an open problem. In this thesis, we address the problem of Android security by presenting a new quantitative program analysis framework for security vetting of Android apps. We first introduce a highly accurate proactive detection solution for detecting individual malicious apps. Our approach enforces benign property as opposed of chasing malware signatures, and uses one complex feature rather than multi-feature as in the existing malware detection methods. In particular, we statically extract a data-flow feature on how user inputs trigger sensitive critical operations, a property referred to as the user-trigger dependence. This feature is extracted through nontrivial Android-specific static program analysis, which can be used in various quantitative analytical methods. Our evaluation on thousands of malicious apps and free popular apps gives a detection accuracy (2% false negative rate and false positive rate) that is better than, or at least competitive against, the state-of-the-art. Furthermore, our method discovers new malicious apps available in the Google Play store that have not been previously detected by anti-virus scanning tools. Second, we present a new app collusion detection approach and algorithms to analyze pairs or groups of communicating apps. App collusion is a new technique utilized by the attackers to evade standard detection. It is a new threat where two or more apps, appearing benign, communicate to perform malicious task. Most of the existing solutions assume the attack model of a stand-alone malicious app, and hence cannot detect app collusion. We first demonstrate experimental evidence on the technical challenges associated with detecting app collusion. Then, we address these challenges by introducing a scalable and an in-depth cross-app static flow analysis approach to identify the risk level associated with communicating apps. Our approach statically analyzes the sensitivity and the context of each inter-app communication with low analysis complexity, and defines fine-grained security policies for the inter-app communication risk detection. Our evaluation results on thousands of free popular apps indicate that our technique is effective. It generates four times fewer false positives compared to the state-of-the-art collusion-detection solution, enhancing the detection capability. The advantages of our inter-app communication analysis approach are the analysis scalability with low complexity, and the substantially improved detection accuracy compared to the state-of-the-art solution. These types of proactive defenses solutions allow defenders to stay proactive when defending against constantly evolving malware threats. / Ph. D.

Ce travail de thèse fait partie du projet 3D NeuroSecure. C'est un projet d'investissement d'avenir, qui vise à développer une solution de collaboration sécurisée pour l'innovation thérapeutique appliquant les traitements de haute performance (HPC) au monde biomédical. Cette solution donnera la possibilité pour les experts de différents domaines de naviguer intuitivement dans l'imagerie Big Data avec un accès via des terminaux mobile. La protection des données contre les fuites de données est primordiale. En tant que tel, l'environnement client et les communications avec le serveur doivent être sécurisé. Nous avons concentré notre travail sur le développement d'une solution antivirale sur le système d'exploitation Android. Nous avons promu la création de nouveaux algorithmes, méthodes et outils qui apportent des avantages par rapport à état de l'art, mais plus important encore, qui peuvent être utilisés efficacement dans un contexte de production. C'est pourquoi, ce qui est proposé ici est souvent un compromis entre ce qui peut théoriquement être fait et son applicabilité. Les choix algorithmiques et technologiques sont motivés par une relation entre efficacité et performance. Cette thèse contribue à l'état de l'art dans les domaines suivants:Analyse statique et dynamique d'applications Android, web crawling d'application.Tout d'abord, pour rechercher des fonctions malveillantes et des vulnérabilités, il faut concevoir les outils qui extraient des informations pertinentes des applications Android. C'est la base de toute analyse. En outre, tout algorithme de classification est toujours limité par la qualité discriminative des données sous-jacentes. Une partie importante de cette thèse est la la conception d'outils d'analyse statique et dynamique efficientes, telles qu'un module de reverse engineering, un outil d'analyse de communication, un système Android instrumenté.Algorithme d'initialisation, d'apprentissage et d'anti-saturation pour réseau de neurones.Les réseaux de neurones sont initialisés au hasard. Il est possible de contrôler la distribution aléatoire sous-jacente afin de réduire l'effet de saturation, le temps de l'entrainement et la capacité à atteindre le minimum global. Nous avons développé une procédure d’initialisation qui améliore les résultats par rapport à l'état del'art. Nous avons aussi adapté l'algorithme ADAM pour prendre en compte les interdépendances avec des techniques de régularisation, en particulier le Dropout. Enfin, nous utilisons techniques d'anti-saturation et nous montrons qu'elles sont nécessaires pour entraîner correctement un réseau neuronal.Un algorithme pour représenter les sous-séquences communes à un groupe de séquences.Nous proposons un nouvel algorithme pour construire l'AntichaineEnglobante des sous-séquences communes. Il est capable de traiter et de représenter toutes les sous-séquences d'un ensemble de séquences. C'estun outil qui permet de caractériser de manière systématique un groupe de séquence. Cet algorithme est une nouvelle voie de recherche verscréation automatique de règles de détection de famille de virus. / This thesis work is part of the 3D NeuroSecure project. It is an investment project, that aims to develop a secure collaborative solution for therapeutic innovation using high performance processing(HPC) technology to the biomedical world. This solution will give the opportunity for experts from different fields to navigate intuitivelyin the Big Data imaging with access via 3D light terminals. Biomedicaldata protection against data leaks is of foremost importance. As such,the client environnement and communications with the server must besecured. We focused our work on the development of antimalware solutionon the Android OS. We emphasizes the creation of new algorithms,methods and tools that carry advantages over the current state-of-the-art, but more importantly that can be used effectively ina production context. It is why, what is proposed here is often acompromise between what theoretically can be done and its applicability. Algorithmic and technological choices are motivated by arelation of efficiency and performance results. This thesis contributes to the state of the art in the following areas:Static and dynamic analysis of Android applications, application web crawling.First, to search for malicious activities and vulnerabilities, oneneeds to design the tools that extract pertinent information from Android applications. It is the basis of any analysis. Furthermore,any classifier or detector is always limited by the informative power of underlying data. An important part of this thesis is the designing of efficient static and dynamic analysis tools forapplications, such as an reverse engineering module, a networkcommunication analysis tool, an instrumented Android system, an application web crawlers etc.Neural Network initialization, training and anti-saturation techniques algorithm.Neural Networks are randomly initialized. It is possible to control the underlying random distribution in order to the reduce the saturation effect, the training time and the capacity to reach theglobal minimum. We developed an initialization procedure that enhances the results compared to the state-of-the-art. We also revisited ADAM algorithm to take into account interdependencies with regularization techniques, in particular Dropout. Last, we use anti-saturation techniques and we show that they are required tocorrectly train a neural network.An algorithm for collecting the common sequences in a sequence group.We propose a new algorithm for building the Embedding Antichain fromthe set of common subsequences. It is able to process and represent allcommon subsequences of a sequence set. It is a tool for solving the Systematic Characterization of Sequence Groups. This algorithm is a newpath of research toward the automatic creation of malware familydetection rules.

Android is the second most targeted operating system for malware authors and to counter the development of Android malware, more knowledge about their behavior is needed. There are mainly two approaches to analyze Android malware, namely static and dynamic analysis. Recently in 2017, a study and well labeled dataset, named AMD (Android Malware Dataset), consisting of over 24,000 malware samples was released. It is divided into 135 varieties based on similar malicious behavior, retrieved through static analysis of the file classes.dex in the APK of each malware, whereas the labeled features were determined by manual inspection of three samples in each variety. However, static analysis is known to be weak against obfuscation techniques, such as repackaging or dynamic loading, which can be exploited to avoid the analysis. In this study the second approach is utilized and all malware in the dataset are analyzed at run-time in order to monitor their dynamic behavior. However, analyzing malware at run-time has known weaknesses as well, as it can be avoided through, for instance, anti-emulator techniques. Therefore, the study aimed to explore the available sandbox environments for dynamic analysis, study the effectiveness of fingerprinting Android malware using one of the tools and investigate whether static features from AMD and the dynamic analysis correlate. For instance, by an attempt to classify the samples based on similar dynamic features and calculating the Pearson Correlation Coefficient (r) for all combinations of features from AMD and the dynamic analysis. The comparison of tools for dynamic analysis, showed a need of development, as most popular tools has been released for a long time and the common factor is a lack of continuous maintenance. As a result, the choice of sandbox environment for this study ended up as Droidbox, because of aspects like ease of use/install and easily adaptable for large scale analysis. Based on the dynamic features extracted with Droidbox, it could be shown that Android malware are more similar to the varieties which they belong to. The best metric for classifying samples to varieties, out of four investigated metrics, turned out to be Cosine Similarity, which received an accuracy of 83.6% for the entire dataset. The high accuracy indicated a correlation between the dynamic features and static features which the varieties are based on. Furthermore, the Pearson Correlation Coefficient confirmed that the manually extracted features, used to describe the varieties, and the dynamic features are correlated to some extent, which could be partially confirmed by a manual inspection in the end of the study.