Tato práce se zabývá metodami pro behaviorální detekci malware, které využívají techniky formální analýzy a verifikace. Základem je odvozování stromových automatů z grafů závislostí systémových volání, které jsou získány pomocí statické analýzy LLVM IR. V rámci práce je implementován prototyp detektoru, který využívá překladačovou infrastrukturu LLVM. Pro experimentální ověření detektoru je použit překladač jazyka C/C++, který je schopen generovat mutace malware za pomoci obfuskujících transformací. Výsledky předběžných experimentů a případná budoucí rozšíření detektoru jsou diskutovány v závěru práce.

The internet has become an essential part of most people's daily lives in recent years, and as more devices connect to the internet, the risk of cyber threats increases dramatically. As malware becomes more sophisticated, traditional security prevention measures are becoming less effective at defending from cyber attacks. As a result, Signature Based Detection and Anomaly Detection are two of many advanced techniques that have become crucial to defend against cyber threats such as malware, but even these are sometimes not enough to stop modern cyberattacks. In this literature review the goal is to discuss how trustworthy each of the mentioned malware detection techniques are at detecting malware in wireless networks. The study will measure trustworthiness by looking further into scalability, adaptability and robustness and resource consumption. This study concludes that both anomaly and signature-based malware detection methods exhibit strengths and weaknesses in scalability, robustness, adaptability, and resource consumption. Furthermore, more research is needed and as malware becomes more sophisticated and an increased threat to the world it is an area that is highly relevant.

This research investigates the use of data mining methods for malware (malicious programs) detection and proposed a framework as an alternative to the traditional signature detection methods. The traditional approaches using signatures to detect malicious programs fails for the new and unknown malwares case, where signatures are not available. We present a data mining framework to detect malicious programs. We collected, analyzed and processed several thousand malicious and clean programs to find out the best features and build models that can classify a given program into a malware or a clean class. Our research is closely related to information retrieval and classification techniques and borrows a number of ideas from the field. We used a vector space model to represent the programs in our collection. Our data mining framework includes two separate and distinct classes of experiments. The first are the supervised learning experiments that used a dataset, consisting of several thousand malicious and clean program samples to train, validate and test, an array of classifiers. In the second class of experiments, we proposed using sequential association analysis for feature selection and automatic signature extraction. With our experiments, we were able to achieve as high as 98.4% detection rate and as low as 1.9% false positive rate on novel malwares.

The large scale usage of the PDF coupled with its versatility has made the format an attractive target for carrying and deploying malware. Traditional antivirus software struggles against new malware and PDF's vast obfuscation options. In the search of better detection systems, machine learning based detectors have been developed. Although their approaches vary, some strictly examine structural features of the document whereas other examine the behavior of embedded code, they generally share high accuracy against the evaluation data they have been tested against. However, structural machine learning based PDF malware detectors have been found to be weak against targeted evasion attempts that may be found in more sophisticated malware. Such evasion attempts typically exploit knowledge of what the detection system associates with 'benign' and 'malicious' to emulate benign features or exploit a bug in the implementation, with the purpose of evading the detector. Since the introduction of such evasion attacks more structural detectors have been developed, without introducing mitigations against such evasion attacks. This thesis aggregates the existing knowledge of evasion strategies and applies them against a reproduction of a recent, not previously evasion tested, detection system and finds that it is susceptible to various evasion techniques. Additionally, the produced detector is experimentally trained with a combination of the standard data and the recently published CIC-Evasive-PDFMal2022 dataset which contains malware samples which display evasive properties. The evasive-trained detector is tested against the same set of evasion attacks. The results of the two detectors are compared, concluding that supplementing the training data with evasive samples results in a more evasion resilient detector. / Flexibiliteten och mångsidigheten hos PDF-filer har gjort dessa till attraktiva attackvektorer, där en användare eller ett system riskerar att utsättas för skadlig kod vid läsning av dessa filer. Som åtgärd har formatsspecifika, vanligtvis maskininlärningsbaserade, detektorer utvecklats. Dessa detektorer ämnar att, givet en PDF-fil, ge ett svar: skadlig eller oskadlig, ofta genom att inspektera strukturella egenskaper hos dokumentet. Strukturella detektorer har påvisats sårbara mot riktade undvikningsattacker som, genom att efterlikna egenskaper hos oskadliga dokument, lyckas smuggla skadliga dokument förbi sådana detektorer. Trots detta har liknande detektorer fortsatt utvecklas, utan att implementera försvar mot sådana attacker. Detta arbete testar en modern strukturell detektor med undvikningsattacker bestående av attackfiler av olika obfuskeringsnivåer och bekräftar att dessa svagheter kvarstår. Dessutom prövas en experimentell försvarsåtgärd i form av att tillsätta typiskt normavvikande PDF-filer (från datasetet CIC-Evasive-PDFMal2022) till träningssteget under konstruktionen av detektorn, för att identifiera hur detta påverkar resistensen mot undvikningsattacker. Detektorvarianterna prövas mot samma attackfiler för att jämföras mot varandra. Resultaten från detta påvisar en ökad resistens i detektorn med tillskottet av avikande träningsdata.

The continuity in the financial and other related losses due to cyber-attacks prove the substantial growth of malware and their lethal proliferation techniques. Every successful malware attack highlights the weaknesses in the defence mechanisms responsible for securing the targeted computer or a network. The recent cyber-attacks reveal the presence of sophistication and intelligence in malware behaviour having the ability to conceal their code and operate within the system autonomously. The conventional detection mechanisms not only possess the scarcity in malware detection capabilities, they consume a large amount of resources while scanning for malicious entities in the system. Many recent reports have highlighted this issue along with the challenges faced by the alternate solutions and studies conducted in the same area. There is an unprecedented need of a resilient and autonomous solution that takes proactive approach against modern malware with stealth behaviour. This thesis proposes a multi-aspect solution comprising of an intelligent malware detection framework and an energy efficient hosting model. The malware detection framework is a combination of conventional and novel malware detection techniques. The proposed framework incorporates comprehensive feature heuristics of files generated by a bespoke static feature extraction tool. These comprehensive heuristics are used to train the machine learning algorithms; Support Vector Machine, Decision Tree, and Boosting to differentiate between clean and malicious files. Both these techniques; feature heuristics and machine learning are combined to form a two-factor detection mechanism. This thesis also presents a cloud-based energy efficient and scalable hosting model, which combines multiple infrastructure components of Amazon Web Services to host the malware detection framework. This hosting model presents a client-server architecture, where client is a lightweight service running on the host machine and server is based on the cloud. The proposed framework and the hosting model were evaluated individually and combined by specifically designed experiments using separate repositories of clean and malicious files. The experiments were designed to evaluate the malware detection capabilities and energy efficiency while operating within a system. The proposed malware detection framework and the hosting model showed significant improvement in malware detection while consuming quite low CPU resources during the operation.

This research work presents the design and implementation of a host-based Intrusion Detection and Prevention System (IDPS) called HIDROID (Host-based Intrusion Detection and protection system for andROID) for Android smartphones. It runs completely on the mobile device, with a minimal computation burden. It collects data in real-time, periodically sampling features that reflect the overall utilisation of scarce resources of a mobile device (e.g. CPU, memory, battery, bandwidth, etc.). The Detection Engine of HIDROID adopts an anomaly-based approach by exploiting statistical and machine learning algorithms. That is, it builds a data-driven model for benign behaviour and looks for the outliers considered as suspicious activities. Any observation failing to match this model triggers an alert and the preventive agent takes proper countermeasure(s) to minimise the risk. The key novel characteristic of the Detection Engine of HIDROID is the fact that it requires no malicious data for training or tuning. In fact, the Detection Engine implements the following two anomaly detection algorithms: a variation of K-Means algorithm with only one cluster and the univariate Gaussian algorithm. Experimental test results on a real device show that HIDROID is well able to learn and discriminate normal from anomalous behaviour, demonstrating a very promising detection accuracy of up to 0.91, while maintaining false positive rate below 0.03. Finally, it is noteworthy to mention that to the best of our knowledge, publicly available datasets representing benign and abnormal behaviour of Android smartphones do not exist. Thus, in the context of this research work, two new datasets were generated in order to evaluate HIDROID. / Fundação para a Ciência e Tecnologia (FCT-Portugal) with reference SFRH/BD/112755/2015, European Regional Development Fund (FEDER), through the Competitiveness and Internationalization Operational Programme (COMPETE 2020), Regional Operational Program of the Algarve (2020), Fundação para a Ciência e Tecnologia; i-Five .: Extensão do acesso de espectro dinâmico para rádio 5G, POCI-01-0145-FEDER-030500, Instituto de telecomunicações, (IT-Portugal) as the host institution.

Since 2015, malicious actors have been using SVG files to obfuscate malware from potential defensive mechanisms and carry out attacks undetected through the use of smuggling tech- niques [1]. Throughout this thesis, we use the Design Science Research methodology in order to design and develop an artefact able to detect these attacks within a real network infrastruc- ture, while minimising the impact on the user experience. For the designed artefact to answer these challenges, we conduct two scoping reviews: an analysis of seven of these incidents to determine the technique used to perform the smuggling. This is followed by a map of the dif- ferent security processes available to network administrators and individuals who search for open-source technologies and aim to close the gap left by lack of these solutions.  Moreover the paper proposes a SVG parser and a Random Forest classifier to extract valu- able features needed to find the malicious payloads hidden in the graphics. The performance of the artefact is analysed to determine its suitability for real-world usage and if an adequate success rate is reached. The paper finally concludes that the task of obfuscated malware de- tection is a multi-faceted problem and the artefact, while successful, is a suitable blueprint for exploring future improvements in the field.

L’analyse comportementale traditionnelle opère en général au niveau de l’implantation de comportements malveillants. Pourtant, elle s’intéresse surtout à l’identification de fonctionnalités données et elle se situe donc plus naturellement à un niveau fonctionnel. Dans cette thèse, nous définissons une forme d’analyse comportementale de programmes qui opère non pas sur les interactions élémentaires d’un programme avec le système mais sur la fonction que le programme réalise. Cette fonction est extraite des traces d’un pro- gramme, un procédé que nous appelons abstraction. Nous définissons de façon simple, intuitive et formelle les fonctionnalités de base à abstraire et les comportements à détecter, puis nous proposons un mécanisme d’abstraction applicable à un cadre d’analyse statique ou dynamique, avec des algorithmes pratiques à complexité raisonnable, enfin nous décrivons une technique d’analyse comportementale intégrant ce mécanisme d’abstraction. Notre méthode est particulièrement adaptée à l’analyse des programmes dans des langages de haut niveau ou dont le code source est connu, pour lesquels l’analyse statique est facilitée : applications mobiles en .NET ou Java, scripts, extensions de navigateurs, composants off-the-shelf.Le formalisme d’analyse comportementale par abstraction que nous proposons repose sur la théorie de la réécriture de mots et de termes, les langages réguliers de mots et de termes et le model checking. Il permet d’identifier efficacement des fonctionnalités dans des traces et ainsi d’obtenir une représentation des traces à un niveau fonctionnel; il définit les fonctionnalités et les comportements de façon naturelle, à l’aide de formules de logique temporelle, ce qui garantit leur simplicité et leur flexibilité et permet l’utilisation de techniques de model checking pour la détection de ces comportements ; il opère sur un ensemble quelconque de traces d’exécution ; il prend en compte le flux de données dans les traces d’exécution; et il permet, sans perte d’efficacité, de tenir compte de l’incertitude dans l’identification des fonctionnalités. Un cadre d’expérimentation a été mis en place dans un contexte d’analyse dynamique comme statique / Traditional behavior analysis usually operates at the implementation level of malicious behaviors. Yet, it is mostly concerned with the identification of given functionalities and is therefore more naturally defined at a functional level. In this thesis, we define a form of program behavior analysis which operates on the function realized by a program rather than on its elementary interactions with the system. This function is extracted from program traces, a process we call abstraction. We define in a simple, intuitive and formal way the basic functionalities to abstract and the behaviors to detect, then we propose an abstraction mechanism applicable both to a static or to a dynamic analysis setting, with practical algorithms of reasonable complexity, finally we describe a behavior analysis technique integrating this abstraction mechanism. Our method is particularly suited to the analysis of programs written in high level languages or with a known source code, for which static analysis is facilitated: mobile applications for .NET or Java, scripts, browser addons, off-the-shelf components.The formalism we propose for behavior analysis by abstraction relies on the theory of string and terms rewriting, word and tree languages and model checking. It allows an efficient identification of functionalities in traces and thus the construction of a represen- tation of traces at a functional level; it defines functionalities and behaviors in a natural way, using temporal logic formulas, which assure their simplicity and their flexibility and enables the use of model checking techniques for behavior detection; it operates on an unrestricted set of execution traces; it handles the data flow in execution traces; and it allows the consideration of uncertainty in the identification of functionalities, with no complexity overhead. Experiments have been conducted in a dynamic and static analysis setting

We have entered the age of big data. Massive datasets are now common in science, government and enterprises. Yet, making sense of these data remains a fundamental challenge. Where do we start our analysis? Where to go next? How to visualize our findings? We answers these questions by bridging Data Mining and Human- Computer Interaction (HCI) to create tools for making sense of graphs with billions of nodes and edges, focusing on: (1) Attention Routing: we introduce this idea, based on anomaly detection, that automatically draws people’s attention to interesting areas of the graph to start their analyses. We present three examples: Polonium unearths malware from 37 billion machine-file relationships; NetProbe fingers bad guys who commit auction fraud. (2) Mixed-Initiative Sensemaking: we present two examples that combine machine inference and visualization to help users locate next areas of interest: Apolo guides users to explore large graphs by learning from few examples of user interest; Graphite finds interesting subgraphs, based on only fuzzy descriptions drawn graphically. (3) Scaling Up: we show how to enable interactive analytics of large graphs by leveraging Hadoop, staging of operations, and approximate computation. This thesis contributes to data mining, HCI, and importantly their intersection, including: interactive systems and algorithms that scale; theories that unify graph mining approaches; and paradigms that overcome fundamental challenges in visual analytics. Our work is making impact to academia and society: Polonium protects 120 million people worldwide from malware; NetProbe made headlines on CNN, WSJ and USA Today; Pegasus won an opensource software award; Apolo helps DARPA detect insider threats and prevent exfiltration. We hope our Big Data Mantra “Machine for Attention Routing, Human for Interaction” will inspire more innovations at the crossroad of data mining and HCI.

The goal of this thesis is to create a virtual machine that emulates a running programs written in Java programing language, which would be suitable for malware analysis and detection. The emulator is able to detect arguments of exploitable methods from Java standard classes, the order of calling these exploitable methods and also execution the test application. Overall functionality was tested on appropriate examples in which held its own measurements. At the end of the paper we describe testing of the emulator, which also contains tables and graphs for better results visualization.