Worldwide computer systems continue to execute malicious software that degrades the systemsâ performance and consumes network capacity by generating high volumes of unwanted traffic. Network-based detectors can effectively identify machines participating in the ongoing attacks by monitoring the traffic to and from the systems. But, network detection alone is not enough; it does not improve the operation of the Internet or the health of other machines connected to the network. We must identify malicious code running on infected systems, participating in global attack networks. This dissertation describes a robust and secure approach that identifies malware present on infected systems based on its undesirable use of network. Our approach, using virtualization, attributes malicious traffic to host-level processes responsible for the traffic. The attribution identifies on-host processes, but malware instances often exhibit parasitic behaviors to subvert the execution of benign processes. We then augment the attribution software with a host-level monitor that detects parasitic behaviors occurring at the user- and kernel-level. User-level parasitic attack detection happens via the system-call interface because it is a non-bypassable interface for user-level processes. Due to the unavailability of one such interface inside the kernel for drivers, we create a new driver monitoring interface inside the kernel to detect parasitic attacks occurring through this interface. Our attribution software relies on a guest kernelâ s data to identify on-host processes. To allow secure attribution, we prevent illegal modifications of critical kernel data from kernel-level malware. Together, our contributions produce a unified research outcome --an improved malicious code identification system for user- and kernel-level malware.

Doctor Educationis / Cybersecurity defense tools, techniques and methodologies are constantly faced with increasing challenges including the evolution of highly intelligent and powerful new generation threats. The main challenges posed by these modern digital multi-vector attacks is their ability to adapt with machine learning. Research shows that many existing defense systems fail to provide adequate protection against these latest threats. Hence, there is an ever-growing need for self-learning technologies that can autonomously adjust according to the behaviour and patterns of the offensive actors and systems. The accuracy and effectiveness of existing methods are dependent on decision making and manual input by human expert. This dependence causes 1) administration overhead, 2) variable and potentially limited accuracy and 3) delayed response time. In this thesis, Autonomous Threat Detection and Response (ATDR) is a proposed general method aimed at contributing toward security related self-protected networks. Through a combination of unsupervised machine learning and Deep learning, ATDR is designed as an intelligent and autonomous decision-making system that uses big data processing requirements and data frame pattern identification layers to learn sequences of patterns and derive real-time data formations. This system enhances threat detection and response capabilities, accuracy and speed. Research provided a solid foundation for the proposed method around the scope of existing methods and the unanimous problem statements and findings by other authors.

Cybersecurity has become an increasingly important concern for both military and civilian infrastructure globally. Because of the complexity that comes with wireless networks, adversaries have many means of infiltration and disruption of wireless networks. While there is much research done in defending these networks, understanding the robustness of these networks is tantamount for both designing new networks and examining possible security deficiencies in preexisting networks. This dissertation proposes to examine the robustness of wireless networks on three major fronts: the physical layer, the data-link layer, and the network layer. At the physical layer, denial-of-service jamming attacks are considered, and both additive interference and no interference are modeled in an optimal configuration and five common network topologies. At the data-link layer, data transmission efficacy and denial-of-sleep attacks are considered with the goal of maximizing throughput under a constrained lifetime. At the network layer, valid and anomalous communications are considered with the goal of classifying those anomalous communications apart from valid ones. This dissertation proposes that a thorough analysis of the aforementioned three layers provides valuable insights to robustness on general wireless networks.

As the IoT market continues to grow, so does the need for secure wireless communication. Drones have become a popular gadget among both individuals and various industries during the last decade, and the popularity continues to grow. Some drones use Wi-Fi technology for communication, such as the Parrot ANAFI, which introduces many of the same security threats that are frequently found in general IoT. Therefore, this report covers a common group of cyberattacks, known as denial-of-service attacks, their effects on the Parrot ANAFI, and their ease of use. A threat model was created to have an overview of the system architecture, and all of the identified threats were assessed using DREAD. All of the software tools used in this report can be found for free on the Internet using search engines and simple key words. The results showed that the drone is generally secure, but it is vulnerable to a certain denial-of-service attack, which can open the door to multiple attack surfaces if the password for the drone’s Wi-Fi is not strong enough. Some suggestions for mitigating these threats are presented at the end of the report. / I takt med att IoT-marknaden fortsätter att växa ökar också behovet av säker trådlös kommunikation. Drönare har blivit en populär pryl bland såväl privatpersoner som diverse industrier under det senaste decenniet, och populariteten fortsätter att växa. Vissa drönare använder Wi-Fi-teknik för kommunikation, till exempel Parrot ANAFI, vilket introducerar många av de säkerhetshot som ofta existerar bland IoT i allmänhet. Den här rapporten täcker därför en välkänd grupp av cyberattacker, som kallas denial-of-service-attacker, deras effekter på Parrot ANAFI och deras användarvänlighet. En hotmodell skapades för att ha en överblick över systemarkitekturen och alla identifierade hot rangordnades med hjälp av DREAD. Alla programvaruverktyg som används i denna rapport kan hittas gratis på Internet med hjälp av enkla sökningar på nyckelord. Resultaten påvisar att drönaren i allmänhet är säker, men att den är sårbar för en viss typ av denial-of-service-attack, vilket kan öppna dörren till flera attackytor om lösenordet för drönarens Wi-Fi inte är tillräckligt starkt. Några förslag för att mildra dessa hot presenteras i slutet av rapporten.

Yes / Software-defined networking (SDN) and network functions virtualisation (NFV) are crucial technologies for integration in the fifth generation of cellular networks (5G). However, they also pose new security challenges, and a timely research subject is working on intrusion detection systems (IDSs) for 5G networks. Current IDSs suffer from several limitations, resulting in a waste of resources and some security threats. This work proposes a new three-layered solution that includes forwarding and data transport, management and control, and virtualisation layers, emphasising distributed controllers in the management and control layer. The proposed solution uses entropy detection to classify arriving packets as normal or suspicious and then forwards the suspicious packets to a centralised controller for further processing using a self-organising map (SOM). A dynamic OpenFlow switch relocation method is introduced based on deep reinforcement learning to address the unbalanced burden among controllers and the static allocation of OpenFlow switches. The proposed system is analysed using the Markov decision process, and a Double Deep Q-Network (DDQN) is used to train the system. The experimental results demonstrate the effectiveness of the proposed approach in mitigating DDoS attacks, efficiently balancing controller workloads, and reducing the duration of the balancing process in 5G networks.

Um die ständig wachsenden Anforderungen zur Übertragung von Live Video Streams im Internet zu erfüllen werden kosteneffektive und resourceneffiziente Lösungen benötigt. Eine adäquate Lösung bietet die Peer-to-Peer (P2P) Streaming Architektur an, welche bereits heute in unterschiedlichsten Systemen zum Einsatz kommt. Solche Systeme erfordern von der Streaming Quelle nur moderate Bandbreiten, da die Nutzer (bzw. Peers) ihre eigene Bandbreite zur Verbreitung des Streams einbringen. Dazu werden die Peers oberhalb der Internetarchitektur zu einem Overlay verbunden. Das geplante Verlassen, sowie der ungewollte Absturz von Peers (genannt Churn) kann das Overlay schädigen und den Empfang einiger Peers unterbrechen. Weitaus kritischer sind Angriffe auf die Verfügbarkeit des Systems indem relevante Knoten des Overlays von Angreifern attackiert werden, um die Verteilung des Streams gezielt zu stören. Um Overlays zu konstruieren, die robust gegenüber Churn sind, nutzen so genannte pull-basierte P2P Streaming Systeme eine Mesh Topologie um jeden Peer über mehrere Pfade mit der Quelle zu verbinden. Peers fordern regelmäßig Teile des Videos, sog. Chunks, von ihren Partnern im Overlay an. Selbst wenn einige Partner plötzlich nicht mehr im System verfügbar sind kann ein Peer alle Chunks von den verbleibenden Nachbarn beziehen. Um dies zu ermöglichen tauschen Peers regelmäßig sog. Buffer Maps aus. Diese kleinen Pakete enthalten Informationen über die Verfügbarkeit von Chunks im Puffer eines Peers. Um dadurch entstehende Latenzen und den zusätzlichen Mehraufwand zu reduzieren wurden hybride Systeme entwickelt. Ein solches System beginnt pull-basiert und formt mit der Zeit einen Baum aus einer kleinen Untermenge aller Peers um Chunks ohne explizite Anfrage weiterzuleiten. Unglücklicherweise sind sowohl pull-basierte, als auch hybride Systeme anfällig gegenüber Denial-of-Service Angriffen (DoS). Insbesondere fehlen Maßnahmen zur Abschwächung von DoS Angriffen auf die Partner der Quelle. Die genannten Angriffe werden weiterhin dadurch erleichtert, dass die Identität der Quelle-nahen Knoten akkurat aus den ausgetauschten Buffer Maps extrahiert werden kann. Hybride Systeme sind außerdem anfällig für Angriffe auf den zugrundeliegenden Baum. Aufgrund der schwerwiegenden Auswirkungen von DoS Angriffen auf pull-basierte, sowie hybride Systeme stellen wir drei Gegenmaßnahmen vor. Zuerst entwickeln wir das Striping Schema zur Abschwächung von DoS Angriffen auf die Partner der Quelle. Hierbei werden Peers dazu angeregt ihre Chunk-Anfragen an unterschiedliche Partner zu senden. Als zweites entwickeln wir das SWAP Schema, welches Peers dazu bringt proaktiv ihre Partner zu wechseln um Angreifer daran zu hindern die Quellenahe zu identifizieren. Als drittes entwickeln wir RBCS, einen widerstandsfähigen Baum zur Abschwächung von DoS Angriffen auf hybride Systeme. Da bisher kein Simulator für die faire Evaluation von P2P-basierten Live Video Streaming Algorithmen verfügbar war, entwickeln wir OSSim, ein generalisiertes Simulations-Framework für P2P-basiertes Video Streaming. Des weiteren entwickeln wir etliche Angreifermodelle sowie neuartige Resilienzmetriken on OSSim. Ausgiebige Simulationsstudien zeigen, dass die entwickelten Schemata signifikant die Widerstandsfähigkeit von pull-basierten und hybriden Systemen gegenüber Churn und DoS Angriffen erhöhen. / The constantly growing demand to watch live videos over the Internet requires streaming systems to be cost-effective and resource-efficient. The Peer-to-Peer (P2P) streaming architecture has been a viable solution with various deployed systems to date. The system only requires a modest amount of bandwidth from the streaming source, since users (or peers) contribute their bandwidth to disseminate video streams. To enable this, the system interconnects peers into an overlay. However, churn–meaning the leaving and failing of peers–can break the overlay, making peers unable to receive the stream. More severely, an adversary aiming to sabotage the system can attack relevant nodes on the overlay, disrupting the stream delivery. To construct an overlay robust to churn, pull-based P2P streaming systems use a mesh topology to provide each peer with multiple paths to the source. Peers regularly request video chunks from their partners in the overlay. Therefore, even if some partners are suddenly absent, due to churn, a peer still can request chunks from its remaining partners. To enable this, peers periodically exchange buffer maps, small packets containing the availability information of peers’ video buffers. To reduce latency and overhead caused by the periodic buffer map exchange and chunk requests, hybrid systems have been proposed. A hybrid system bootstraps from a pull-based one and gradually forms a tree backbone consisting of a small subset of peers to deliver chunks without requests. Unfortunately, both pull-based and hybrid systems lack measures to mitigate Denial-of-Service (DoS) attacks on head nodes (or the source’s partners). More critically, they can be identified accurately by inferring exchanged buffer maps. Furthermore, hybrid systems are vulnerable to DoS attacks on their backbones. Since DoS attacks can badly affect both pull-based and hybrid systems, we introduce three countermeasures. First, we develop the striping scheme to mitigate DoS attacks targeting head nodes. The scheme enforces peers to diversify their chunk requests. Second, to prevent attackers from identifying head nodes, we develop the SWAP scheme, which enforces peers to proactively change their partners. Third, we develop RBCS, a resilient backbone, to mitigate DoS attacks on hybrid systems. Since a simulator for a fair evaluation is unavailable so far, we develop OSSim, a general-purpose simulation framework for P2P video streaming. Furthermore, we develop several attacker models and novel resilience metrics in OSSim. Extensive simulation studies show that the developed schemes significantly improve the resilient of pull-based and hybrid systems to both churn and DoS attacks.

If not the most, one of the serious threats to data networks, particularly pervasive commercial networks such as Voice-over-IP (VoIP) providers is Denial-of-Service (DoS) attack. Currently, majority of solutions for these attacks focus on observing detailed server state changes due to any or some of the incoming messages. This approach however requires significant amount of server’s memory and processing time. This results in detectors not being able to scale up to the network edge points that receive millions of connections (requests) per second. To solve this problem, it is desirable to design stateless detection mechanisms. One approach is to aggregate transactions into groups. This research focuses on stateless scalable DoS intrusion detection mechanisms to obviate keeping detailed state for connections while maintaining acceptable efficiency. To this end, we adopt a two-layer aggregation scheme termed Advanced Partial Completion Filters (APCF), an intrusion detection model that defends against DoS attacks without tracking state information of each individual connection. Analytical as well as simulation analysis is performed on the proposed APCF. A simulation test bed has been implemented in OMNET++ and through simulations it is observed that APCF gained notable detection rates in terms of false positive and true positive detections, as opposed to its predecessor PCF. Although further study is needed to relate APCF adjustments to a certain network situation, this research shows invaluable gain to mitigate intrusion detection from not so scalable state-full mechanisms to aggregate scalable approach.

Denial-of-service attacks (DoS) and distributed denial-of-service attacks (DDoS) attempt to temporarily disrupt users or computer resources to cause service un- availability to legitimate users in the internetworking system. The most common type of DoS attack occurs when adversaries °ood a large amount of bogus data to interfere or disrupt the service on the server. The attack can be either a single-source attack, which originates at only one host, or a multi-source attack, in which multiple hosts coordinate to °ood a large number of packets to the server. Cryptographic mechanisms in authentication schemes are an example ap- proach to help the server to validate malicious tra±c. Since authentication in key establishment protocols requires the veri¯er to spend some resources before successfully detecting the bogus messages, adversaries might be able to exploit this °aw to mount an attack to overwhelm the server resources. The attacker is able to perform this kind of attack because many key establishment protocols incorporate strong authentication at the beginning phase before they can iden- tify the attacks. This is an example of DoS threats in most key establishment protocols because they have been implemented to support con¯dentiality and data integrity, but do not carefully consider other security objectives, such as availability. The main objective of this research is to design denial-of-service resistant mechanisms in key establishment protocols. In particular, we focus on the design of cryptographic protocols related to key establishment protocols that implement client puzzles to protect the server against resource exhaustion attacks. Another objective is to extend formal analysis techniques to include DoS- resistance. Basically, the formal analysis approach is used not only to analyse and verify the security of a cryptographic scheme carefully but also to help in the design stage of new protocols with a high level of security guarantee. In this research, we focus on an analysis technique of Meadows' cost-based framework, and we implement DoS-resistant model using Coloured Petri Nets. Meadows' cost-based framework is directly proposed to assess denial-of-service vulnerabil- ities in the cryptographic protocols using mathematical proof, while Coloured Petri Nets is used to model and verify the communication protocols using inter- active simulations. In addition, Coloured Petri Nets are able to help the protocol designer to clarify and reduce some inconsistency of the protocol speci¯cation. Therefore, the second objective of this research is to explore vulnerabilities in existing DoS-resistant protocols, as well as extend a formal analysis approach to our new framework for improving DoS-resistance and evaluating the performance of the new proposed mechanism. In summary, the speci¯c outcomes of this research include following results; 1. A taxonomy of denial-of-service resistant strategies and techniques used in key establishment protocols; 2. A critical analysis of existing DoS-resistant key exchange and key estab- lishment protocols; 3. An implementation of Meadows's cost-based framework using Coloured Petri Nets for modelling and evaluating DoS-resistant protocols; and 4. A development of new e±cient and practical DoS-resistant mechanisms to improve the resistance to denial-of-service attacks in key establishment protocols.

Le cloud computing, solution souple et peu couteuse, est aujourd'hui largement adopté pour la production à grande échelle de services IT. Toutefois, des utilisateurs malveillants tirent parti de ces caractéristiques pour bénéficier d'une plate-forme d'attaque prête à l'emploi dotée d'une puissance colossale. Parmi les plus grands bénéficiaires de cette conversion en vecteur d’attaque, les botclouds sont utilisés pour perpétrer des attaques de déni de service distribuées (DDoS) envers tout tiers connecté à Internet.Si les attaques de ce type, perpétrées par des botnets ont été largement étudiées par le passé, leur mode opératoire et leur contexte de mise en œuvre sont ici différents et nécessitent de nouvelles solutions. Pour ce faire, nous proposons dans le travail de thèse exposé dans ce manuscrit, une approche distribuée pour la détection à la source d'attaques DDoS perpétrées par des machines virtuelles hébergées dans un cloud public. Nous présentons tout d'abord une étude expérimentale qui a consisté à mettre en œuvre deux botclouds dans un environnement de déploiement quasi-réel hébergeant une charge légitime. L’analyse des données collectées permet de déduire des invariants comportementaux qui forment le socle d'un système de détection à base de signature, fondé sur une analyse en composantes principales. Enfin, pour satisfaire au support du facteur d'échelle, nous proposons une solution de distribution de notre détecteur sur la base d'un réseau de recouvrement pair à pair structuré qui forme une architecture hiérarchique d'agrégation décentralisée / Currently, cloud computing is a flexible and cost-effective solution widely adopted for the large-scale production of IT services. However, beyond a main legitimate usage, malicious users take advantage of these features in order to get a ready-to-use attack platform, offering a massive power. Among the greatest beneficiaries of this cloud conversion into an attack support, botclouds are used to perpetrate Distributed Denial of Service (DDoS) attacks toward any third party connected to the Internet.Although such attacks, when perpetrated by botnets, have been extensively studied in the past, their operations and their implementation context are different herein and thus require new solutions. In order to achieve such a goal, we propose in the thesis work presented in this manuscript, a distributed approach for a source-based detection of DDoS attacks perpetrated by virtual machines hosted in a public cloud. Firstly, we present an experimental study that consists in the implementation of two botclouds in a real deployment environment hosting a legitimate workload. The analysis of the collected data allows the deduction of behavioural invariants that form the basis of a signature based detection system. Then, we present in the following a detection system based on the identification of principal components of the deployed botclouds. Finally, in order to deal with the scalability issues, we propose a distributed solution of our detection system, which relies on a mesh peer-to- peer architecture resulting from the overlap of several overlay trees

Networks of maliciously compromised computers, known as botnets, consisting of thousands of hosts have emerged as a serious threat to Internet security in recent years. These compromised systems, under the control of an operator are used to steal data, distribute malware and spam, launch phishing attacks and in Distributed Denial-of-Service (DDoS) attacks. The operators of these botnets use Command and Control (C2) servers to communicate with the members of the botnet and send commands. The communications channels between the C2 nodes and endpoints have employed numerous detection avoidance mechanisms to prevent the shutdown of the C2 servers. Two prevalent detection avoidance techniques used by current botnets are algorithmically generated domain names and DNS Fast-Flux. The use of these mechanisms can however be observed and used to create distinct signatures that in turn can be used to detect DNS domains being used for C2 operation. This report details research conducted into the implementation of three classes of classification techniques that exploit these signatures in order to accurately detect botnet traffic. The techniques described make use of the traffic from DNS query responses created when members of a botnet try to contact the C2 servers. Traffic observation and categorisation is passive from the perspective of the communicating nodes. The first set of classifiers explored employ frequency analysis to detect the algorithmically generated domain names used by botnets. These were found to have a high degree of accuracy with a low false positive rate. The characteristics of Fast-Flux domains are used in the second set of classifiers. It is shown that using these characteristics Fast-Flux domains can be accurately identified and differentiated from legitimate domains (such as Content Distribution Networks exhibit similar behaviour). The final set of classifiers use spatial autocorrelation to detect Fast-Flux domains based on the geographic distribution of the botnet C2 servers to which the detected domains resolve. It is shown that botnet C2 servers can be detected solely based on their geographic location. This technique is shown to clearly distinguish between malicious and legitimate domains. The implemented classifiers are lightweight and use existing network traffic to detect botnets and thus do not require major architectural changes to the network. The performance impact of implementing classification of DNS traffic is examined and it is shown that the performance impact is at an acceptable level.