This research created a new theoretical Soft Computing (SC) hybridized network intrusion detection diagnostic system including complex hybridization of a 3D full color Self-Organizing Map (SOM), Artificial Immune System Danger Theory (AISDT), and a Fuzzy Inference System (FIS). This SOM+ diagnostic archetype includes newly defined intrusion types to facilitate diagnostic analysis, a descriptive computational model, and an Invisible Mobile Network Bridge (IMNB) to collect data, while maintaining compatibility with traditional packet analysis. This system is modular, multitaskable, scalable, intuitive, adaptable to quickly changing scenarios, and uses relatively few resources.

The metric of entropy provides a measure about the randomness of data and a measure of information gained by comparing different attributes. Intrusion detection systems can collect very large amounts of data, which are not necessarily manageable by manual means. Collected intrusion detection data often contains redundant, duplicate, and irrelevant entries, which makes analysis computationally intensive likely leading to unreliable results. Reducing the data to what is relevant and pertinent to the analysis requires the use of data mining techniques and statistics. Identifying patterns in the data is part of analysis for intrusion detections in which the patterns are categorized as normal or anomalous. Anomalous data needs to be further characterized to determine if representative attacks to the network are in progress. Often time subtleties in the data may be too muted to identify certain types of attacks. Many statistics including entropy are used in a number of analysis techniques for identifying attacks, but these analyzes can be improved upon. This research expands the use of Approximate entropy and Sample entropy for feature selection and attack analysis to identify specific types of subtle attacks to network systems. Through enhanced analysis techniques using entropy, the granularity of feature selection and attack identification is improved.

As the Internet becomes an ubiquitous medium of communication, it carries more and more malicious activities like spam, worms or denial of service attacks. One solution to detect and collect such malicious traffic is to use honeypots. They are devices or pieces of information that are not part of the usual production system. Their goals are to lure the attackers into a trap to study them, divert their attention from another target or collect statistics. In this work, we propose a lightweight framework to build honeytanks, which are very efficient low-interaction honeypots. We present and evaluate techniques and algorithms to simulate the presence of a large number of hosts with various degrees of realism and scalability, from a completely stateless approach to a stateful approach able, amongst other things, to mimic the behavior of various TCP/IP stacks. Our framework is based on ASAX, a generic and lightweight data stream analyzer. We instantiate ASAX to build powerful traffic handlers. We introduce several extensions to ASAX and to RUSSEL, its programming language. These extensions allow us to develop new concurrent programming techniques to simulate hosts and protocols in a simple and modular way. We use a recently optimized version of ASAX that makes it possible to simulate tens of thousands hosts while keeping the simulation at a high level of realism. To show the benefits of our approach, i.e., greater simplicity, flexibility, and independence of other technologies, we compare our honeytanks to Honeyd and Nepenthes, two well-known low-interaction honeypots.

Voice Over IP (VoIP) telephony is becoming widespread, and is often integrated into computer networks. Because of his, it is likely that malicious software will threaten VoIP systems the same way traditional computer systems have been attacked by viruses, worms, and other automated agents. While most users have become familiar with email spam and viruses in email attachments, spam and malicious traffic over telephony currently is a relatively unknown threat. VoIP networks are a challenge to secure against such malware as much of the network intelligence is focused on the edge devices and access environment. A novel security architecture is being developed which improves the security of a large VoIP network with many inexperienced users, such as non-IT office workers or telecommunication service customers. The new architecture establishes interaction between the VoIP backend and the end users, thus providing information about ongoing and unknown attacks to all users. An evaluation of the effectiveness and performance of different implementations of this architecture is done using virtual machines and network simulation software to emulate vulnerable clients and servers through providing apparent attack vectors.

Modern organisations rely intensively on information and communicationtechnology infrastructures. Such infrastructures offer a range of servicesfrom simple mail transport agents or blogs to complex e-commerce platforms,banking systems or service hosting, and all of these depend on distributedsystems. The security of these systems, with their increasing complexity, isa challenge. Cloud services are replacing traditional infrastructures byproviding lower cost alternatives for storage and computational power, butat the risk of relying on third party companies. This risk becomesparticularly critical when such services are used to host privileged companyinformation and applications, or customers' private information. Even in thecase where companies host their own information and applications, the adventof BYOD (Bring Your Own Device) leads to new security relatedissues.In response, our research investigated the characterization and detection ofmalicious activities at the operating system level and in distributedsystems composed of multiple hosts and services. We have shown thatintrusions in an operating system spawn abnormal information flows, and wedeveloped a model of dynamic information flow tracking, based on taintmarking techniques, in order to detect such abnormal behavior. We trackinformation flows between objects of the operating system (such as files,sockets, shared memory, processes, etc.) and network packetsflowing between hosts. This approach follows the anomaly detection paradigm.We specify the legal behavior of the system with respect to an informationflow policy, by stating how users and programs from groups of hosts areallowed to access or alter each other's information. Illegal informationflows are considered as intrusion symptoms. We have implemented this modelin the Linux kernel (the source code is availableat http://www.blare-ids.org), as a Linux Security Module (LSM), andwe used it as the basis for practical demonstrations. The experimentalresults validated the feasibility of our new intrusion detection principles.

The IEEE 802.11 networks are vulnerable to numerous wireless-specific attacks. Attackers can implement MAC address spoofing techniques to launch these attacks, while masquerading themselves behind a false MAC address. The implementation of Intrusion Detection Systems has become fundamental in the development of security infrastructures for wireless networks. This thesis proposes the designing a novel security system that makes use of metrics from multiple layers of observation to produce a collective decision on whether an attack is taking place. The Dempster-Shafer Theory of Evidence is the data fusion technique used to combine the evidences from the different layers. A novel, unsupervised and self- adaptive Basic Probability Assignment (BPA) approach able to automatically adapt its beliefs assignment to the current characteristics of the wireless network is proposed. This BPA approach is composed of three different and independent statistical techniques, which are capable to identify the presence of attacks in real time. Despite the lightweight processing requirements, the proposed security system produces outstanding detection results, generating high intrusion detection accuracy and very low number of false alarms. A thorough description of the generated results, for all the considered datasets is presented in this thesis. The effectiveness of the proposed system is evaluated using different types of injection attacks. Regarding one of these attacks, to the best of the author knowledge, the security system presented in this thesis is the first one able to efficiently identify the Airpwn attack.

La virtualisation est une technologie dont la popularité ne cesse d’augmenter dans le monde de l’entreprise, et ce pour l’efficacité et la facilité de gestion qu’elle apporte. Cependant, le problème majeur de cette technologie est la sécurité. Dans cette thèse, nous proposons de renforcer la sécurité des systèmes virtualisés et nous introduisons de nouvelles approches pour répondre aux différents besoins en sécurité de cette technologie et aussi aux aspects liés à à son fonctionnement et son déploiement. Nous proposons une nouvelle architecture de supervision qui permet de contrôler la totalité de la plateforme virtualisée en temps réel. L’idée est de simuler une supervision décentralisée (plusieurs postes) sur un seul poste physique. Nous étudions les avantages et les limites de cette approche et nous montrons que cette solution est incapable de réagir é à un certain nombre d’attaques nouvelles. Comme remède, nous introduisons une nouvelle procédure qui permet de sécuriser les ressources critiques d’un système virtualisé pour s’assurer que des familles d’attaques ne peuvent être exécutées en ayant accès à ces ressources. Nous introduisons une variante de LTL avec de nouveaux opérateurs de passé et nous démontrons comment des politiques de sécurité formulées à l’aide de ce langage peuvent être facilement traduites en signatures d’attaques qui sont indispensables à la détection des intrusions dans le système. Nous analysons aussi l’impact d’une communication réseau non sécurisée entre machines virtuelles sur la sécurité globale du système virtualisé. Nous proposons un modèle d’une politique de sécurité multi-niveaux qui couvre la majorité des opérations liées au réseau et qui peuvent être exécutées par une machine virtuelle. Notre modèle couvre aussi des opérations de gestion de l’infrastructure virtualisée et étudie les contraintes de sécurité qui doivent être satisfaites. / Virtual machine technology is rapidly gaining acceptance as a fundamental building block in enterprise data centers. It is most known for improving efficiency and ease of management. However, the central issue of this technology is security. We propose in this thesis to enforce the security of virtualized systems and introduce new approaches that deal with different security aspects related not only to the technology itself but also to its deployment and maintenance. We first propose a new architecture that offers real-time supervision of a complete virtualized architecture. The idea is to implement decentralized supervision on one single physical host. We study the advantages and the limits of this architecture and show that it is unable to react according to some new stealthy attacks. As a remedy, we introduce a new procedure that permits to secure the sensitive resources of a virtualized system and make sure that families of attacks can not be run at all. We introduce a variant of the LTL language with new past operators and show how policies written in this language can be easily translated to attack signatures that we use to detect attacks on the system. We also analyse the impact that an insecure network communication between virtual machines can have on the global security of the virtualized system. We propose a multilevel security policy model that covers almost all the network operations that can be performed by a virtual machine. We also deal with some management operations and introduce the related constraints that must be satisfied when an operation is performed.

Doctor of Philosophy / Department of Computing and Information Sciences / Xinming Ou / Intrusion analysis, i.e., the process of combing through Intrusion Detection System (IDS) alerts and audit logs to identify true successful and attempted attacks, remains a difficult problem in practical network security defense. The primary cause of this problem is the high false positive rate in IDS system sensors used to detect malicious activity. This high false positive rate is attributed to an inability to differentiate nearly certain attacks from those that are merely possible. This inefficacy has created high uncertainty in intrusion analysis and consequently causing an overwhelming amount of work for security analysts. As a solution, practitioners typically resort to a specific IDS-rules set that precisely captures specific attacks. However, this results in failure to discern other forms of the targeted attack because an attack’s polymorphism reflects human intelligence. Alternatively, the addition of generic rules so that an activity with remote indication of an attack will trigger an alert, requires the security analyst to discern true alerts from a multitude of false alerts, thus perpetuating the original problem. The perpetuity of this trade-off issue is a dilemma that has puzzled the cyber-security community for years. A solution to this dilemma includes reducing uncertainty in intrusion analysis by making IDS-nearly-certain alerts prominently discernible. Therefore, I propose alerts prioritization, which can be attained by integrating multiple methods. I use IDS alerts correlation by building attack scenarios in a ground-up manner. In addition, I use Dempster-Shafer Theory (DST), a non-traditional theory to quantify uncertainty, and I propose a new method for fusing non-independent alerts in an attack scenario. Finally, I propose usage of semi-supervised learning to capture an organization’s contextual knowledge, consequently improving prioritization. Evaluation of these approaches was conducted using multiple datasets. Evaluation results strongly indicate that the ranking provided by the approaches gives good prioritization of IDS alerts based on their likelihood of indicating true attacks.

Les systèmes d'information actuels, qu'il s'agisse de réseaux d'entreprises, deservices en ligne ou encore d'organisations gouvernementales, reposent trèssouvent sur des systèmes distribués, impliquant un ensemble de machinesfournissant des services internes ou externes. La sécurité de tels systèmesd'information est construite à plusieurs niveaux (défense en profondeur). Lors de l'établissementde tels systèmes, des politiques de contrôle d'accès, d'authentification, defiltrage (firewalls, etc.) sont mises en place afin de garantir lasécurité des informations. Cependant, ces systèmes sont très souventcomplexes, et évoluent en permanence. Il devient alors difficile de maintenirune politique de sécurité sans faille sur l'ensemble du système (quand bienmême cela serait possible), et de résister aux attaques auxquelles ces servicessont quotidiennement exposés. C'est ainsi que les systèmes de détectiond'intrusions sont devenus nécessaires, et font partie du jeu d'outils desécurité indispensables à tous les administrateurs de systèmes exposés enpermanence à des attaques potentielles.Les systèmes de détection d'intrusions se classifient en deux grandes familles,qui diffèrent par leur méthode d'analyse: l'approche par scénarios et l'approchecomportementale. L'approche par scénarios est la plus courante, et elle estutilisée par des systèmes de détection d'intrusions bien connus tels queSnort, Prélude et d'autres. Cette approche consiste à reconnaître des signaturesd'attaques connues dans le trafic réseau (pour les IDS réseau) et des séquencesd'appels systèmes (pour les IDS hôtes). Il s'agit donc de détecter descomportements anormaux du système liés à la présence d'attaques. Bien que l'onpuisse ainsi détecter un grand nombre d'attaques, cette approche ne permet pasde détecter de nouvelles attaques, pour lesquelles aucune signature n'estconnue. Par ailleurs, les malwares modernes emploient souvent des techniquesdites de morphisme binaire, afin d'échapper à la détection parsignatures.L'approche comportementale, à l'inverse de l'approche par signature, se basesur la modélisation du fonctionnement normal du système. Cette approche permetainsi de détecter de nouvelles attaques tout comme des attaques plus anciennes,n'ayant recours à aucune base de données de connaissance d'attaques existantes.Il existe plusieurs types d'approches comportementales, certains modèles sontstatistiques, d'autres modèles s'appuient sur une politique de sécurité.Dans cette thèse, on s'intéresse à la détection d'intrusions dans des systèmesdistribués, en adoptant une approche comportementale basée sur une politique desécurité. Elle est exprimée sous la forme d'une politique de flux d'information. Les fluxd'informations sont suivis via une technique de propagation de marques (appeléeen anglais « taint marking ») appliquées sur les objets du systèmed'exploitation, directement au niveau du noyau. De telles approchesexistent également au niveau langage (par exemple par instrumentation de lamachine virtuelle Java, ou bien en modifiant le code des applications), ou encoreau niveau de l'architecture (en émulant le microprocesseur afin de tracer lesflux d'information entre les registres, pages mémoire etc.), etpermettent ainsi une analyse fine des flux d'informations. Cependant, nous avons choisi de nous placer au niveau du système d'exploitation, afin de satisfaire les objectifs suivants:• Détecter les intrusions à tous les niveaux du système, pas spécifiquement au sein d'une ou plusieurs applications.• Déployer notre système en présence d'applications natives, dont le code source n'est pas nécessairement disponible (ce qui rend leur instrumentation très difficile voire impossible).• Utiliser du matériel standard présent sur le marché. Il est très difficile de modifier physiquement les microprocesseurs, et leur émulation a un impact très important sur les performances du système. / Modern organisations rely intensively on information and communicationtechnology infrastructures. Such infrastructures offer a range of servicesfrom simple mail transport agents or blogs to complex e-commerce platforms,banking systems or service hosting, and all of these depend on distributedsystems. The security of these systems, with their increasing complexity, isa challenge. Cloud services are replacing traditional infrastructures byproviding lower cost alternatives for storage and computational power, butat the risk of relying on third party companies. This risk becomesparticularly critical when such services are used to host privileged companyinformation and applications, or customers' private information. Even in thecase where companies host their own information and applications, the adventof BYOD (Bring Your Own Device) leads to new security relatedissues.In response, our research investigated the characterization and detection ofmalicious activities at the operating system level and in distributedsystems composed of multiple hosts and services. We have shown thatintrusions in an operating system spawn abnormal information flows, and wedeveloped a model of dynamic information flow tracking, based on taintmarking techniques, in order to detect such abnormal behavior. We trackinformation flows between objects of the operating system (such as files,sockets, shared memory, processes, etc.) and network packetsflowing between hosts. This approach follows the anomaly detection paradigm.We specify the legal behavior of the system with respect to an informationflow policy, by stating how users and programs from groups of hosts areallowed to access or alter each other's information. Illegal informationflows are considered as intrusion symptoms. We have implemented this modelin the Linux kernel (the source code is availableat http://www.blare-ids.org), as a Linux Security Module (LSM), andwe used it as the basis for practical demonstrations. The experimentalresults validated the feasibility of our new intrusion detection principles.

De nos jours, les systèmes de production sont confrontés à leur 4e révolution. Celle-ci est numérique avec des réseaux toujours plus denses et complexes s’ouvrant sur l’extérieur. Cette ouverture rend ces systèmes plus vulnérables. Les menaces sur ces Systèmes Cyber-Physiques de Production (SCPP) ne sont plus seulement théoriques. L’attaque sur l’aciérie allemande ou le cryptovirus Wannacry en sont de parfaits exemples. Ce travail propose un outil contribuant à la sécurité des SCPP. Nos contributions sont triples : La conception d'un Système de Détection et Réaction aux Anomalies (SDRA) placé sur le réseau de terrain. Celui-ci intègre des méthodes de détection comportementales et informationnelles. Il comprend également des capacités de réaction à la fois passives, mettant en œuvre de la remontée d'information vers l'humain ou vers des systèmes de niveaux supérieurs, et actives intégrant du filtrage d'ordre ou de la mise en repli. L'application des méthodes proposées entraîne naturellement un effort de conception supplémentaire qui doit être réduit. Nous avons donc mis au point une démarche permettant d’assister les concepteurs pour la configuration de notre SDRA. Cette dernière se base sur une approche hybride (composant/opération) et étend un flot de conception existant. Plusieurs transformations raffinent des vues surveillance/supervision des composants alors que d’autres génèrent la configuration du SDRA. Une troisième contribution propose un démonstrateur réaliste basé sur un environnement virtuel de test. Ce dernier intègre la simulation conjointe de la partie opérative et de la partie commande et permet de montrer les qualités fonctionnelles des solutions face à des scénarios d’attaque ou de défaillance. / Today, production systems are facing their 4th revolution. This revolution is digital with increasingly dense and complex networks opening on the outside. This openness makes these systems more vulnerable. The threats on these Cyber-Physical Production Systems (CPPS) are no longer just theoretical. The attacks on the German steel mill or the Wannacry crypto virus are perfect examples. This work proposes a tool contributing to the security of the SCPP. Our contributions are threefold: The design of an Anomaly Detection and Response System (ADRS) placed on the field network. It integrates behavioral and informational detection methods. It also includes passive response capabilities, implementing feedback to the human or to higher level systems, and active integrating order filtering or fallback. The application of the proposed methods naturally entails an additional design effort which must be reduced. We have therefore developed an approach to assist designers in the configuration of our ADRS. It is based on a hybrid approach (component / operation) and extends an existing design flow. Several transformations refine monitoring / supervision views of the components while others generate the configuration of the ADRS. A third contribution proposes a realistic demonstrator based on a virtual test environment. It integrates the joint simulation of the operative part and the control part and makes it possible to show the functional qualities of the solutions in the face of attack or failure scenarios.