Information security communities are always talking about "attackers" or "blackhats", but in reality very little is known about their skills. The idea of studying attacker behaviors was pioneered in the early nineties. In the last decade the number of attacks has increased exponentially and honeypots were introduced in order to gather information about attackers and to develop early-warning systems. Honeypots come in different flavors with respect to their interaction potential. A honeypot can be very restrictive, but this implies only a few interactions. However, if a honeypot is very tolerant, attackers can quickly achieve their goal. Choosing the best trade-off between attacker freedom and honeypot restrictions is challenging. In this dissertation, we address the issue of self-adaptive honeypots that can change their behavior and lure attackers into revealing as much information as possible about themselves. Rather than being allowed simply to carry out attacks, attackers are challenged by strategic interference from adaptive honeypots. The observation of the attackers' reactions is particularly interesting and, using derived measurable criteria, the attacker's skills and capabilities can be assessed by the honeypot operator. Attackers enter sequences of inputs on a compromised system which is generic enough to characterize most attacker behaviors. Based on these principles, we formally model the interactions of attackers with a compromised system. The key idea is to leverage game-theoretic concepts to define the configuration and reciprocal actions of high-interaction honeypots. We have also leveraged machine learning techniques for this task and have developed a honeypot that uses a variant of reinforcement learning in order to arrive at the best behavior when facing attackers. The honeypot is capable of adopting behavioral strategies that vary from blocking commands or returning erroneous messages, right up to insults that aim to irritate the intruder and serve as a reverse Turing Test distinguishing human attackers from machines. Our experimental results show that behavioral strategies are dependent on contextual parameters and can serve as advanced building blocks for intelligent honeypots. The knowledge obtained can be used either by the adaptive honeypots themselves or to reconfigure low-interaction honeypots.

Les SIEMs (systèmes pour la Sécurité de l’Information et la Gestion des Événements) sont les cœurs des centres opérationnels de la sécurité. Ils corrèlent un nombre important d’événements en provenance de différents capteurs (anti-virus, pare-feux, systèmes de détection d’intrusion, etc), et offrent des vues synthétiques pour la gestion des menaces ainsi que des rapports de sécurité. La gestion et l’analyse de ce grand nombre d’alertes est une tâche difficile pour l’administrateur de sécurité. La corrélation d’alertes a été conçue afin de remédier à ce problème. Des solutions de corrélation ont été développées pour obtenir une vue plus concise des alertes générées et une meilleure description de l’attaque détectée. Elles permettent de réduire considérablement le volume des alertes remontées afin de soutenir l’administrateur dans le traitement de ce grand nombre d’alertes. Malheureusement, ces techniques ne prennent pas en compte les connaissances sur le comportement de l’attaquant, les fonctionnalités de l’application et le périmètre de défense du réseau supervisé (pare-feu, serveurs mandataires, Systèmes de détection d’intrusions, etc). Dans cette thèse, nous proposons deux nouvelles approches de corrélation d’alertes. La première approche que nous appelons corrélation d’alertes basée sur les pots de miel utilise des connaissances sur les attaquants recueillies par le biais des pots de miel. La deuxième approche de corrélation est basée sur une modélisation des points d’application de politique de sécurité / Security Information and Event Management (SIEM) systems provide the security analysts with a huge amount of alerts. Managing and analyzing such tremendous number of alerts is a challenging task for the security administrator. Alert correlation has been designed in order to alleviate this problem. Current alert correlation techniques provide the security administrator with a better description of the detected attack and a more concise view of the generated alerts. That way, it usually reduces the volume of alerts in order to support the administrator in tackling the amount of generated alerts. Unfortunately, none of these techniques consider neither the knowledge about the attacker’s behavior nor the enforcement functionalities and the defense perimeter of the protected network (Firewalls, Proxies, Intrusion Detection Systems, etc). It is still challenging to first improve the knowledge about the attacker and second to identify the policy enforcement mechanisms that are capable to process generated alerts. Several authors have proposed different alert correlation methods and techniques. Although these approaches support the administrator in processing the huge number of generated alerts, they remain limited since these solutions do not provide us with more information about the attackers’ behavior and the defender’s capability in reacting to detected attacks. In this dissertation, we propose two novel alert correlation approaches. The first approach, which we call honeypot-based alert correlation, is based on the use of knowledge about attackers collected through honeypots. The second approach, which we call enforcement-based alert correlation, is based on a policy enforcement and defender capabilities’ model