Domain generation algorithm (DGA) is a popular technique for evading detection used by many sophisticated malware families. Since the DGA domains are randomly generated, they tend to exhibit properties that are different from legitimate domain names. It is observed that shorter DGA domains used in emerging malware are more difficult to detect, in contrast to regular DGA domains that are unusually long. While length was considered as a contributing feature in earlier approaches, there has not been a systematic focus on how to leverage its impact on DGA domains detection accuracy. Through our study, we present a new detection model based on semantic and information theory features. The research applies concept of domain length threshold to detect DGA domains regardless of their lengths. The experimental evaluation of the proposed approach, using public datasets, yield a detection rate (DR) of 98.96% and a false positive rate (FPR) of 2.1%, when using random forests classification technique / Graduate

Malware is becoming more intelligent as static methods for blocking communication with Command and Control (C&C) server are becoming obsolete. Domain Generation Algorithms (DGAs) are a common evasion technique that generates pseudo-random domain names to communicate with C&C servers in a difficult way to detect using handcrafted methods. Trying to detect DGAs by looking at the domain name is a broad and efficient approach to detect malware-infected hosts. This gives us the possibility of detecting a wider assortment of malware compared to other techniques, even without knowledge of the malware’s existence. Our study compared the effectiveness of three different machine learning classifiers: Convolutional Neural Network (CNN), Generative Adversarial Network (GAN) and Random Forest (RF) when recognizing patterns and identifying these pseudo-random domains. The result indicates that CNN differed significantly from GAN and RF. It achieved 97.46% accuracy in the final evaluation, while RF achieved 93.89% and GAN achieved 60.39%. In the future, network traffic (efficiency) could be a key component to examine, as productivity may be harmed if the networkis over burdened by domain identification using machine learning algorithms.

La problématique des botnets, réseaux de machines infectées par des logiciels malveillants permettant de les contrôler à distance, constitue une préoccupation majeure du fait du nombre de machines infectées et des menaces associées: attaque par déni de service distribué (DDoS), spam, vol de données bancaires. Les solutions de lutte contre les botnets proposées présentent des limitations majeures dans le contexte d'un opérateur réseau (contraintes de volumétrie et de passage à l'échelle, respect de la confidentialité et de la vie privée des utilisateurs). Cette thèse propose quatre contributions orientées réseau de lutte contre les botnets. Chaque contribution traite d'une étape complémentaire dans la problématique des botnets: la première contribution permet de remonter à la source d'attaques par déni de service, et ainsi d'identifier un groupe de machines infectées à l'origine de ces attaques. La deuxième contribution concerne la détection des communications entre les machines infectées et leurs serveurs de contrôle et commande dans un réseau à large échelle, et offre ainsi l'opportunité de bloquer ces serveurs pour limiter le risque de nouvelles attaques. La troisième contribution permet une détection collaborative de botnets dans un contexte inter-domaine et inter-opérateur, permettant ainsi de lutter contre l'aspect hautement distribué de ces botnets. Enfin, la dernière contribution proposée permet de remédier aux botnets en ralentissant les communications entre les machines infectées et leur serveur de contrôle, offrant par ce biais une contre-mesure aux stratégies d'évasions développées par les cybercriminels afin de rendre leurs botnets plus résilients. / The problem of botnets, networks of infected hosts controlled remotely by attackers, is a major concern because of the number of infected hosts and associated threats, like distributed denial of service (DDoS), spams, and data theft. State of the art solutions to fight against botnets have major limitations in a context of a network operator (scalability of the solution, confidentiality and privacy of users). In this thesis, we propose four network-based contributions to fight against botnets. Each solution address a different and complementary issue in this area: the first contribution tracebacks the source of denial of service attacks which threaten the network availability, allowing by that way to identify infected devices used to perpetrate these attacks. The second contribution detects the communications between infected computers and their command and control server (C&C) in a large scale network and offers the opportunity to block these servers to minimize the risk of future attacks. The third contribution enables collaborative detection of botnets in an inter-domain and inter-operator context in order to fight against the highly distributed aspect of these botnets. Finally, the last contribution mitigates botnets by slowing down the communication between infected hosts and their C&C server, providing a countermeasure against evasion techniques developed by cybercriminals to make their botnets more resilient