There are three mail types of Botnet: IRC-based Botnet, P2P-based Botnet,Web-based Botnet and they have become major threat to the Internet recently. Web-based Botnet is popular and more harmful to users. The architecture of Web-based Botnet is simpler than P2P-based Botnet, and its malicious traffic can be hidden in a large number of normal traffic. In this study, we built an experimental environment of using malicious bot programs to detect suspicious traffic and malware features. Except network attacking and identity theft, Botnet could also be used by hackers to extend the life time of rouge websites by combining with the technology of Fast Flux Domain. Botnet and the technology of Fast Flux Domain closely link to each other in the real world. Both of Web-based Botnet and Fast Flux Domain technology use HTTP protocol to communicate, and Botnet provides a large number of infected hosts to be Fast Flux Agents which act like a relay station to block the direct link of malicious websites from clients, but completes the mutual connection. In the research, not only the analysis and detection of Web-based Botnet are focused, but also the impact of Fast Flux Domain technology is included. We expect to clear the architecture of Botnet and the technology of Fast Flux Domain, and make the detection mechanism more precisely.

Submitted by Kaio Barbosa (kaiorafael@gmail.com) on 2018-11-23T19:03:22Z No. of bitstreams: 4 main8.pdf: 5626368 bytes, checksum: d3778f7a787ea1c33de8006e8e9f83b4 (MD5) 39 ATA de Defesa - Kaio Rafael (Assinada).pdf: 168862 bytes, checksum: 949b4b93de6b1b36821292d15ed216af (MD5) 39 Folha de Aprovação - Kaio Rafael (Assinada).pdf: 192234 bytes, checksum: e04a9d8029f559ecc1aa33e00c5e4618 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Approved for entry into archive by Secretaria PPGI (secretariappgi@icomp.ufam.edu.br) on 2018-11-23T19:09:52Z (GMT) No. of bitstreams: 4 main8.pdf: 5626368 bytes, checksum: d3778f7a787ea1c33de8006e8e9f83b4 (MD5) 39 ATA de Defesa - Kaio Rafael (Assinada).pdf: 168862 bytes, checksum: 949b4b93de6b1b36821292d15ed216af (MD5) 39 Folha de Aprovação - Kaio Rafael (Assinada).pdf: 192234 bytes, checksum: e04a9d8029f559ecc1aa33e00c5e4618 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Approved for entry into archive by Divisão de Documentação/BC Biblioteca Central (ddbc@ufam.edu.br) on 2018-11-23T20:52:12Z (GMT) No. of bitstreams: 4 main8.pdf: 5626368 bytes, checksum: d3778f7a787ea1c33de8006e8e9f83b4 (MD5) 39 ATA de Defesa - Kaio Rafael (Assinada).pdf: 168862 bytes, checksum: 949b4b93de6b1b36821292d15ed216af (MD5) 39 Folha de Aprovação - Kaio Rafael (Assinada).pdf: 192234 bytes, checksum: e04a9d8029f559ecc1aa33e00c5e4618 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Made available in DSpace on 2018-11-23T20:52:12Z (GMT). No. of bitstreams: 4 main8.pdf: 5626368 bytes, checksum: d3778f7a787ea1c33de8006e8e9f83b4 (MD5) 39 ATA de Defesa - Kaio Rafael (Assinada).pdf: 168862 bytes, checksum: 949b4b93de6b1b36821292d15ed216af (MD5) 39 Folha de Aprovação - Kaio Rafael (Assinada).pdf: 192234 bytes, checksum: e04a9d8029f559ecc1aa33e00c5e4618 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) Previous issue date: 2018-04-04 / FAPEAM - Fundação de Amparo à Pesquisa do Estado do Amazonas / Remote control and remote access of malicious code-enabled computers allow the network operator (botnet) to perform various fraudulent activities such as orchestrating distributed denial of service (DDoS) attacks or propagating malicious code such as virus and IT worms. To maintain control of these infected machines, it is necessary to use a robust communication mechanism against attempts to disrupt network services and to be able to evade intrusion detection systems. Such a mechanism is also known as Command and Control (C&C) channel. To do this, some malicious networks often adopt the Domain Name System (DNS) because of its global and distributed operation, allowing them to simulate legitimate network behaviors from techniques such as Round-Robin DNS (RRDNS) and Content Distribution Networks (CDN). Malicious networks that employ these strategies are called Fast Flow Service Networks, because they are able to modify their behavior to ensure the continuous operation of the services, as well as the Command and Control (C&C) channel. To identify such networks, current intrusion detection systems are constructed from models based on a fixed set of attributes observed at a given time point. However, the operators of these networks are able to subvert such detection models by modifying characteristics such as the number of IP addresses or the lifetime (TTL) of a domain name. For these reasons, this work presents a bioinspired model in the concept of Optimization by Colony of Ants for detection of botnets based on Fast Flow Service Networks. The main objective is to analyze a suspicious domain from different perspectives, because even if it is possible to manipulate certain features, the operator is unlikely to modify a of attributes to evade different classification models at the same time. The experimental results using a real database show that the model is able to generate classification rules that prioritize lower cost from the combination of different detection methods, obtaining an accuracy of more than 93%. / O controle e o acesso remoto de computadores infectados por códigos maliciosos permitem ao operador desse tipo de rede (botnet) realizar diferentes atividades fraudulentas como orquestrar ataques distribuídos de negação de serviço (DDoS) ou propagar códigos maliciosos como vírus e worms. Para manter o controle dessas máquinas infectadas, é necessário utilizar um mecanismo de comunicação robusto contra tentativas de interrupção dos serviços da rede e que seja capaz de evadir sistemas de detecção de intrusos. Tal mecanismo é também conhecido como canal de Comando e Controle (C&C). Para isso, algumas redes maliciosas adotam com frequência o Sistema de Nomes de Domínios (DNS) devido ao seu funcionamento global e distribuído, permitindo assim que simulem comportamentos de redes legítimas a partir de técnicas como Round-Robin DNS (RRDNS) e Redes de Distribuição de Conteúdo (CDN). Redes maliciosas que empregam essas estratégias são denominadas como Redes de Serviço de Fluxo Rápido, pois são capazes de modificar seu comportamento para garantir a operação contínua dos serviços, assim como do canal de Comando e Controle (C&C). Para identificar essas redes, os sistemas de detecção de intrusos atuais são construídos a partir de modelos baseados em um conjunto fixo de atributos observados em determinado instante de tempo. No entanto, os operadores dessas redes são capazes de subverter tais modelos de detecção pela modificação de características como a quantidade de endereços IP ou tempo de vida (TTL) de um nome de domínio. Por esses motivos, este trabalho apresenta um modelo bioinspirado no conceito de Otimização por Colônia de Formigas para detecção de botnets baseadas em Redes de Serviço de Fluxo Rápido. O principal objetivo é analisar um domínio suspeito a partir de diferentes perspectivas, pois mesmo que seja possível a manipulação de determinadas características, é improvável que o operador modifique um conjunto considerável de atributos para evadir diferentes modelos de classificação ao mesmo tempo. Os resultados experimentais usando uma base de dados real mostram que o modelo é capaz de gerar regras de classificação que priorizam menor custo a partir da combinação de diferentes métodos de detecção, obtendo uma acurácia superior a 93%.

HTTP botnets are currently the most popular form of botnets compared to IRC and P2P botnets. This is because, they are not only easier to implement, operate, and maintain, but they can easily evade detection. Likewise, HTTP botnets flows can easily be buried in the huge volume of legitimate HTTP traffic occurring in many organizations, which makes the detection harder. In this thesis, a new detection framework involving three detection models is proposed, which can run independently or in tandem. The first detector profiles the individual applications based on their interactions, and isolates accordingly the malicious ones. The second detector tracks the regularity in the timing of the bot DNS queries, and uses this as basis for detection. The third detector analyzes the characteristics of the domain names involved in the DNS, and identifies the algorithmically generated and fast flux domains, which are staples of typical HTTP botnets. Several machine learning classifiers are investigated for each of the detectors. Experimental evaluation using public datasets and datasets collected in our testbed yield very encouraging performance results. / Graduate

Botnets are generally recognized as one of the most challenging threats on the Internet today. Botnets have been involved in many attacks targeting multinational organizations and even nationwide internet services. As more effective detection and mitigation approaches are proposed by security researchers, botnet developers are employing new techniques for evasion. It is not surprising that the Domain Name System (DNS) is abused by botnets for the purposes of evasion, because of the important role of DNS in the operation of the Internet. DNS provides a flexible mapping between domain names and IP addresses, thus botnets can exploit this dynamic mapping to mask the location of botnet controllers. Domain-flux and fast-flux (also known as IP-flux) are two emerging techniques which aim at exhausting the tracking and blacklisting effort of botnet defenders by rapidly changing the domain names or their associated IP addresses that are used by the botnet. In this thesis, we employ passive DNS analysis to develop an anomaly-based technique for detecting the presence of a domain-flux or fast- flux botnet in a network. To do this, we construct a lookup graph and a failure graph from captured DNS traffic and decompose these graphs into clusters which have a strong correlation between their domains, hosts, and IP addresses. DNS related features are extracted for each cluster and used as input to a classication module to identify the presence of a domain-flux or fast-flux botnet in the network. The experimental evaluation on captured traffic traces veried that the proposed technique successfully detected domain-flux botnets in the traces. The proposed technique complements other techniques for detecting botnets through traffic analysis. / Botnets betraktas som ett av de svåraste Internet-hoten idag. Botnets har använts vid många attacker mot multinationella organisationer och även nationella myndigheters och andra nationella Internet-tjänster. Allt eftersom mer effektiva detekterings - och skyddstekniker tas fram av säkerhetsforskare, har utvecklarna av botnets tagit fram nya tekniker för att undvika upptäckt. Därför är det inte förvånande att domännamnssystemet (Domain Name System, DNS) missbrukas av botnets för att undvika upptäckt, på grund av den viktiga roll domännamnssystemet har för Internets funktion - DNS ger en flexibel bindning mellan domännamn och IP-adresser. Domain-flux och fast-flux (även kallat IP-flux) är två relativt nya tekniker som används för att undvika spårning och svartlistning av IP-adresser av botnet-skyddsmekanismer genom att snabbt förändra bindningen mellan namn och IP-adresser som används av botnets. I denna rapport används passiv DNS-analys för att utveckla en anomali-baserad teknik för detektering av botnets som använder sig av domain-flux eller fast-flux. Tekniken baseras på skapandet av en uppslagnings-graf och en fel-graf från insamlad DNS-traffik och bryter ned dessa grafer i kluster som har stark korrelation mellan de ingående domänerna, maskinerna, och IP-adresserna. DNSrelaterade egenskaper extraheras för varje kluster och används som indata till en klassifficeringsmodul för identiffiering av domain-flux och fast-flux botnets i nätet. Utvärdering av metoden genom experiment på insamlade traffikspår visar att den föreslagna tekniken lyckas upptäcka domain-flux botnets i traffiken. Genom att fokusera på DNS-information kompletterar den föreslagna tekniken andra tekniker för detektering av botnets genom traffikanalys.

This master thesis deals with the design and implementation of a tool for malware detection using DNS traffic analysis. Text of the thesis is divided into theoretical and practical part. In theoretical part the reader will be acknowledged with the domain of malware and botnet detection. Consequently, various options and methods of malware detection will be described. Practical part of the thesis contains description of malware detection tool architecture as well as key aspects of its implementation. Moreover, the emphasis is being placed on testing and experiments. The result of the thesis is a tool, written in python, for malware detection using DNS traffic analysis, that uses a combination of several methods of detection.