Network access control (NAC) systems have a very important role in network security. However, NAC policy configuration is an extremely complicated and error-prone task due to the semantic complexity of NAC policies and the large number of rules that could exist. This significantly increases the possibility of policy misconfigurations and network vulnerabilities. NAC policy misconfigurations jeopardize network security and can result in a severe consequence such as reachability and denial of service problems. In this thesis, we choose to study and analyze the NAC policy configuration of two significant network security devices, namely, firewall and IDS/IPS. In the first part of the thesis, a visualization technique is proposed to visualize firewall rules and policies to efficiently enhance the understanding and inspection of firewall configuration. This is implemented in a tool called PolicyVis. Our tool helps the user to answer general questions such as ‘‘Does this policy satisfy my connection/security requirements’’. If not, the user can detect all misconfigurations in the firewall policy. In the second part of the thesis, we study various policy misconfigurations of Snort, a very popular IDS/IPS. We focus on the misconfigurations of the flowbits option which is one of the most important features to offers a stateful signature-based NIDS. We particularly concentrate on a class of flowbits misconfiguration that makes Snort susceptible to false negatives. We propose a method to detect the flowbits misconfiguration, suggest practical solutions with controllable false positives to fix the misconfiguration and formally prove that the solutions are complete and sound.

A variety of intrusion prevention techniques, such as user authentication (e.g.: using passwords), avoidance of programming errors, and information protection, have been used to protect computer systems. However, intrusion prevention alone is not sufficient to protect our systems, as those systems become ever more complex with the rapid growth and expansion of Internet technology and local network systems. Moreover, programming errors, firewall configuration errors, and ambiguous or undefined security policies add to the system’s complexity. An Intrusion Detection System (IDS) is therefore needed as another layer to protect computer systems. The IDS is one of the most important techniques of information dynamic security technology. It is defined as a process of monitoring the events occurring in a computer system or network and analyzing them to differentiate between normal activities of the system and behaviours that can be classified as suspicious or intrusive. Current Intrusion Detection Systems have several known shortcomings, such as: low accuracy (registering high False Positives and False Negatives); low real-time performance (processing a large amount of traffic in real time); limited scalability (storing a large number of user profiles and attack signatures); an inability to detect new attacks (recognizing new attacks when they are launched for the first time); and weak system-reactive capabilities (efficiency of response). This makes the area of IDS an attractive research field. In recent years, researchers have investigated techniques such as artificial intelligence, autonomous agents, and distributed systems for detecting intrusion in network environments. This thesis presents a novel IDS distributed architecture – Collaborative Distributed Intrusion Detection System (C-dIDS), based on lightweight IDS modules – that integrates two main concepts in order to improve IDS performance and the scalability: lightweight IDS and collaborative architecture. To accomplish the first concept, lightweight IDS, we apply two different approaches: a features selection approach and an IDS classification scheme. In the first approach, each detector (IDS module) uses smaller amounts of data in the detection process by applying a novel features selection approach called the Fuzzy Enhanced Support Vector Decision Function (Fuzzy ESVDF). This approach improves the system scalability in terms of reducing the number of needed features without degrading the overall system performance. The second approach uses a new IDS classification scheme. The proposed IDS classification scheme employs multiple specialized detectors in each layer of the TCP/IP network model. This helps collecting efficient and useful information for dIDS, increasing the system’s ability to detect different attack types and reducing the system’s scalability. The second concept uses a novel architecture for dIDS called Collaborative Distributed Intrusion Detection System (C-dIDS) to integrate these different specialized detectors (IDS modules) that are distributed on different points in the network. This architecture is a single-level hierarchy dIDS with a non-central analyzer. To make the detection decision for a specific IDS module in the system, this module must collaborate with the previous IDS module (host) in the lower level of the hierarchy only. Collaborating with other IDS modules improves the overall system accuracy without creating a heavy system overload. Also, this architecture avoids both single point of failure and scalability bottleneck problems. Integration of the two main concepts, lightweight IDS and a distributed collaborative architecture, has shown very good results and has addressed many IDS limitations.

Intrusion detection systems (IDSes) are an important security measure that network administrators adopt to defend computer networks against malicious attacks and intrusions. The field of IDS research includes many challenges. However, one open problem remains orthogonal to the others: IDS evaluation. In other words, researchers have not yet succeeded to agree on a general systematic methodology and/or a set of metrics to fairly evaluate different IDS algorithms. This leads to another problem: the lack of an appropriate IDS evaluation dataset that satisfies the common research needs. One major contribution in this area is the DARPA dataset offered by the Massachusetts Institute of Technology Lincoln Lab (MIT/LL), which has been extensively used to evaluate a number of IDS algorithms proposed in the literature. Despite this, the DARPA dataset received a lot of criticism concerning the way it was designed, especially concerning its obsoleteness and inability to incorporate new sorts of network attacks. In this thesis, we survey previous research projects that attempted to provide a system for IDS offline evaluation. From the survey, we identify a set of design requirements for such a system based on the research community needs. We, then, propose Algorizmi as an open-source configurable virtual testbed for generating datasets for offline IDS evaluation. We provide an architectural overview of Algorizmi and its software and hardware components. Algorizmi provides its users with tools that allow them to create their own experimental testbed using the concepts of virtualization and cloud computing. Algorizmi users can configure the virtual machine instances running in their experiments, select what background traffic those instances will generate and what attacks will be launched against them. At any point in time, an Algorizmi user can generate a dataset (network traffic trace) for any of her experiments so that she can use this dataset afterwards to evaluate an IDS the same way the DARPA dataset is used. Our analysis shows that Algorizmi satisfies more requirements than previous research projects that target the same research problem of generating datasets for IDS offline evaluation. Finally, we prove the utility of Algorizmi by building a sample network of machines, generate both background and attack traffic within that network. We then download a snapshot of the dataset for that experiment and run it against Snort IDS. Snort successfully detected the attacks we launched against the sample network. Additionally, we evaluate the performance of Algorizmi while processing some of the common usages of a typical user based on 5 metrics: CPU time, CPU usage, memory usage, network traffic sent/received and the execution time.

We present a new detection model include monitoring network perimeter and hosts logs to counter the new method of attacking involve different hosts source during an attacking sequence. The new attacking sequence we called ¡§Scout and Intruder¡¨ involve two separate hosts. The scout will scan and evaluate the target area to find the possible victims and their vulnerability, and the intruder launch the precision strike with login activities looked as same as authorized users. By launching the scout and assassin attack, the attacker could access the system without being detected by the network and system intrusion detection system. In order to detect the Scout and intruder attack, we correlate the netflow connection records, the system logs and network data dump, by finding the states of the attack and the corresponding features we create the detection model using the Hidden Markov Chain. With the model we created, we could find the potential Scout and the Intruder attack in the initial state, which gives the network/system administrator more response time to stop the attack from the attackers.

Cyber space is like a society. Attacking events happen all the time. No matter what is in the cyber space. We need to do many things to defend our computers and network devices form attackers, for example: update patches, install anti-virus software, firewalls and intrusion detection system. In all kinds of network attacks, it is hard to detect that an attacker install a backdoor after he crack the system. He can do many things by the backdoor, like steal sensitive or secret information. Otherwise, intrusion detection systems are responsible for early warnings, but they usually need to capture all the network packets include the headers and contents to analyze. It costs many overheads for the system. The goal of our research is to detect backdoors correctly, and we only use the network packet headers to analyze.

Exploit codes based on system vulnerabilities are often used by attackers to attack target computers or services. Such exploit programs often send attack packets in the first few packets right after a connection established with the target machine or service. And such attacks are often launched via Telnet service as well. A lightweight network-based intrusion detection system is proposed on detecting such attacks on Telnet traffic. The proposed system filters the first a few packets after each Telnet connection established and only uses partial data of a packet rather than total of it to detect intrusion, i.e. such design makes system load reduced a lot. This research is anomaly detection. The proposed system characterizes the normal traffic behavior and constructs it as a normal model based on the filtered normal traffic. In detection phase, the system examines the deviation of current filtered packet from the normal model via an anomaly score function, i.e. a more deviate packet will receive a higher anomaly score. Finally, we use 1999 DARPA Intrusion Detection Evaluation Data Set which contains 5 days of training data and 10 days of testing data, and 44 attack instances of 16 types of attacks, to evaluate our proposed system. The proposed system has the detection rate of 73% under a low false alarm rate of 2 false alarms per day; 80% for the hard detected attacks which are poorly detected in 1999 DARPA IDEP.

With the improvement of computer technologies and the wide use of the Internet, network security becomes more and more significant. According to the relevant statistics, malicious codes such as virus, worms, backdoors, and Trojans launch a lot of attacks. Backdoors are especially critical. Not only can it cross firewalls and antivirus software but also will steal confidential information and misuse network resources and launch attacks such as DDoS¡]Distributed Denial of Service¡^. In this research, we analyze the properties and categories of backdoors and the application of data mining and support vector machines in intrusion detection. This research will focus on detecting the behavior of backdoor connection, and we propose a detecting architecture. The architecture is based on SVM, which is a machine learning method based on statistic theory and proposed by Vapnik to solve the problems in Neural Network techniques. In system modules, this research chooses IPAudit as our network monitor and libsvm as a SVM classifier. The packets captured by IPAudit will be classified into interactive or non-interactive flow by libsvm, and the result will be compared with legal service lists to determine whether a connection is a backdoor connection. We compare the accuracy of SVM, C4.5, and Na

The number of significant security incidents tends to increase day by day in recent years. The distributed denial of service attacks and worm attacks extensively influence the network and cause serious damages. In the thesis, we analyze these two critical distributed attacks. We propose an intrusion detection approach against this kind of attacks and implement an attack detection system based on the approach. We use anomaly detection of intrusion detecting techniques and observed the anomalous distribution of packet fields to perform the detection. The proposed approach records the characteristics of normal traffic volumes so that to make detections more flexible and more precise. Finally, we evaluated our approach by experiments.

Most existent protocols, applications and services for Mobile Ad Hoc NET-works (MANETs) assume a cooperative and friendly network environment and do not accommodate security. Therefore, Intrusion Detection Systems (IDSs), serving as the second line of defense for information systems, are indispensable for MANETs with high security requirements. Central to the research described in this dissertation is the proposed two-level nonoverlapping Zone-Based Intrusion Detection System (ZBIDS) which fit the unique requirement of MANETs. First, in the low-level of ZBIDS, I propose an intrusion detection agent model and present a Markov Chain based anomaly detection algorithm. Local and trusted communication activities such as routing table related features are periodically selected and formatted with minimum errors from raw data. A Markov Chain based normal profile is then constructed to capture the temporal dependency among network activities and accommodate the dynamic nature of raw data. A local detection model aggregating abnormal behaviors is constructed to reflect recent subject activities in order to achieve low false positive ratio and high detection ratio. A set of criteria to tune parameters is developed and the performance trade-off is discussed. Second, I present a nonoverlapping Zone-based framework to manage locally generated alerts from a wider area. An alert data model conformed to the Intrusion Detection Message Exchange Format (IDMEF) is presented to suit the needs of MANETs. Furthermore, an aggregation algorithm utilizing attribute similarity from alert messages is proposed to integrate security related information from a wider area. In this way, the gateway nodes of ZBIDS can reduce false positive ratio, improve detection ratio, and present more diagnostic information about the attack. Third, MANET IDSs need to consider mobility impact and adjust their behavior dynamically. I first demonstrate that nodes?? moving speed, a commonly used parameter in tuning IDS performance, is not an effective metric for the performance measurement of MANET IDSs. A new feature -link change rate -is then proposed as a unified metric for local MANET IDSs to adaptively select normal profiles . Different mobility models are utilized to evaluate the performance of the adaptive mechanisms.

The intrusion detection in computer networks is a complex research problem, which requires the understanding of computer networks and the mechanism of intrusions, the configuration of sensors and the collected data, the selection of the relevant attributes, and the monitor algorithms for online detection. It is critical to develop general methods for data dimension reduction, effective monitoring algorithms for intrusion detection, and means for their performance improvement. This dissertation is motivated by the timely need to develop statistics-based machine learning methods for effective detection of computer network anomalies.Three fundamental research issues related to data dimension reduction, control charts design and performance improvement have been addressed accordingly. The major research activities and corresponding contributions are summarized as follows:(1) Filter and Wrapper models are integrated to extract a small number of the informative attributes for computer network intrusion detection. A two-phase analyses method is proposed for the integration of Filter and Wrapper models. The proposed method has successfully reduced the original 41 attributes to 12 informative attributes while increasing the accuracy of the model. The comparison of the results in each phase shows the effectiveness of the proposed method.(2) Supervised kernel based control charts for anomaly intrusion detection. We propose to construct control charts in a feature space. The first contribution is the use of multi-objective Genetic Algorithm in the parameter pre-selection for SVM based control charts. The second contribution is the performance evaluation of supervised kernel based control charts.(3) Unsupervised kernel based control charts for anomaly intrusion detection. Two types of unsupervised kernel based control charts are investigated: Kernel PCA control charts and Support Vector Clustering based control charts. The applications of SVC based control charts on computer networks audit data are also discussed to demonstrate the effectiveness of the proposed method.Although the developed methodologies in this dissertation are demonstrated in the computer network intrusion detection applications, the methodologies are also expected to be applied to other complex system monitoring, where the database consists of a large dimensional data with non-Gaussian distribution.