The malicious intrusions to computer systems result in the loss of money, time and hidden information which require deployment of intrusion detection systems. Existing intrusion detection methods analyze packet payload to search for certain strings and to match them with a rule database which takes a long time in large size packets. Because of buffer limits, packets may be dropped or the system may stop working due to high CPU load. In this thesis, we investigate signature based intrusion detection with signatures that only depend on the packet header information without payload inspection. To this end, we analyze the well-known DARPA 1998 dataset to manually extract such signatures and construct a new rule set to detect the intrusions. We implement our rule set in a popular intrusion detection software tool, Snort. Furthermore we enhance our rule set with the existing rules of Snort which do not depend on payload inspection. We test our rule set on DARPA data set as well as a new data set that we collect using attack generator tools. Our results show around 30% decrease in detection time with a tolerable decrease in the detection rate. We believe that our method can be used as a complementary component to speed up intrusion detection systems.

Buffer overflow vulnerabilities continue to prevail and the sophistication of attacks targeting these vulnerabilities is continuously increasing. As a successful attack of this type has the potential to completely compromise the integrity of the targeted host, early detection is vital. This thesis examines generic approaches for detecting executable payload attacks, without prior knowledge of the implementation of the attack, in such a way that new and previously unseen attacks are detectable. Executable payloads are analysed in detail for attacks targeting the Linux and Windows operating systems executing on an Intel IA-32 architecture. The execution flow of attack payloads are analysed and a generic model of execution is examined. A novel classification scheme for executable attack payloads is presented which allows for characterisation of executable payloads and facilitates vulnerability and threat assessments, and intrusion detection capability assessments for intrusion detection systems. An intrusion detection capability assessment may be utilised to determine whether or not a deployed system is able to detect a specific attack and to identify requirements for intrusion detection functionality for the development of new detection methods. Two novel detection methods are presented capable of detecting new and previously unseen executable attack payloads. The detection methods are capable of identifying and enumerating the executable payload’s interactions with the operating system on the targeted host at the time of compromise. The detection methods are further validated using real world data including executable payload attacks.

In recent years network intrusions have become a severe threat to the privacy and safety of computer users. Recent cyber attacks compromise a large number of hosts to form botnets. Hackers not only aim at harvesting private data and identity information from compromised nodes, but also use the compromised nodes to launch attacks such as distributed denial-of-service (DDoS) attacks. As a counter measure, Intrusion Detection Systems (IDS) are used to identify intrusions by comparing observable behavior against suspicious patterns. Traditional IDSs monitor computer activities on a single host or network traffic in a sub-network. They do not have a global view of intrusions and are not effective in detecting fast spreading attacks, unknown, or new threats. In turn, they can achieve better detection accuracy through collaboration. An Intrusion Detection Network (IDN) is such a collaboration network allowing IDSs to exchange information with each other and to benefit from the collective knowledge and experience shared by others. IDNs enhance the overall accuracy of intrusion assessment as well as the ability to detect new intrusion types. Building an effective IDN is however a challenging task. For example, adversaries may compromise some IDSs in the network and then leverage the compromised nodes to send false information, or even attack others in the network, which can compromise the efficiency of the IDN. It is, therefore, important for an IDN to detect and isolate malicious insiders. Another challenge is how to make efficient intrusion detection assessment based on the collective diagnosis from other IDSs. Appropriate selection of collaborators and incentive-compatible resource management in support of IDSs' interaction with others are also key challenges in IDN design. To achieve efficiency, robustness, and scalability, we propose an IDN architecture and especially focus on the design of four of its essential components, namely, trust management, acquaintance management, resource management, and feedback aggregation. We evaluate our proposals and compare them with prominent ones in the literature and show their superiority using several metrics, including efficiency, robustness, scalability, incentive-compatibility, and fairness. Our IDN design provides guidelines for the deployment of a secure and scalable IDN where effective collaboration can be established between IDSs.

In recent years network intrusions have become a severe threat to the privacy and safety of computer users. Recent cyber attacks compromise a large number of hosts to form botnets. Hackers not only aim at harvesting private data and identity information from compromised nodes, but also use the compromised nodes to launch attacks such as distributed denial-of-service (DDoS) attacks. As a counter measure, Intrusion Detection Systems (IDS) are used to identify intrusions by comparing observable behavior against suspicious patterns. Traditional IDSs monitor computer activities on a single host or network traffic in a sub-network. They do not have a global view of intrusions and are not effective in detecting fast spreading attacks, unknown, or new threats. In turn, they can achieve better detection accuracy through collaboration. An Intrusion Detection Network (IDN) is such a collaboration network allowing IDSs to exchange information with each other and to benefit from the collective knowledge and experience shared by others. IDNs enhance the overall accuracy of intrusion assessment as well as the ability to detect new intrusion types. Building an effective IDN is however a challenging task. For example, adversaries may compromise some IDSs in the network and then leverage the compromised nodes to send false information, or even attack others in the network, which can compromise the efficiency of the IDN. It is, therefore, important for an IDN to detect and isolate malicious insiders. Another challenge is how to make efficient intrusion detection assessment based on the collective diagnosis from other IDSs. Appropriate selection of collaborators and incentive-compatible resource management in support of IDSs' interaction with others are also key challenges in IDN design. To achieve efficiency, robustness, and scalability, we propose an IDN architecture and especially focus on the design of four of its essential components, namely, trust management, acquaintance management, resource management, and feedback aggregation. We evaluate our proposals and compare them with prominent ones in the literature and show their superiority using several metrics, including efficiency, robustness, scalability, incentive-compatibility, and fairness. Our IDN design provides guidelines for the deployment of a secure and scalable IDN where effective collaboration can be established between IDSs.