This research investigates the denial of service problem, in the context of services provided over a network, and contributes to improved techniques for modelling, detecting, and preventing denial of service attacks against these services. While the majority of currently employed denial of service attacks aim to pre-emptively consume the network bandwidth of victims, a significant amount of research effort is already being directed at this problem. This research is instead concerned with addressing the inevitable migration of denial of service attacks up the protocol stack to the application layer. Of particular interest is the denial of service resistance of key establishment protocols (security protocols that enable an initiator and responder to mutually authenticate and establish cryptographic keys for establishing a secure communications channel), which owing to the computationally intensive activities they perform, are particularly vulnerable to attack. Given the preponderance of wireless networking technologies this research hasalso investigated denial of service and its detection in IEEE 802.11 standards based networks. Specific outcomes of this research include: - investigation of the modelling and application of techniques to improve the denial of service resistance of key establishment protocols; - a proposal for enhancements to an existing modelling framework to accommodate coordinated attackers; - design of a new denial of service resistant key establishment protocol for securing signalling messages in next generation, mobile IPv6 networks; - a comprehensive survey of denial of service attacks in IEEE 802.11 wireless networks; discovery of a significant denial of service vulnerability in the clear channel assessment procedure implemented by the medium access control layer of IEEE 802.11 compliant devices; and - design of a novel, specification-based intrusion detection system for detecting denial of service attacks in IEEE 802.11 wireless networks.

Denial-of-service attacks (DoS) and distributed denial-of-service attacks (DDoS) attempt to temporarily disrupt users or computer resources to cause service un- availability to legitimate users in the internetworking system. The most common type of DoS attack occurs when adversaries °ood a large amount of bogus data to interfere or disrupt the service on the server. The attack can be either a single-source attack, which originates at only one host, or a multi-source attack, in which multiple hosts coordinate to °ood a large number of packets to the server. Cryptographic mechanisms in authentication schemes are an example ap- proach to help the server to validate malicious tra±c. Since authentication in key establishment protocols requires the veri¯er to spend some resources before successfully detecting the bogus messages, adversaries might be able to exploit this °aw to mount an attack to overwhelm the server resources. The attacker is able to perform this kind of attack because many key establishment protocols incorporate strong authentication at the beginning phase before they can iden- tify the attacks. This is an example of DoS threats in most key establishment protocols because they have been implemented to support con¯dentiality and data integrity, but do not carefully consider other security objectives, such as availability. The main objective of this research is to design denial-of-service resistant mechanisms in key establishment protocols. In particular, we focus on the design of cryptographic protocols related to key establishment protocols that implement client puzzles to protect the server against resource exhaustion attacks. Another objective is to extend formal analysis techniques to include DoS- resistance. Basically, the formal analysis approach is used not only to analyse and verify the security of a cryptographic scheme carefully but also to help in the design stage of new protocols with a high level of security guarantee. In this research, we focus on an analysis technique of Meadows' cost-based framework, and we implement DoS-resistant model using Coloured Petri Nets. Meadows' cost-based framework is directly proposed to assess denial-of-service vulnerabil- ities in the cryptographic protocols using mathematical proof, while Coloured Petri Nets is used to model and verify the communication protocols using inter- active simulations. In addition, Coloured Petri Nets are able to help the protocol designer to clarify and reduce some inconsistency of the protocol speci¯cation. Therefore, the second objective of this research is to explore vulnerabilities in existing DoS-resistant protocols, as well as extend a formal analysis approach to our new framework for improving DoS-resistance and evaluating the performance of the new proposed mechanism. In summary, the speci¯c outcomes of this research include following results; 1. A taxonomy of denial-of-service resistant strategies and techniques used in key establishment protocols; 2. A critical analysis of existing DoS-resistant key exchange and key estab- lishment protocols; 3. An implementation of Meadows's cost-based framework using Coloured Petri Nets for modelling and evaluating DoS-resistant protocols; and 4. A development of new e±cient and practical DoS-resistant mechanisms to improve the resistance to denial-of-service attacks in key establishment protocols.