Denial-of-service attacks (DoS) and distributed denial-of-service attacks (DDoS) attempt to temporarily disrupt users or computer resources to cause service un- availability to legitimate users in the internetworking system. The most common type of DoS attack occurs when adversaries °ood a large amount of bogus data to interfere or disrupt the service on the server. The attack can be either a single-source attack, which originates at only one host, or a multi-source attack, in which multiple hosts coordinate to °ood a large number of packets to the server. Cryptographic mechanisms in authentication schemes are an example ap- proach to help the server to validate malicious tra±c. Since authentication in key establishment protocols requires the veri¯er to spend some resources before successfully detecting the bogus messages, adversaries might be able to exploit this °aw to mount an attack to overwhelm the server resources. The attacker is able to perform this kind of attack because many key establishment protocols incorporate strong authentication at the beginning phase before they can iden- tify the attacks. This is an example of DoS threats in most key establishment protocols because they have been implemented to support con¯dentiality and data integrity, but do not carefully consider other security objectives, such as availability. The main objective of this research is to design denial-of-service resistant mechanisms in key establishment protocols. In particular, we focus on the design of cryptographic protocols related to key establishment protocols that implement client puzzles to protect the server against resource exhaustion attacks. Another objective is to extend formal analysis techniques to include DoS- resistance. Basically, the formal analysis approach is used not only to analyse and verify the security of a cryptographic scheme carefully but also to help in the design stage of new protocols with a high level of security guarantee. In this research, we focus on an analysis technique of Meadows' cost-based framework, and we implement DoS-resistant model using Coloured Petri Nets. Meadows' cost-based framework is directly proposed to assess denial-of-service vulnerabil- ities in the cryptographic protocols using mathematical proof, while Coloured Petri Nets is used to model and verify the communication protocols using inter- active simulations. In addition, Coloured Petri Nets are able to help the protocol designer to clarify and reduce some inconsistency of the protocol speci¯cation. Therefore, the second objective of this research is to explore vulnerabilities in existing DoS-resistant protocols, as well as extend a formal analysis approach to our new framework for improving DoS-resistance and evaluating the performance of the new proposed mechanism. In summary, the speci¯c outcomes of this research include following results; 1. A taxonomy of denial-of-service resistant strategies and techniques used in key establishment protocols; 2. A critical analysis of existing DoS-resistant key exchange and key estab- lishment protocols; 3. An implementation of Meadows's cost-based framework using Coloured Petri Nets for modelling and evaluating DoS-resistant protocols; and 4. A development of new e±cient and practical DoS-resistant mechanisms to improve the resistance to denial-of-service attacks in key establishment protocols.

Wireless networking introduces a whole range of challenges to the traditional TCP/IP network. In particular, IP address the issue of overloading because IP addresses are used as a network locator and an end point identity in the different layers in an OSI model. Even though Mobile IP is widely deployed, it has significant problems relating to performance and security. The Host Identity Protocol (HIP) provides secure mobility management by solving the IP address overloading from another angle. It restructures the TCP/IP model and introduces a new layer and a new namespace. The performance of HIP has proven to be better than Mobile IP and also opens a range of new research opportunities. This dissertation proposes and analyses a new step-stone solution from the Mobile IP-based network into a HIP-based network. The main advantage of this new solution is that much less change is required to the operating system kernel of the end point compared to a full HIP implementation. The new step-stone solution allows Mobile IP to use some HIP features to provide better security and handover performance. This dissertation also proposes several new and novel HIP-based wireless communication network architectures. An HIP-based heterogeneous wireless network architecture and handover scheme has been proposed and analysed. These schemes limit the HIP signalling in the wireless network if no communication to external networks is needed. Beside the network architecture modification, the hybrid Session Initial Protocol (SIP) and HIP-based Voice over IP (VoIP) scheme is proposed and analysed. This novel scheme improves the handover latency and security. This dissertation also proposes and analyses a new and novel extension to HIP, a HIP-based micro-mobility management, micro-HIP (mHIP). mHIP provides a new secure framework for micro-mobility management. It is a more complete HIP-based micro-mobility solution than any other proposed in existing studies. mHIP improves the intra-domain handover performance, the security, and the distribution of load in the intra-domain handover signalling. The new work presented opens up a number of very interesting research opportunities.