The key update cost is an important parameter of the performance evaluation of the secure group communications in the wireless networks. It is a very public issue to reduce the key update cost. In the tree-based multicast key management scheme, a user is randomly assigned to one of the all leaf nodes. In this thesis, we divide the users into two groups which are new call users and handoff call users. Then, we propose that new call users are assigned to some of the special leaf nodes in the key tree and the handoff call users are assigned to others. This scheme is called class-based multicast key management scheme. We analyze this two multicast key management schemes for secure group communications. This thesis shows that class-based scheme could reduce the key update cost in some special case.

In this thesis, we propose and analyze a multicast key backbone for secure group communications. We also utilize the correlated relationships between the mobile users in the wireless communications networks. When a batch member joins or leaves the group communications, the system has to update and distribute encryption keys to assure that only active members could receive the latest information. In previous tree-based multicast key management schemes, the depth of the key tree is unbounded and analytically deriving the exact value of the corresponding average update cost remains an open problem. And in previous schemes, the different mobile user arrives in and leaves from the system at different time. In contrast, the depth of the proposed multicast key backbone is fixed and the arriving or leaving users are more than one. We utilize these two characteristics and simulate the system to get the average update cost per time unit. We can find that this scheme can improve the efficiency of the system in some special cases when updating the new key.

Many mobile applications in wireless networks such as military battlefield, emergency response, and mobile commerce are based on the notion of secure group communications. Unlike traditional security protocols which concern security properties only, in this dissertation research we design and analyze a class of QoS-aware protocols for secure group communications in wireless networks with the goal to satisfy not only security requirements in terms of secrecy, confidentiality, authentication, availability and data integrity, but also performance requirements in terms of latency, network traffic, response time, scalability and reconfigurability. We consider two elements in the dissertation research: design and analysis. The dissertation research has three major contributions. First, we develop three "threshold-based" periodic batch rekeying protocols to reduce the network communication cost caused by rekeying operations to deal with outsider attacks. Instead of individual rekeying, i.e., performing a rekeying operation right after each group membership change event, these protocols perform batch rekeying periodically. We demonstrate that an optimal rekey interval exists that would satisfy an imposed security requirement while minimizing the network communication cost. Second, we propose and analyze QoS-aware intrusion detection protocols for secure group communications in mobile ad hoc networks to deal with insider attacks. We consider a class of intrusion detection protocols including host-based and voting-based protocols for detecting and evicting compromised nodes and examine their effect on the mean time to security failure metric versus the response time metric. Our analysis reveals that there exists an optimal intrusion detection interval under which the system lifetime metric can be best traded off for the response time performance metric, or vice versa. Furthermore, the intrusion detection interval can be dynamically adjusted based on the attacker behaviors to maximize the system lifetime while satisfying a system-imposed response time or network traffic requirement. Third, we propose and analyze a scalable and efficient region-based group key management protocol for managing mobile groups in mobile ad hoc networks. We take a region-based approach by which group members are broken into region-based subgroups, and leaders in subgroups securely communicate with each other to agree on a group key in response to membership change and member mobility events. We identify the optimal regional area size that minimizes the network communication cost while satisfying the application security requirements, allowing mobile groups to react to network partition/merge events for dynamic reconfigurability and survivability. We further investigate the effect of integrating QoS-aware intrusion detection with region-based group key management and identify combined optimal settings in terms of the optimal regional size and the optimal intrusion detection interval under which the security and performance properties of the system can be best optimized. We evaluate the merits of our proposed QoS-aware security protocols for mobile group communications through model-based mathematical analyses with extensive simulation validation. We perform thorough comparative analyses against baseline secure group communication protocols which do not consider security versus performance tradeoffs, including those based on individual rekeying, no intrusion detection, and/or no-region designs. The results obtained show that our proposed QoS-aware security protocols outperform these baseline algorithms. â / Ph. D.