An ad hoc network adopts a decentralized unstructured networking model that depends on node cooperation for key network functionalities such as routing and medium access. The significance of node cooperation in ad hoc networks makes network survival particularly sensitive to insider node behavior. The presence of selfish or malicious nodes in an ad hoc network could greatly degrade the network performance and might even result in a total communication breakdown. Consequently, it is important for both security and performance reasons to discourage, expose, and react to such damaging misbehavior. Reputation management systems have been proposed to mitigate against such misbehavior in ad hoc networks. The functions of a reputation management system are to evaluate nodes' quality of behavior based on their cooperation (evaluation), distinguish between well-behaved and misbehaving nodes (detection), and appropriately react to misbehaving nodes (reaction). A significant number of reputation management systems have been proposed for ad hoc networks to date. However, there has been no attempt to consolidate all current research into a formal framework for reputation management systems. The lack of a formal framework is a potential weakness of the research field. For example, a formal comparison of proposed reputation management systems has remained difficult, mainly due to the lack of a formal framework upon which the comparison could be based. There is also a lack of formal metrics that could be used for quantitative evaluation and comparison of reputation management systems. Another major shortcoming in this research field is the assumption that the functions of reputation management (evaluation, detection, and reaction) are carried out homogeneously across time and space at different nodes. The dynamic nature of ad hoc networks causes node behavior to vary spatially and temporally due to changes in the local and network-wide conditions. Reputation management functions do not adapt to such changes, which may impact the system accuracy and promptness. We herein recognize an adaptive reputation management system as one where nodes carry out the reputation management functions heterogeneously across time and space according to the instantaneous perception of each of its surrounding network conditions. In this work, we address the above concerns. We develop a formal framework for reputation management systems upon which design, evaluation, and comparison of reputation management systems can be based. We define and discuss the different components of the framework and the interactions among them. We also define formal metrics for evaluation of reputation management systems. The metrics assess both, the effectiveness (security issues) of a reputation management system in detecting misbehavior and limiting its negative impact on the network, and its efficiency (performance issues) in terms of false positives and overhead exerted by the reputation management system on the network. We also develop ARMS, an autonomous reputation management system, based on the formal framework. The theoretical foundation of ARMS is based on the theory of Sequential Probability Ratio Test introduced by Wald. In ARMS, nodes independently and without cooperation manage their reputation management system functions. We then use ARMS to investigate adaptation in reputation management systems. We discuss some of the characteristics of an adaptive reputation management system such as sensitivity, adaptability, accuracy, and promptness. We consider how the choice of evaluation metric, typically employed by the evaluation function for assessment of node behavior, may impact the sensitivity and accuracy of node behavior evaluation. We evaluate the sensitivity and accuracy of node behavior evaluation using a number of metrics from the network and medium access layer. We then introduce a time-slotted approach to enhance the sensitivity of the evaluation function and show how the duration of an evaluation slot can adapt according to the network activity to enhance the system accuracy and promptness. We also show how the detection function can adapt to the network conditions by using the node's own behavior as a benchmark to set its detection parameters. To the best of our knowledge, this is the first work to explore the adaptation of the reputation management functions in ad hoc networks. / Ph. D.

Security of communications is a major requirement for Mobile Adhoc NETworks(MANETs) since they use wireless channel for communications which can be easily tapped, and physical capture of MANET nodes is also quite easy. From the point of view of providing security in MANETs, there are basically two types of MANETs, viz., authoritarian MANETs, in which there exist one or more authorities who decide the members of the network, and self-organized MANETs, in which there is no such authority. Ensuring security of communications in the MANETs is a challenging task due to the resource constraints and infrastructure-less nature of these networks, and the limited physical security of MANET nodes. Attacks on security in a MANET can be launched by either the external attackers which are not legitimate members of the MANET or the internal attackers which are compromised members of the MANET and which can hold some valid security credentials or both. Key management and authentication protocols(KM-APs)play an important role in preventing the external attackers in a MANET. However, in order to prevent the internal attackers, an intrusion detection system(IDS) is essential. The routing protocols running in the network layer of a MANET are most vulnerable to the internal attackers, especially to the attackers which launch packet dropping attack during data packet forwarding in the MANET. For an authoritarian MANET, an arbitrated KM-AP protocol is perfectly suitable, where trusts among network members are coordinated by a trusted authority. Moreover, due to the resource constraints of a MANET, symmetric key management protocols are more efficient than the public key management protocols in authoritarian MANETs. The existing arbitrated symmetric key management protocols in MANETs, that do not use any authentication server inside the network are susceptible to identity impersonation attack during shared key establishments. On the other hand, the existing server coordinated arbitrated symmetric key management protocols in MANETs do not differentiate the role of a membership granting server(MGS) from the role of an authentication server, and so both are kept inside the network. However, keeping the MGS outside the network is more secure than keeping it inside the network for a MANET. Also, the use of a single authentication server inside the network cannot ensure robustness against authentication server compromise. In self-organized MANETs, public key management is more preferable over symmetric key management, since the distribution of public keys does not require a pre-established secure channel. The main problem for the existing self-organized public key management protocols in MANETs is associated with the use of large size certificate chains. Besides, the proactive certificate chaining based approaches require each member of a MANET to maintain an updated view of the trust graph of the entire network, which is highly resource consuming. Maintaining a hierarchy of trust relationships among members of a MANET is also problematic for the same reason. Evaluating the strength of different alternative trust chains and restricting the length of a trust chain used for public key verification is also important for enhancing the security of self-organized public key management protocols. The existing network layer IDS protocols in MANETs that try to defend against packet dropping attack use either a reputation based or an incentive based approach. The reputation based approaches are more effective against malicious principals than the incentive based approaches. The major problem associated with the existing reputation based IDS protocols is that they do not consider the protocol soundness issue in their design objectives. Besides, most of the existing protocols incorporate no mechanism to fight against colluding principals. Also, an IDS protocol in MANETs should incorporate some secure and efficient mechanism to authenticate the control packets used by it. In order to mitigate the above mentioned problems in MANETs, we have proposed new models and designed novel security protocols in this thesis that can enhance the security of communications in MANETs at lesser or comparable cost. First, in order to perform security analysis of KM-AP protocols, we have extended the well known strand space verification model to overcome some of its limitations. Second, we have proposed a model for the study of membership of principals in MANETs with a view to utilize the concept for analyzing the applicability and the performance of KM-AP protocols in different types of MANETs. Third and fourth, we have proposed two novel KM-AP protocols, SEAP and CLPKM, applicable in two different types of MANET scenarios. The SEAP protocol is an arbitrated symmetric key management protocol designed to work in an authoritarian MANET, whereas the CLPKM protocol is a self-organized public key management protocol designed for self-organized MANETs. Fifth, we have designed a novel reputation based network layer IDS protocol, named EVAACK protocol, for the detection of packet dropping misbehavior in MANETs. All of the three proposed protocols try to overcome the limitations of the existing approaches in their respective categories. We have provided rigorous mathematical proofs for the security properties of the proposed protocols. Performance of the proposed protocols have been compared with those of the other existing similar approaches using simulations in the QualNet simulator. In addition, we have also implemented the proposed SEAP and CLPKM protocols on a real MANET test bed to test their performances in real environments. The analytical, simulation and experimentation results confirm the effectiveness of the proposed schemes.