In the world of high mobility, there is a growing need for people to communicate with each other and have timely access to information regardless of the location of the individuals or the information. This need is supported by the advances in the technologies of networking, wireless communications, and portable computing devices with reduction in the physical size of computers, lead to the rapid development in mobile communication infrastructure. Hence, mobile and wireless networks present many challenges to application, hardware, software and network designers and implementers. One of the biggest challenge is to provide a secure mobile environment. Security plays a more important role in mobile communication systems than in systems that use wired communication. This is mainly because of the ubiquitous nature of the wireless medium that makes it more susceptible to security attacks than wired communications. The aim of the thesis is to develop an integrated dynamic security and authentication system for mobile transactions. The proposed system operates at the transactions-level of a mobile application, by intelligently selecting the suitable security technique and authentication protocol for ongoing transaction. To do this, we have designed two schemes: the transactions-based security selection scheme and the transactions-based authentication selection scheme. These schemes use transactions sensitivity levels and the usage context, which includes users behaviors, network used, device used, and so on, to decide the required security and authentication levels. Based on this analysis, requisite security technique, and authentication protocols are applied for the trans-action in process. The Behaviors-Observations-Beliefs (BOB) model is developed using cognitive agents to supplement the working of the security and authentication selection schemes. A transaction classification model is proposed to classify the transactions into various sensitivity levels. The BOB model The BOB model is a cognitive theory based model, to generate beliefs over a user, by observing various behaviors exhibited by a user during transactions. The BOB model uses two types of Cognitive Agents (CAs), the mobile CAs (MCAs) and the static CAs (SCAs). The MCAs are deployed over the client devices to formulate beliefs by observing various behaviors of a user during the transaction execution. The SCA performs belief analysis, and identifies the belief deviations w.r.t. established beliefs. We have developed four constructs to implement the BOB model, namely: behaviors identifier, observations generator, beliefs formulator, and beliefs analyser. The BOB model is developed by giving emphasis on using the minimum computation and minimum code size, by keeping the resource restrictiveness of the mobile devices and infrastructure. The knowledge organisation using cognitive factors, helps in selecting the rational approach for deciding the legitimacy of a user or a session. It also reduces the solution search space by consolidating the user behaviors into an high-level data such as beliefs, as a result the decision making time reduces considerably. The transactions classification model This model is proposed to classify the given set of transactions of an application service into four sensitivity levels. The grouping of transactions is based on the operations they perform, and the amount of risk/loss involved if they are misused. The four levels are namely, transactions who’s execution may cause no-damage (level-0), minor-damage (level-1), significant-damage (level-2) and substantial-damage (level-3). A policy-based transaction classifier is developed and incorporated in the SCA to decide the transaction sensitivity level of a given transaction. Transactions-based security selection scheme (TBSS-Scheme) The traditional security schemes at application-level are either session or transaction or event based. They secure the application-data with prefixed security techniques on mobile transactions or events. Generally mobile transactions possesses different security risk profiles, so, empirically we may find that there is a need for various levels of data security schemes for the mobile communications environment, which face the resource insufficiency in terms of bandwidth, energy, and computation capabilities. We have proposed an intelligent security techniques selection scheme at the application-level, which dynamically decides the security technique to be used for a given transaction in real-time. The TBSS-Scheme uses the BOB model and transactions classification model, while deciding the required security technique. The selection is purely based on the transaction sensitivity level, and user behaviors. The Security techniques repository is used in the proposed scheme, organised under three levels based on the complexity of security techniques. The complexities are decided based on time and space complexities, and the strength of the security technique against some of the latest security attacks. The credibility factors are computed using the credibility module, over transaction network, and transaction device are also used while choosing the security technique from a particular level of security repository. Analytical models are presented on beliefs analysis, security threat analysis, and average security cost incurred during the transactions session. The results of this scheme are compared with regular schemes, and advantageous and limitations of the proposed scheme are discussed. A case study on application of the proposed security selection scheme is conducted over mobile banking application, and results are presented. Transactions-based authentication selection scheme (TBAS-Scheme) The authentication protocols/schemes are used at the application-level to authenticate the genuine users/parties and devices used in the application. Most of these protocols challenges the user/device to get the authentication information, rather than deploying the methods to identify the validity of a user/device. Therefore, there is a need for an authentication scheme, which intelligently authenticates a user by continuously monitoring the genuinity of the activities/events/ behaviors/transactions through out the session. Transactions-based authentication selection scheme provides a new dimension in authenticating users of services. It enables strong authentication at the transaction level, based on sensitivity level of the given transaction, and user behaviors. The proposed approach intensifies the procedure of authentication by selecting authentication schemes by using the BOB-model and transactions classification models. It provides effective authentication solution, by relieving the conventional authentication systems, from being dependent only on the strength of authentication identifiers. We have made a performance comparison between transactions-based authentication selection scheme with session-based authentication scheme in terms of identification of various active attacks, and average authentication delay and average authentication costs are analysed. We have also shown the working of the proposed scheme in inter-domain and intra-domain hand-off scenarios, and discussed the merits of the scheme comparing it with mobile IP authentication scheme. A case study on application of the proposed authentication selection scheme for authenticating personalized multimedia services is presented. Implementation of the TBSS and the TBAS schemes for mobile commerce application We have implemented the integrated working of both the TBSS and TBAS schemes for a mo-bile commerce application. The details on identifying vendor selection, day of purchase, time of purchase, transaction value, frequency of purchase behaviors are given. A sample list of mobile commerce transactions is presented along with their classification into various sensitivity levels. The working of the system is discussed using three cases of purchases, and the results on trans-actions distribution, deviation factor generation, security technique selection, and authentication challenge generation are presented. In summary, we have developed an integrated dynamic security and authentication system using, the above mentioned selection schemes for mobile transactions, and by incorporating the BOB model, transactions classification model, and credibility modules. We have successfully implemented the proposed schemes using cognitive agents based middleware. The results of experiments suggest that incorporating user behaviors, and transaction sensitivity levels will bring dynamism and adaptiveness to security and authentication system. Through which the mobile communication security could be made more robust to attacks, and resource savvy in terms of reduced bandwidth and computation requirements by using an appropriate security and authentication technique/protocol.