Electric power systems are among the most complex man made systems on the world. Most of the time, they operate under quasi-steady state. With the ever increasing load demand and the advent of the deregulated power market recently, the power systems are pushed more often to operate close to their design limits and with more uncertainty of the system operating mode. With the increasing complexity and more interconnected systems, power systems are operating closer to their performance limits. As a result, maintaining system security and facilitating efficient system operation have been challenging tasks. Transmission systems are considered the most vital components in power systems connecting both generating/substation and consumer areas with several interconnected networks. In the past, they were owned by regulated, vertically integrated utility companies. They have been designed and operated so that conditions in close proximity to security boundaries are not frequently encountered. However, in the new open access environment, operating conditions tend to be much closer to security boundaries, as transmission use is increasing in sudden and unpredictable directions. Transmission unbundling, coupled with other regulatory requirements, has made new transmission facility construction more difficult. Unfortunately these transmission lines are frequently subjected to a wide variety of faults. Thus, providing proper protective functions for them is essential. Generally the protection of Extra High Voltage (EHV) and Ultra High Voltage (UHV) transmission lines are carried out by the use of distance relays in view of the fact that they provide fast fault clearance and system coordination. Transmission line relaying involves detection, classification and location of transmission line faults. Fast detections of faults enable quick isolation of the faulty line from service and hence, protecting it from the harmful effects of fault. Classification of faults means identification of the type of fault and faulted line section, and this information is required for finding the fault location and assessing the extent of repair work to be carried out. Accurate fault location is necessary for facilitating quick repair and restoration of the line, to improve the reliability and availability of the power supply. Generally, the protection system using conventional distance relaying algorithm involves three zones. The first zone (Z1) of the relay is set to detect faults on 80%90% of the protected line without any intentional time delay. The second zone (Z2) is set to protect the remainder of the line plus an adequate margin. Second zone relays are time delayed for 1530 cycles to coordinate with relays at remote bus. The settings of the third zone (Z3) ideally will cover the protected line, plus all of the longest line leaving the remote station. Z3 of a distance relay is used to provide the remote backup protection in case of the failure of the primary protection. Since Z3 covers an adjacent line, a large infeed (outfeed) from the remote terminal causes the relay to underreach (overreach). Thus, a very large load at the remote terminal may cause distance relays to mal-operate. Settings for conventional distance relays are selected to avoid overreach/underreach operation under the worst case scenarios. Studies of significant power system disturbances reported by North American Electric Reliability Council (NERC) indicate that protective relays are involved, one way or another, in 75% of the major disturbances and the most troublesome ones are backup protection relays. With their limited view of the interconnected network based on their locally measured inputs, conventional backup protection relays generally take actions to protect a localized region of the network without considering the impact on the whole network. Relay mal-operations or unintended operations due to overload, power swing, and relay hidden failure are the main factors contributing to the blackouts. Most of the problems are associated with relays tripping too many healthy lines. Since a relay makes the decision automatically to remove a component from the system according to its internal mechanism, the relay mal-operation or unintended operation can make an effective influence on the system stability. Approaches to reduce the relay misbehavior need to be identified. Real time monitoring tools to assess the relay misbehavior are needed, providing the system operator, the accurate information about unfolding events. Existing transmission line protection scheme still has drawbacks. Advanced fault analysis mechanism to enhance the system dependability and security simultaneously is desirable. Relay settings play a significant role in major blackouts. So correct settings should be calculated and coordinated by suitable studies. Attempts are to be made to employ highly accurate AI techniques in protective system implementation. The research work focussed on developing knowledge based intelligent tools for the improving the transmission system security. A process to obtain knowledgebase using SVMs for ready post-fault diagnosis purpose is developed. SVMs are used as Intelligent tool for identifying the faulted line that is emanating from a substation and finding the distance from the substation. The approach uses phasor values of the line voltages and currents after the fault has been detected. The approach is particularly important for post-fault diagnosis of any mal-operation of relays following a disturbance in the neighboring line connected to the same substation. This may help in improving the fault monitoring/diagnosis process and coordination of the protective relays, thus assuring secure operation of the power systems. The approach based on SVMs, exploits the first part of this goal. For comparison, a classifier and regression tools based on the RBFNNs was also investigated. The RBFNNs and SVM networks are introduced and considered as an appropriate tool for pattern recognition problems. Results on a practical 24Bus equivalent EHV transmission system of Indian Southern region and on IEEE39 bus New England system are presented to illustrate the proposed method. In a large connected power network, the number of generators are more in number and their set patterns number will be large. As the line flows are sensitive to generator set patterns, it is difficult to consider all the combinations of generators while simulating the training and testing patterns as input to SVMs. To simulate the training and testing patterns corresponding to possible changes in line flows to meet the load in the present deregulated environment, line flow sensitive generators set to be identified/merit-listed. In this regard, to identify the most sensitive generators for a particular line of interest, a method from the literature is adopted and developed a software program based on the graph theory concepts. Case studies on generator contributions towards loads and transmission lines are illustrated on an equivalent 33bus system, a part of Indian Northern grid with major part of Uttar Pradesh and also with an equivalent 246bus system of practical Indian Southern grid. A distance relay coordination approach is proposed using detailed simulation studies, taking into account various operating conditions and fault resistances. Support Vector Machines as a pattern classifier is used for obtaining distance relay coordination. The scheme uses the apparent impedance values observed during fault as inputs. SVMs are used to build the underlying concept between reach of different zones and the impedance trajectory during fault. An improved performance with the use of SVMs, keeping the reach when faced with different fault conditions as well as line flow changes are illustrated with an equivalent 246bus system of practical Indian Southern grid and also with an equivalent 265bus system of practical Indian Western grid. A strategy of Supportive System is described to aid the conventional protection philosophy in combating situations where protection systems are mal-operated and/or information is missing and provide selective and secure coordination. Highly accurate identification/discrimination of zones plays a key role in effective implementation of the region-wide supportive system. This typically requires a multiclass SVM classifier to effectively analyze/build the underlying concept between reach of different zones and the apparent impedance trajectory during fault. Different multiclass methods are compared for their performance with respect to accuracy, number of iterations, number of support vectors, training and testing time. The performance analysis of these methods is presented on three data sets belonging to the training and testing patterns of three supportive systems for a region, part of a network, which is an equivalent 265bus system of practical Indian Western grid.