Electrical Power System is one of the most complex real time operating systems. It is probably one of the best examples of a large interconnected nonlinear system of varying nature. The system needs to be operated and controlled with component or system problems, often with combinatorial complexity. In addition, time scales of operation and control can vary from milliseconds to minutes to hours. It is difficult to maintain such a system at constant operating condition due to both small and large disturbances such as sudden change in loads, change in network configuration, fluctuations in turbine output, and various types of faults etc. The system is therefore affected by a variety of instability problems. Among all these instability problems one of the important modes of instability is related to dynamic instability or more precisely the small perturbation oscillatory instability. Oscillations of small magnitude and low frequency (in the range of 0.1Hz to 2.5Hz) could persist for long periods, limiting the power transfer capability of the transmission lines. Power System Stabilizers (PSS) were developed as auxiliary controllers on the excitation system to improve the system damping performance by modulating the generator excitation voltage. However, the synthesis of an effective PSS for all operating conditions still remains a difficult and challenging task. The design and tuning of PSS for robust operation is a laborious process. The existing PSS design techniques require considerable expertise, the complete system information and extensive eigenvalue calculations which increases the computational burden as the system size increases. Conventional automatic voltage regulator (AVR) and PSS designs are based on linearized models of power systems which fail to stabilize the system over a wide range of operating conditions. In the last decade or so, a variety of nonlinear control techniques have become available. In this thesis, an attempt is made to explore the suitability of some of these design techniques for designing excitation controllers to enhance small perturbation stability of power systems over a wide range of operating and system conditions. This thesis first proposes a method of designing power system stabilizers based on local measurements alone, in multi-machine systems. Next, a method has been developed to analyze and quantify the small signal performance benefits of replacing the existing AVR+PSS structure with nonlinear voltage regulators. A number of new nonlinear controller designs have been proposed subsequently. These include, (a) a new decentralized nonlinear voltage regulator for multi machine power systems with a single tunable parameter that can achieve effective trade of between both the voltage regulation and small signal objectives, (b) a decentralized Interconnection and Damping Assignment Passivity Based Controller in addition to a proportional controller that can achieve all the requirements of an excitation system and (c) a Nonlinear Quadratic Regulator PSS using Single Network Adaptive Critic architecture in the frame work of approximate dynamic programming. Performance of all the proposed controllers has been analyzed using a number of multi machine test systems over a range of operating conditions.