This thesis proposes a novel out-of-step protection technique using the state-plane representation of the generator speed and power angle. The critical clearing angle is computed using the principle that the total energy of the system at the instant the fault is cleared should be equal to the maximum potential energy of the system. The critical clearing time corresponding to this value of critical clearing angle is obtained directly using the time calibration of the relative speed versus power angle solution curve. The simultaneous calculation of the critical clearing angle and the time makes the proposed state plane approach much faster than the two-blinder scheme, Equal Area Criterion (EAC) method, rate of change of impedance method, the Swing Center Voltage (SCV) technique, transient energy calculation method, and the frequency deviation calculation from voltage signal method discussed in the literature. The proposed state plane prediction scheme is used to detect the rst swing out-of-step condition in single machine in nite bus (SMIB) system as well as larger power system con gurations (two-area and IEEE 39-bus test systems) using system wide information. A coherency analysis is performed in a multi-machine system to find out the two critical groups of generators. The critical generator groups are then represented with a SMIB equivalent system, and the state plane algorithm is applied to the reduced equivalent. Electromagnetic transient simulations are carried out using PSCAD/EMTDCTM to test the proposed algorithm in the above discussed test systems. The simulation studies show that the proposed method is computationally e cient, and accurate even for the larger power systems. The technique also does not require any o ine studies. This thesis also proposes another out-of-step protection technique using generator state deviations to detect multi-swing instability conditions in power system. It uses wide-area measurements of generator electrical power and speed deviations as inputs to the proposed scheme to detect instability. This technique is not as fast as the state plane approach but can predict multi-swing instability conditions in power system. The state plane method and state deviation method are used together to nd rst swing and multi-swing instability conditions. Two-area power system con guration is used to demonstrate multi-swing instability prediction. Di erent power swing conditions such as stable, rst swing unstable and multi-swing unstable scenarios are created and the proposed techniques are tested to verify their performance. The proposed techniques are also compared with the conventional two blinder technique. A facility for hardware-in-the-loop testing of the relays using a digital simulator is available in the Power System Laboratory at the University of Saskatchewan. An out-of-step relay module is developed in a digital signal processing board (ADSP BF533TM from Analog Devices Inc.) and a closed loop test is performed using the real time digital simulator (RTDSTM). The simulator mimics the power system behaviour in real time, and the analog time signals from simulator can be communicated to the relay module. The relay can also feed back the signals to the simulator which can be used to operate the circuit breaker elements in the power system. The SMIB and two area systems are used to test the relay in real time. The relay prototypes for both of the proposed techniques are developed in this thesis. The hardware-in-the-loop implementation and testing show that the calculation times required for the proposed methods are small, and the state plane method especially can predict instability condition much faster than all other methods in current literature.