Face recognition finds various applications in surveillance, Law enforcement etc. These applications require fast image processing in real time. Modern GPUs have evolved fully programmable parallel stream processors. The problem of face recognition in real time system is benefited by parallelism. With the aim of fulfilling both speed and accuracy criteria we present a GPU accelerated Face Recognition system. OpenCL is a heterogeneous computing language that allows extracting parallelism on different platforms like DSP processors, FPGAs, GPUs. The proposed kernel on GPU exploits coarse grain parallelism for Local Binary Pattern (LBP) histogram computation and ELTP (Enhanced Local Ternary Pattern) feature extraction. The proposed optimization techniques on local memory, work group size and work group dimension enhances the computation of face recognition on GPU further. As a result, we have achieved a speed up of 30 times to 300 times for 124*124 to 2048*2048 image sizes for LBP and ELTP feature extraction compared to CPU. We also present a robust real time face recognition and tracking on GPU using fusion of RGB and Depth images taken from Kinect sensor. The proposed segmentation after detection algorithm enhances the performances of recognition using LBP.