This thesis presents a robust real-time active tracking system with a pan-tilt camera. The proposed visual servo framework is able to track a deformed object and maintain the target always inside the field of view. For the image processing, an efficient template matching and searching method using the mean-shift theory is developed. The robustness is achieved by appending the ratio histogram, a kernel function, and the template update to the framework when the target is deformed. Then the pan-tilt unit turns towards the target and keeps the target inside the field of view of the camera by feeding back the position information to a Kalman filter. Experimental results show that the presented scheme works successfully when the target is vague or concealed or deformed. The visual tracking task can also be accomplished even when a similar object crosses over the target. In addition, the refreshing rate can be up to 60 frames per second.