Face recognition is one of the most natural activities performed by the human beings. It has wide range of applications in the areas of Human Computer Interaction, Surveillance, Security etc. Face information of people can be obtained in a non-intrusive manner, without violating privacy. But, robust face recognition which is invariant under varying pose, illumination etc is still a challenging problem. The main aim of this thesis is to explore the usefulness of smile profile of human beings as an extra aid in recognizing people by faces. Smile profile of a person is the sequence of images captured by a camera when the person voluntarily smiles. Using sequence of images instead of a single image will increase the required computational resources significantly. The challenge here is to design a feature extraction technique from a smile sample, which is useful for authentication and is also efficient in terms of storage and computational aspects. There are some experimental evidences which support the claim that facial expressions have some person specific information. But, to the best of our knowledge, systematic study of a particular facial expression for biometrical purposes has not been done so far. The smile profile of human beings, which is captured under some reasonably controlled setup, is used for first time for face recognition purpose. As a first step, we applied two of the recent subspace based face classifiers on the smile samples. We were not able to obtain any conclusive results out of this experiment. Next we extracted features using only the difference vectors obtained from smile samples. The difference vectors depend only on the variations which occur in the corresponding smile profile. Hence any characterization we obtain from such features can be fully attributed to the smiling action. The feature extraction technique we employed is very much similar to PCA. The smile signature that we have obtained is named as Principal Direction of Change(PDC). PDC is a unit vector (in some high dimensional space) which represents the direction in which the major changes occurred during the smile. We obtained a reasonable recognition rate by applying Nearest Neighbor Classifier(NNC) on these features. In addition to that, these features turn out to be less sensitive to the speed of smiling action and minor variations in face detection and head orientation, while capturing the pattern of variations in various regions of face due to smiling action. Using set of experiments on PDC based features we establish that smile has some person specific characteristics. But the recognition rates of PDC based features are less than the recent conventional techniques. Next we have used PDC based features to aid a conventional face classifier. We have used smile signatures to reject some candidate faces. Our experiments show that, using smile signatures, we can reject some of the potential false candidate faces which would have been accepted by the conventional face classifier. Using this smile signature based rejection, the performance of the conventional classifier is improved significantly. This improvement suggests that, the biometric information available in smile profiles does not exist in still images. Hence the usefulness of smile profiles for biometric applications is established through this experimental investigation.

We present a theory for constructing linear, black box approximations to face recognition algorithms and empirically demonstrate that a surprisingly diverse set of face recognition approaches can be approximated well using a linear model. The construction of the linear model to a face recognition algorithm involves embedding of a training set of face images constrained by the distances between them, as computed by the face recognition algorithm being approximated. We accomplish this embedding by iterative majorization, initialized by classical multi-dimensional scaling (MDS). We empirically demonstrate the adequacy of the linear model using six face recognition algorithms, spanning both template based and feature based approaches on standard face recognition benchmarks such as the Facial Recognition Technology (FERET) and Face Recognition Grand Challenge (FRGC) data sets. The experimental results show that the average Error in Modeling for six algorithms is 6.3% at 0.001 False Acceptance Rate (FAR), for FERET fafb probe set which contains maximum number of subjects among all the probe sets. We demonstrate the usefulness of the linear model for algorithm dependent indexing of face databases and find that it results in more than 20 times reduction in face comparisons for Bayesian Intra/Extra-class person classifier (BAY), Elastic Bunch Graph Matching algorithm (EBGM), and the commercial face recognition algorithms. We also propose a novel paradigm to reconstruct face templates from match scores using the linear model and use the reconstructed templates to explore the security breach in a face recognition system. We evaluate the proposed template reconstruction scheme using three, fundamentally different, face recognition algorithms: Principal Component Analysis (PCA), Bayesian Intra/Extra-class person classifier (BAY), and a feature based commercial algorithm. With an operational point set at 1% False Acceptance Rate (FAR) and 99% True Acceptance Rate (TAR) for 1196 enrollments (FERET gallery), we show that at most 600 attempts (score computations) are required to achieve 73%, 72% and 100% chance of breaking in as a randomly chosen target subject for the commercial, BAY and PCA based face recognition system, respectively. We also show that the proposed reconstruction scheme has 47% more probability of breaking in as a randomly chosen target subject for the commercial system as compared to a hill climbing approach with the same number of attempts.