In this study a fully automatic and scale invariant feature extractor which does not require manual initialization or special equipment is proposed. Face location and size is extracted using skin segmentation and ellipse fitting. Extracted face region is scaled to a predefined size, later upper and lower facial templates are used for feature extraction. Template localization and template parameter calculations are carried out using Principal Component Analysis. Changes in facial feature coordinates between analyzed image and neutral expression image are used for expression classification. Performances of different classifiers are evaluated. Performance of proposed feature extractor is also tested on sample video sequences. Facial features are extracted in the first frame and KLT tracker is used for tracking the extracted features. Lost features are detected using face geometry rules and they are relocated using feature extractor. As an alternative to feature based technique an available holistic method which analyses face without partitioning is implemented. Face images are filtered using Gabor filters tuned to different scales and orientations. Filtered images are combined to form Gabor jets. Dimensionality of Gabor jets is decreased using Principal Component Analysis. Performances of different classifiers on low dimensional Gabor jets are compared. Feature based and holistic classifier performances are compared using JAFFE and AF facial expression databases.

In this dissertation, we focus on solving Nonlinear Least Squares problems using a supervised approach. In particular, we developed a Supervised Descent Method (SDM), performed thorough theoretical analysis, and demonstrated its effectiveness on optimizing analytic functions, and four other real-world applications: Inverse Kinematics, Rigid Tracking, Face Alignment (frontal and multi-view), and 3D Object Pose Estimation. In Rigid Tracking, SDM was able to take advantage of more robust features, such as, HoG and SIFT. Those non-differentiable image features were out of consideration of previous work because they relied on gradient-based methods for optimization. In Inverse Kinematics where we minimize a non-convex function, SDM achieved significantly better convergence than gradient-based approaches. In Face Alignment, SDM achieved state-of-the-arts results. Moreover, it was extremely computationally efficient, which makes it applicable for many mobile applications. In addition, we provided a unified view of several popular methods including SDM on sequential prediction, and reformulated them as a sequence of function compositions. Finally, we suggested some future research directions on SDM and sequential prediction.

Accurate and efficient localization of facial features is a crucial first step in many face-related computer vision tasks. Some of these tasks include, but not limited to: identity recognition, expression recognition, and head-pose estimation. Most effort in the field has been exerted towards developing better ways of modeling prior appearance knowledge and image observations. Modeling prior shape knowledge, on the other hand, has not been explored as much. In this dissertation I primarily focus on the limitations of the existing methods in terms of modeling the prior shape knowledge. I first introduce a new pose-constrained shape model. I describe my shape model as being "highly flexible yet sufficiently strict". Existing pose-constrained shape models are either too strict, and have questionable generalization power, or they are too loose, and have questionable localization accuracies. My model tries to find a good middle-ground by learning which shape constraints are more "informative" and should be kept, and which ones are not-so-important and may be omitted. I build my pose-constrained facial feature localization approach on this new shape model using a probabilistic graphical model framework. Within this framework, observed and unobserved variables are defined as the local image observations, and the feature locations, respectively. Feature localization, or "probabilistic inference", is then achieved by nonparametric belief propagation. I show that this approach outperforms other popular pose-constrained methods through qualitative and quantitative experiments. Next, I expand my pose-constrained localization approach to unconstrained setting using a multi-model strategy. While doing so, once again I identify and address the two key limitations of existing multi-model methods: 1) semantically and manually defining the models or "guiding" their generation, and 2) not having efficient and effective model selection strategies. First, I introduce an approach based on unsupervised clustering where the models are automatically learned from training data. Then, I complement this approach with an efficient and effective model selection strategy, which is based on a multi-class naive Bayesian classifier. This way, my method can have many more models, each with a higher level of expressive power, and consequently, provides a more effective partitioning of the face image space. This approach is validated through extensive experiments and comparisons with state-of-the-art methods on state-of-the-art datasets. In the last part of this dissertation I discuss a particular application of the previously introduced techniques; facial feature localization in unconstrained videos. I improve the frame-by-frame localization results, by estimating the actual head-movement from a sequence of noisy head-pose estimates, and then using this information for detecting and fixing the localization failures. / text

The problem of automatic face recognition is to visually identify a person in an input image. This task is performed by matching the input face against the faces of known people in a database of faces. Most existing work in face recognition has limited the scope of the problem, however, by dealing primarily with frontal views, neutral expressions, and fixed lighting conditions. To help generalize existing face recognition systems, we look at the problem of recognizing faces under a range of viewpoints. In particular, we consider two cases of this problem: (i) many example views are available of each person, and (ii) only one view is available per person, perhaps a driver's license or passport photograph. Ideally, we would like to address these two cases using a simple view-based approach, where a person is represented in the database by using a number of views on the viewing sphere. While the view-based approach is consistent with case (i), for case (ii) we need to augment the single real view of each person with synthetic views from other viewpoints, views we call 'virtual views'. Virtual views are generated using prior knowledge of face rotation, knowledge that is 'learned' from images of prototype faces. This prior knowledge is used to effectively rotate in depth the single real view available of each person. In this thesis, I present the view-based face recognizer, techniques for synthesizing virtual views, and experimental results using real and virtual views in the recognizer.

We propose a novel approach for segmentation of human objects, including face and body, in image sequences. In modern video coding techniques, e.g., MPEG-4 and MPEG-7, human objects are usually the main focus for multimedia applications. We combine temporal and spatial information and employ a neuro-fuzzy mechanism to extract human objects. A fuzzy self-clustering technique is used to divide the video frame into a set of segments. The existence of a face within a candidate face region is ensured by searching for possible constellations of eye-mouth triangles and verifying each eye-mouth combination with the predefined template. Then rough foreground and background are formed based on a combination of multiple criteria. Finally, human objects in the base frame and the remaining frames of the video stream are precisely located by a fuzzy neural network which is trained by a SVD-based hybrid learning algorithm. Through experiments, we compare our system with two other approaches, and the results have shown that our system can detect face locations and extract human objects more accurately.

Video annotation has become an important issue due to the rapidly increasing amount of video available. For efficient video content searches, annotation has to be done beforehand, which is a time-consuming process if done manually. Automatic annotation of faces for person identification is a major challenge in the context of content-based video retrieval. This thesis work focuses on the development of a semi-automatic face annotation system which benefits from online learning methods. The system creates a face database by using face detection and tracking algorithms to collect samples of the encountered faces in the video and by receiving labels from the user. Using this database a learner model is trained. While the training session continues, the system starts offering labels for the newly encountered faces and lets the user acknowledge or correct the suggested labels hence a learner is updated online throughout the video. The user is free to train the learner until satisfactory results are obtained. In order to create a face database, a shot boundary algorithm is implemented to partition the video into semantically meaningful segments and the user browses through the video from one shot boundary to the next. A face detector followed by a face tracker is implemented to collect face samples within two shot boundary frames. For online learning, feature extraction and classification methods which are computationally efficient are investigated and evaluated. Sequential variants of some robust batch classification algorithms are implemented. Combinations of feature extraction and classification methods have been tested and compared according to their face recognition accuracy and computational performances.

Abstract Communicating with mobile devices has become an unavoidable part of our daily life. Unfortunately, the current user interface designs are mostly taken directly from desktop computers. This has resulted in devices that are sometimes hard to use. Since more processing power and new sensing technologies are already available, there is a possibility to develop systems to communicate through different modalities. This thesis proposes some novel computer vision approaches, including head tracking, object motion analysis and device ego-motion estimation, to allow efficient interaction with mobile devices. For head tracking, two new methods have been developed. The first method detects a face region and facial features by employing skin detection, morphology, and a geometrical face model. The second method, designed especially for mobile use, detects the face and eyes using local texture features. In both cases, Kalman filtering is applied to estimate the 3-D pose of the head. Experiments indicate that the methods introduced can be applied on platforms with limited computational resources. A novel object tracking method is also presented. The idea is to combine Kalman filtering and EM-algorithms to track an object, such as a finger, using motion features. This technique is also applicable when some conventional methods such as colour segmentation and background subtraction cannot be used. In addition, a new feature based camera ego-motion estimation framework is proposed. The method introduced exploits gradient measures for feature selection and feature displacement uncertainty analysis. Experiments with a fixed point implementation testify to the effectiveness of the approach on a camera-equipped mobile phone. The feasibility of the methods developed is demonstrated in three new mobile interface solutions. One of them estimates the ego-motion of the device with respect to the user's face and utilises that information for browsing large documents or bitmaps on small displays. The second solution is to use device or finger motion to recognize simple gestures. In addition to these applications, a novel interactive system to build document panorama images is presented. The motion estimation and recognition techniques presented in this thesis have clear potential to become practical means for interacting with mobile devices. In fact, cameras in future mobile devices may, for the most of time, be used as sensors for self intuitive user interfaces rather than using them for digital photography.

No / In this paper we present a new method for accurate real-time facial feature detection. Our method is based on local feature detection and enhancement. Previous work in this area, such as that of Viola and Jones, require looking at the face as a whole. Consequently, such approaches have increased chances of reporting negative hits. Furthermore, such algorithms require greater processing power and hence they are especially not attractive for real-time applications. Through our recent work, we have devised a method to identify the face from real-time images and divide it into regions of interest (ROI). Firstly, based on a face detection algorithm, we identify the face and divide it into four main regions. Then, we undertake a local search within those ROI, looking for specific facial features. This enables us to locate the desired facial features more efficiently and accurately. We have tested our approach using the Cohn-Kanade’s Extended Facial Expression (CK+) database. The results show that applying the ROI has a relatively low false positive rate as well as provides a marked gain in the overall computational efficiency. In particular, we show that our method has a 4-fold increase in accuracy when compared to existing algorithms for facial feature detection.

No / The purpose of this chapter is to discuss and present a computational framework for detecting and analysing facial expressions efficiently. The approach here is to identify the face and estimate regions of facial features of interest using the optical flow algorithm. Once the regions and their dynamics are computed a rule based system can be utilised for classification. Using this framework, we show how it is possible to accurately identify and classify facial expressions to match with FACS coding and to infer the underlying basic emotions in real time.

The wellbeing of horses is very important to their care takers, trainers, veterinarians, and owners. This thesis describes the development of a non-invasive image processing technique that allows for automatic detection and tracking of horse head and ear motion, respectively, in videos or camera feed, both of which may provide indications of horse pain, stress, or well-being. The algorithm developed here can automatically detect and track head motion and ear motion, respectively, in videos of a standing horse. Results demonstrating the technique for nine different horses are presented, where the data from the algorithm is utilized to plot absolute motion vs. time, velocity vs. time, and acceleration vs. time for the head and ear motion, respectively, of a variety of horses and ponies. Two-dimensional plotting of x and y motion over time is also presented. Additionally, results of pilot work in eye detection in light colored horses is also presented. Detection of pain in horses is particularly difficult because they are prey animals and have mechanisms to disguise their pain, and these instincts may be particularly strong in the presence of an unknown human, such as a veterinarian. Current state-of-the art for detecting pain in horses primarily involves invasive methods, such as heart rate monitors around the body, drawing blood for cortisol levels, and pressing on painful areas to elicit a response, although some work has been done for humans to sort and score photographs subjectively in terms of a "horse grimace scale." The algorithms developed in this thesis are the first that the author is aware for exploiting proven image processing approaches from other applications for development of an automatic tool for detection and tracking of horse facial indicators. The algorithms were done in common open source programs Python and OpenCV, and standard image processing approaches including Canny Edge detection Hue, Saturation, Value color filtering, and contour tracking were utilized in algorithm development. The work in this thesis provides the foundational development of a non -invasive and automatic detection and tracking program for horse head and ear motion, including demonstration of the viability of this approach using videos of standing horses. This approach lays the groundwork for robust tool development for monitoring horses non-invasively and without the required presence of humans in such applications as post-operative monitoring, foaling, evaluation of performance horses in competition and/or training, as well as for providing data for research on animal welfare, among other scenarios. / MS / There are many things that cause pain in horses, including improper saddle fit, inadequate care, laminitis, lameness, surgery, and colic, among others.The well-being of horses is very important to their care takers, trainers, veterinarians, and owners. Monitoring the well-being of horses is particularly important in many scenarios including post-operative monitoring, therapeutic riding programs, racing, dressage, and rodeo events, among numerous other activities. This thesis describes the development of a computer-based image processing technique for automatic detection and tracking of both horse head and ear motion, respectively, in videos of standing horses. The techniques developed here allow for the collection of data on head and ear motion over time, facilitating analysis of these motions that may provide reliable indicators of horse pain, stress, or well-being. Knowing if a horse is in pain is difficult because horses are prey animals that have mechanisms in place that minimize the display of pain so that they do not become easy targets for predators. Computer vision systems, like the one developed here, may be well suited to detect subtle changes in horse behavior for detecting distress in horses. The ability to remotely and automatically monitor horse well-being by exploiting computer-based image-processing techniques will create significant opportunities to improve the welfare of horses. The work presented here looks at the first use of image-processing approaches to detect and track facial features of standing horses in videos to help facilitate the development of automatic pain and stress detection in videos and camera feeds for owners, veterinarians, and horse-related organizations, among others.