No / There is substantial interest in detection of human behaviour that may reveal people with deliberate malicious intent, who are engaging in deceit. Technology exists that is able to detect changes in facial patterns of movement and thermal signatures on the face. However, there is data deficiency in the research community for further study. Therefore this project aims to overcome the data deficiency in psychology study and algorithms development. A within-subjects design experiment was conducted, using immigration as a scenario for investigate participants in control and experimental conditions. A random sample of 32 volunteers were recruited, their age group is within 18 - 33. The study design required participants to answer questions on two topics, one as themselves and one as a predefined character. Data regarding visible and thermal images of facial movement and behaviour were collected. A rich FACS-coded database with high quality thermal images was established. Finally, recommendations for development and subsequent implementation of the facial analysis technique were made.

Facial expressions convey non-verbal cues. Humans use facial expressions to show emotions, which play an important role in interpersonal relations and can be of use in many applications involving psychology, human-computer interaction, health care, e-commerce, and many others. Although humans recognize facial expressions in a scene with little or no effort, reliable expression recognition by machine is still a challenging problem. Automatic facial expression recognition (FER) has several related problems: face detection, face representation, extraction of the facial expression information, and classification of expressions, particularly under conditions of input data variability such as illumination and pose variation. A system that performs these operations accurately and in realtime would be a major step forward in achieving a human-like interaction between the man and machine. This document introduces novel approaches for the automatic recognition of the basic facial expressions, namely, happiness, surprise, sadness, fear, disgust, anger, and neutral using relatively low-resolution noisy sensor such as the Microsoft Kinect. Such sensors are capable of fast data collection, but the low-resolution noisy data present unique challenges when identifying subtle changes in appearance. This dissertation will present the work that has been done to address these challenges and the corresponding results. The lack of Kinect-based FER datasets motivated this work to build two Kinect-based RGBD+time FER datasets that include facial expressions of adults and children. To the best of our knowledge, they are the first FER-oriented datasets that include children. Availability of children data is important for research focused on children (e.g., psychology studies on facial expressions of children with autism), and also allows researchers to do deeper studies on automatic FER by analyzing possible differences between data coming from adults and children. The key contributions of this dissertation are both empirical and theoretical. The empirical contributions include the design and successful test of three FER systems that outperform existing FER systems either when tested on public datasets or in realtime. One proposed approach automatically tunes itself to the given 3D data by identifying the best distance metric that maximizes the system accuracy. Compared to traditional approaches where a fixed distance metric is employed for all classes, the presented adaptive approach had better recognition accuracy especially in non-frontal poses. Another proposed system combines high dimensional feature vectors extracted from 2D and 3D modalities via a novel fusion technique. This system achieved 80% accuracy which outperforms the state of the art on the public VT-KFER dataset by more than 13%. The third proposed system has been designed and successfully tested to recognize the six basic expressions plus neutral in realtime using only 3D data captured by the Kinect. When tested on a public FER dataset, it achieved 67% (7% higher than other 3D-based FER systems) in multi-class mode and 89% (i.e., 9% higher than the state of the art) in binary mode. When the system was tested in realtime on 20 children, it achieved over 73% on a reduced set of expressions. To the best of our knowledge, this is the first known system that has been tested on relatively large dataset of children in realtime. The theoretical contributions include 1) the development of a novel feature selection approach that ranks the features based on their class separability, and 2) the development of the Dual Kernel Discriminant Analysis (DKDA) feature fusion algorithm. This later approach addresses the problem of fusing high dimensional noisy data that are highly nonlinear distributed. / PHD

This thesis discusses a novel method for modelling facial action units. It presents facial action units model based on boundary value problems for accurate representation of human facial expression in three-dimensions. In particular, a solution to a fourth order elliptic Partial Differential Equation (PDE) subject to suitable boundary conditions is utilized, where the chosen boundary curves are based on muscles movement defined by Facial Action Coding System (FACS). This study involved three stages: modelling faces, manipulating faces and application to simple facial animation. In the first stage, PDE method is used in modelling and generating a smooth 3D face. The PDE formulation using small sets of parameters contributes to the efficiency of human face representation. In the manipulation stage, a generic PDE face of neutral expression is manipulated to a face with expression using PDE descriptors that uniquely represents an action unit. A combination of the PDE descriptor results in a generic PDE face having an expression, which successfully modelled four basic expressions: happy, sad, fear and disgust. An example of application is given using simple animation technique called blendshapes. This technique uses generic PDE face in animating basic expressions.

This thesis discusses a novel method for modelling facial action units. It presents facial action units model based on boundary value problems for accurate representation of human facial expression in three-dimensions. In particular, a solution to a fourth order elliptic Partial Differential Equation (PDE) subject to suitable boundary conditions is utilized, where the chosen boundary curves are based on muscles movement defined by Facial Action Coding System (FACS). This study involved three stages: modelling faces, manipulating faces and application to simple facial animation. In the first stage, PDE method is used in modelling and generating a smooth 3D face. The PDE formulation using small sets of parameters contributes to the efficiency of human face representation. In the manipulation stage, a generic PDE face of neutral expression is manipulated to a face with expression using PDE descriptors that uniquely represents an action unit. A combination of the PDE descriptor results in a generic PDE face having an expression, which successfully modelled four basic expressions: happy, sad, fear and disgust. An example of application is given using simple animation technique called blendshapes. This technique uses generic PDE face in animating basic expressions. / Ministry of Higher Education, Malaysia and Universiti Malaysia Terengganu

No / In this paper we discuss a novel method of mathematically modelling facial action units for accurate representation of human facial expressions in 3- dimensions. Our method utilizes the approach of Facial Action Coding System (FACS). It is based on a boundary-value approach, which utilizes a solution to a fourth order elliptic Partial Differential Equation (PDE) subject to a suitable set of boundary conditions. Here the PDE surface generation method for human facial expressions is utilized in order to generate a wide variety of facial expressions in an efficient and realistic way. For this purpose, we identify a set of boundary curves corresponding to the key features of the face which in turn define a given facial expression in 3-dimensions. The action units (AUs) relating to the FACS are then efficiently represented in terms of Fourier coefficients relating to the boundary curves which enables us to store both the face and the facial expressions in an efficient way.

No description available.

Submitted by Automa??o e Estat?stica (sst@bczm.ufrn.br) on 2016-01-14T18:48:05Z No. of bitstreams: 1 GiulianaSilvaBezerra_DISSERT.pdf: 12899912 bytes, checksum: 413f2be6aef4a909500e6834e7b0ae63 (MD5) / Approved for entry into archive by Arlan Eloi Leite Silva (eloihistoriador@yahoo.com.br) on 2016-01-15T18:57:11Z (GMT) No. of bitstreams: 1 GiulianaSilvaBezerra_DISSERT.pdf: 12899912 bytes, checksum: 413f2be6aef4a909500e6834e7b0ae63 (MD5) / Made available in DSpace on 2016-01-15T18:57:11Z (GMT). No. of bitstreams: 1 GiulianaSilvaBezerra_DISSERT.pdf: 12899912 bytes, checksum: 413f2be6aef4a909500e6834e7b0ae63 (MD5) Previous issue date: 2014-12-12 / Emotion-based analysis has raised a lot of interest, particularly in areas such as forensics, medicine, music, psychology, and human-machine interface. Following this trend, the use of facial analysis (either automatic or human-based) is the most common subject to be investigated once this type of data can easily be collected and is well accepted in the literature as a metric for inference of emotional states. Despite this popularity, due to several constraints found in real world scenarios (e.g. lightning, complex backgrounds, facial hair and so on), automatically obtaining affective information from face accurately is a very challenging accomplishment. This work presents a framework which aims to analyse emotional experiences through naturally generated facial expressions. Our main contribution is a new 4-dimensional model to describe emotional experiences in terms of appraisal, facial expressions, mood, and subjective experiences. In addition, we present an experiment using a new protocol proposed to obtain spontaneous emotional reactions. The results have suggested that the initial emotional state described by the participants of the experiment was different from that described after the exposure to the eliciting stimulus, thus showing that the used stimuli were capable of inducing the expected emotional states in most individuals. Moreover, our results pointed out that spontaneous facial reactions to emotions are very different from those in prototypic expressions due to the lack of expressiveness in the latter. / Emotion-based analysis has raised a lot of interest, particularly in areas such as forensics, medicine, music, psychology, and human-machine interface. Following this trend, the use of facial analysis (either automatic or human-based) is the most common subject to be investigated once this type of data can easily be collected and is well accepted in the literature as a metric for inference of emotional states. Despite this popularity, due to several constraints found in real world scenarios (e.g. lightning, complex backgrounds, facial hair and so on), automatically obtaining affective information from face accurately is a very challenging accomplishment. This work presents a framework which aims to analyse emotional experiences through naturally generated facial expressions. Our main contribution is a new 4-dimensional model to describe emotional experiences in terms of appraisal, facial expressions, mood, and subjective experiences. In addition, we present an experiment using a new protocol proposed to obtain spontaneous emotional reactions. The results have suggested that the initial emotional state described by the participants of the experiment was different from that described after the exposure to the eliciting stimulus, thus showing that the used stimuli were capable of inducing the expected emotional states in most individuals. Moreover, our results pointed out that spontaneous facial reactions to emotions are very different from those in prototypic expressions due to the lack of expressiveness in the latter.

Recueillir et labelliser un ensemble important et pertinent de données pour apprendre des systèmes de prédiction d'informations à partir de visages est à la fois difficile et long. Par conséquent, les données disponibles sont souvent de taille limitée comparée à la difficultés des tâches. Cela rend le problème du sur-apprentissage particulièrement important dans de nombreuses applications d'apprentissage statistique liées au visage. Dans cette thèse, nous proposons une nouvelle méthode de régression de labels multi-dimensionnels, nommée Hard Multi-Task Metric Learning for Kernel Regression (H-MT-MLKR). Notre méthode a été développée en focalisant sur la réduction du phénomène de sur-apprentissage. La méthode Metric Learning for Kernel Regression qui a été proposée par Kilian Q. Weinberger en 2007 vise à apprendre un sous-espace pour minimiser l'erreur quadratique d'un estimateur de Nadaraya-Watson sur la base d'apprentissage. Dans notre méthode, on étend la méthode MLKR pour une régression de labels multi-dimensionnels en ajoutant une nouvelle régularisation multi-tâches qui réduit les degrés de liberté du modèle appris ainsi que le sur-apprentissage. Nous évaluons notre méthode pour deux applications différentes, à savoir la localisation de points caractéristiques et la prédiction de l'intensité des Action Units. Nous présentons aussi un travail sur la prédiction des émotions en espace continu basé aussi sur l'estimateur de Nadaraya-Watson. Deux des systèmes proposés nous ont permis de remporter deux premières places à des concours internationaux, à savoir le Audio-Visual Emotion Challenge (AVEC'12) et le Facial Expression Recognition and Analysis challenge (FERA'15). / Collecting and labeling various and relevant data for training automatic facial information prediction systems is both hard and time-consuming. As a consequence, available data is often of limited size compared to the difficulty of the prediction tasks. This makes overfitting a particularly important issue in several face-related machine learning applications. In this PhD, we introduce a novel method for multi-dimensional label regression, namely Hard Multi-Task Metric Learning for Kernel Regression (H-MT-MLKR). Our proposed method has been designed taking a particular focus on overfitting reduction. The Metric Learning for Kernel Regression method (MLKR) that has been proposed by Kilian Q. Weinberger in 2007 aims at learning a subspace for minimizing the quadratic training error of a Nadaraya-Watson estimator. In our method, we extend MLKR for multi-dimensional label regression by adding a novel multi-task regularization that reduces the degrees of freedom of the learned model along with potential overfitting. We evaluate our regression method on two different applications, namely landmark localization and Action Unit intensity prediction. We also present our work on automatic emotion prediction in a continuous space which is based on the Nadaraya-Watson estimator as well. Two of our frameworks let us win international data science challenges, namely the Audio-Visual Emotion Challenge (AVEC’12) and the fully continuous Facial Expression Recognition and Analysis challenge (FERA’15).

The work presented in this study is based on the long-term goal of developing a social robot that can be involved in leading a conversation in a language café. In detail, the study has investigated whether it is possible to classify involvement with a conversation participant based on its facial expression and gaze two factors that previous studies have shown to be central to human engagement. To perform the assessment, the software Openface has extracted said parameters from a previous field study which has then been processed with the machine learning model Support Vector Machine. After a lot of hyperparameter tuning, the final model managed to predict engagement on a three-point scale with 54.5% accuracy. Furthermore, the study has also examined the potential of the new technological paradigm that the social robot represents. The potential has been analyzed on the basis of Dosi’s four dimensions: technological possibilities, appropriability of innovation, cumulativeness of technical advances and properties of the knowledge base. The analysis clarifies that the paradigm has the potential to revolutionize a number of industries as a result of its technological opportunities and worldwide stakeholders, but also faces challenges in the form of technical and ethical difficulties. / Arbetet som presenteras i den här studien grundar sig i det långsiktiga målet att utveckla en social robot som kan vara med och leda samtalssessioner på ett språkcafé. I detalj har studien undersökt om det går att klassificera engagemang hos en samtalsdeltagare utifrån dess ansiktsuttryck och blickriktning – två faktorer som tidigare studier visat sig vara centrala för människans engagemang. För att utföra bedömningen har mjukvaran Openface extraherat nämnda parametrar från en tidigare fältstudie vilka sedan har processats med maskininlärningsmodellen Support Vector Machine. Efter gedigna försök att finna optimala värden på hyperparametrar till modellen lyckades den slutligen predicera engagemang på en tregradig skala med 54,5% accuracy. Vidare har studien också undersökt potentialen för det nya teknologiska paradigmet som den sociala roboten utgör. Potentialen har analyserats med utgångspunkt i Dosis fyra dimensioner: teknologiska möjligheter, möjliga vinster från innovation, kumulativ höjd på teknologiska framsteg och egenskaper i kunskapsbasen. Analysen klargör att paradigmet har förutsättningar att revolutionera ett flertal industrier till följd av dess teknologiska möjligheter och världsomfattande intressenter, men står också inför utmaningar i form av tekniska och etiska svårigheter.

<p>We investigate the use of noise degraded facial images in the application of facial expression recognition. In particular, we trained Gabor+SVMclassifiers to recognize facial expressions images with various types of noise. We applied Gaussian noise, Poisson noise, varying levels of salt and pepper noise, and speckle noise to noiseless facial images. Classifiers were trained with images without noise and then tested on the images with noise. Next, the classifiers were trained using images with noise, and then on tested both images that had noise, and images that were noiseless. Finally, classifiers were tested on images while increasing the levels of salt and pepper in the test set. Our results reflected distinct degradation of recognition accuracy. We also discovered that certain types of noise, particularly Gaussian and Poisson noise, boost recognition rates to levels greater than would be achieved by normal, noiseless images. We attribute this effect to the Gaussian envelope component of Gabor filters being sympathetic to Gaussian-like noise, which is similar in variance to that of the Gabor filters. Finally, using linear regression, we mapped a mathematical model to this degradation and used it to suggest how recognition rates would degrade further should more noise be added to the images.</p>