Emotion influences our actions, and this means that emotion has subjective decision value. Emotions, properly interpreted and understood, of those affected by decisions provide feedback to actions and, as such, serve as a basis for decisions. Accordingly, "affective computing" represents a wide range of technological opportunities toward the implementation of emotions to improve human-computer interaction, which also includes insights across a range of contexts of computational sciences into how we can design computer systems to communicate and recognize the emotional states provided by humans. Today, emotional systems such as software-only agents and embodied robots seem to improve every day at managing large volumes of information, and they remain emotionally incapable to read our feelings and react according to them. From a computational viewpoint, technology has made significant steps in determining how an emotional behavior model could be built; such a model is intended to be used for the purpose of intelligent assistance and support to humans. Human emotions are engines that allow people to generate useful responses to the current situation, taking into account the emotional states of others. Recovering the emotional cues emanating from the natural behavior of humans such as facial expressions and bodily kinetics could help to develop systems that allow recognition, interpretation, processing, simulation, and basing decisions on human emotions. Currently, there is a need to create emotional systems able to develop an emotional bond with users, reacting emotionally to encountered situations with the ability to help, assisting users to make their daily life easier. Handling emotions and their influence on decisions can improve the human-machine communication with a wider vision. The present thesis strives to provide an emotional architecture applicable for an agent, based on a group of decision-making models influenced by external emotional information provided by humans, acquired through a group of classification techniques from machine learning algorithms. The system can form positive bonds with the people it encounters when proceeding according to their emotional behavior. The agent embodied in the emotional architecture will interact with a user, facilitating their adoption in application areas such as caregiving to provide emotional support to the elderly. The agent's architecture uses an adversarial structure based on an Adversarial Risk Analysis framework with a decision analytic flavor that includes models forecasting a human's behavior and their impact on the surrounding environment. The agent perceives its environment and the actions performed by an individual, which constitute the resources needed to execute the agent's decision during the interaction. The agent's decision that is carried out from the adversarial structure is also affected by the information of emotional states provided by a classifiers-ensemble system, giving rise to a "decision with emotional connotation" included in the group of affective decisions. The performance of different well-known classifiers was compared in order to select the best result and build the ensemble system, based on feature selection methods that were introduced to predict the emotion. These methods are based on facial expression, bodily gestures, and speech, with satisfactory accuracy long before the final system. / <p>At the time of the doctoral defense, the following paper was unpublished and had a status as follows: Paper 8: Accepted.</p>

<p>We investigate the use of noise degraded facial images in the application of facial expression recognition. In particular, we trained Gabor+SVMclassifiers to recognize facial expressions images with various types of noise. We applied Gaussian noise, Poisson noise, varying levels of salt and pepper noise, and speckle noise to noiseless facial images. Classifiers were trained with images without noise and then tested on the images with noise. Next, the classifiers were trained using images with noise, and then on tested both images that had noise, and images that were noiseless. Finally, classifiers were tested on images while increasing the levels of salt and pepper in the test set. Our results reflected distinct degradation of recognition accuracy. We also discovered that certain types of noise, particularly Gaussian and Poisson noise, boost recognition rates to levels greater than would be achieved by normal, noiseless images. We attribute this effect to the Gaussian envelope component of Gabor filters being sympathetic to Gaussian-like noise, which is similar in variance to that of the Gabor filters. Finally, using linear regression, we mapped a mathematical model to this degradation and used it to suggest how recognition rates would degrade further should more noise be added to the images.</p>

This Ph.D thesis work is dedicated to automatic facial analysis in 3D, including facial landmarking and facial expression recognition. Indeed, facial expression plays an important role both in verbal and non verbal communication, and in expressing emotions. Thus, automatic facial expression recognition has various purposes and applications and particularly is at the heart of "intelligent" human-centered human/computer(robot) interfaces. Meanwhile, automatic landmarking provides aprior knowledge on location of face landmarks, which is required by many face analysis methods such as face segmentation and feature extraction used for instance for expression recognition. The purpose of this thesis is thus to elaborate 3D landmarking and facial expression recognition approaches for finally proposing an automatic facial activity (facial expression and action unit) recognition solution.In this work, we have proposed a Bayesian Belief Network (BBN) for recognizing facial activities, such as facial expressions and facial action units. A StatisticalFacial feAture Model (SFAM) has also been designed to first automatically locateface landmarks so that a fully automatic facial expression recognition system can be formed by combining the SFAM and the BBN. The key contributions are the followings. First, we have proposed to build a morphable partial face model, named SFAM, based on Principle Component Analysis. This model allows to learn boththe global variations in face landmark configuration and the local ones in terms of texture and local geometry around each landmark. Various partial face instances can be generated from SFAM by varying model parameters. Secondly, we have developed a landmarking algorithm based on the minimization an objective function describing the correlation between model instances and query faces. Thirdly, we have designed a Bayesian Belief Network with a structure describing the casual relationships among subjects, expressions and facial features. Facial expression oraction units are modelled as the states of the expression node and are recognized by identifying the maximum of beliefs of all states. We have also proposed a novel method for BBN parameter inference using a statistical feature model that can beconsidered as an extension of SFAM. Finally, in order to enrich information usedfor 3D face analysis, and particularly 3D facial expression recognition, we have also elaborated a 3D face feature, named SGAND, to characterize the geometry property of a point on 3D face mesh using its surrounding points.The effectiveness of all these methods has been evaluated on FRGC, BU3DFEand Bosphorus datasets for facial landmarking as well as BU3DFE and Bosphorus datasets for facial activity (expression and action unit) recognition.

A framework for understanding the social responses to virtual humans suggests that human-like characteristics (e.g., facial expressions, voice, expression of emotion) act as cues that lead a person to place the agent into the category "human" and thus, elicit social responses. Given this framework, this research was designed to answer two outstanding questions that had been raised in the research community (Moon&Nass, 2000): 1) If a virtual human has more human-like characteristics, will it elicit stronger social responses from people? 2) How do the human-like characteristics interact in terms of the strength of social responses? Two social psychological (social facilitation and politeness norm) experiments were conducted to answer these questions. The first experiment investigated whether virtual humans can evoke a social facilitation response and how strong that response is when participants are given different cognitive tasks (e.g., anagrams, mazes, modular arithmetic) that vary in difficulty. They did the tasks alone, in the company of another person, or in the company of a virtual human that varied in terms of features. The second experiment investigated whether people apply politeness norms to virtual humans. Participants were tutored and quizzed either by a virtual human tutor that varied in terms of features or a human tutor. Participants then evaluated the tutor's performance either directly by the tutor or indirectly via a paper and pencil questionnaire. Results indicate that virtual humans can produce social facilitation not only with facial appearance but also with voice recordings. In addition, performance in the presence of voice synced facial appearance seems to elicit stronger social facilitation (i.e., no statistical difference compared to performance in the human presence condition) than in the presence of voice only or face only. Similar findings were observed with the politeness norm experiment. Participants who evaluated their tutor directly reported the tutor's performance more favorably than participants who evaluated their tutor indirectly. In addition, this valence toward the voice synced facial appearance had no statistical difference compared to the valence toward the human tutor condition. The results suggest that designers of virtual humans should be mindful about the social nature of virtual humans.

Nos travaux se sont articulés autour du problème de recouvrement temporel rencontré lors de l'estimation des potentiels évoqués. Il constitue, plus particulièrement, une limitation majeure pour l'estimation des potentiels évoqués par les fixations ou saccades oculaires lors d'une expérience en enregistrement conjoint EEG et oculométrie. En effet, la méthode habituellement utilisée pour estimer ces potentiels évoqués, la méthode par simple moyennage du signal synchronisé sur l'évènement d'intérêt, suppose qu'il y a un seul potentiel évoqué par essai. Or selon les intervalles inter-stimuli, cette hypothèse n'est pas toujours vérifiée. Ceci est d'autant plus vrai dans le contexte des potentiels évoqués par fixations ou saccades oculaires, les intervalles entre ceux-ci n'étant pas contrôlés par l'expérimentateur et pouvant être plus courts que les latences des potentiels d'intérêt. Le fait que cette hypothèse ne soit pas vérifiée donne une estimation biaisée du potentiel évoqué du fait des recouvrements entre les potentiels évoqués.Nous avons donc utilisé le Modèle Linéaire Général (GLM), méthode de régression linéaire bien connue, pour estimer les potentiels évoqués par les mouvements oculaires afin de répondre à ce problème de recouvrement. Tout d'abord, nous avons introduit, dans ce modèle, un terme de régularisation au sens de Tikhonov dans l'optique d'améliorer le rapport signal sur bruit de l'estimation pour un faible nombre d'essais. Nous avons ensuite comparé le GLM à l'algorithme ADJAR dans un contexte d'enregistrement conjoint EEG et oculométrie lors d'une tâche d'exploration visuelle de scènes naturelles. L'algorithme ADJAR ("ADJAcent Response") est un algorithme classique d'estimation itérative des recouvrements temporels développé en 1993 par M. Woldorff. Les résultats ont montré que le GLM était un modèle plus flexible et robuste que l'algorithme ADJAR pour l'estimation des potentiels évoqués par les fixations oculaires. Puis, deux configurations du GLM ont été comparées pour l'estimation du potentiel évoqué à l'apparition du stimulus et du potentiel évoqué par les fixations au début de l'exploration. Toutes deux prenaient en compte les recouvrements entre potentiels évoqués mais l'une distinguait également le potentiel évoqué par la première fixation de l'exploration du potentiel évoqué par les fixations suivantes. Il est apparu que le choix de la configuration du GLM était un compromis entre la qualité de l'estimation des potentiels et les hypothèses émises sur les processus cognitifs sous-jacents.Enfin, nous avons conduit de bout en bout une expérience d'envergure en enregistrement conjoint EEG et oculométrie portant sur l'exploration des expressions faciales émotionnelles naturelles statiques et dynamiques. Nous avons présenté les premiers résultats pour la modalité statique. Après avoir discuté de la méthode d'estimation des potentiels évoqués selon l'impact des mouvements oculaires sur leur fenêtre de latence, nous avons étudié l'effet du type d'émotion. Nous avons trouvé des modulations du potentiel différentiel EPN (Early Posterior Negativity), entre 230 et 350 ms après l'apparition du stimulus et du potentiel LPP (Late Positivity Potential), entre 400 et 600 ms après l'apparition du stimulus. Nous avons également observé des variations du potentiel évoqué par les fixations oculaires. Pour le potentiel LPP, qui est un marqueur de la reconnaissance consciente de l'émotion, nous avons montré qu'il était important de dissocier l'information qui est immédiatement encodée à l'apparition du stimulus émotionnel, de celle qui est apportée à l'issue de la première fixation. Cela met en évidence un motif d'activation différencié pour les stimuli émotionnels à valence négative ou à valence positive. Cette différenciation est en accord avec l'hypothèse d'un traitement plus rapide des stimuli émotionnels à valence négative que des stimuli émotionnels à valence positive. / Our research focuses on the issue of overlapping for evoked potential estimation. More specifically, this issue is a significant limitation for Eye-Fixation Related Potentials and Eye-Saccade Related Potentials estimations during a joint EEG and eye-tracking recording. Indeed, the usual estimation, by averaging the signal time-locked to the event of interest, is based on the assumption that a single evoked potential occurs during a trial. However, depending on the inter-stimulus intervals, this assumption is not always verified. This is especially the case in the context of Eye-Fixation Related Potentials and Eye-Saccade Related Potentials, given the fact that the intervals between fixations (or saccades) are not controlled by the experimenter and can be shorter than the latencies of the potentials of interest.The fact that this assumption is not verified gives a distorted estimate of the evoked potential due to overlaps between the evoked potentials.We have therefore used the Linear Model (GLM), a well-known linear regression method, to estimate the potentials evoked by ocular movements in order to take into account overlaps. First, we decided to introduce a term of Tikhonov regularization into this model in order to improve the signal-to-noise ratio of the estimate for a small number of trials. Then, we compared the GLM to the ADJAR algorithm in a context of joint EEG and eye-tracking recording during a task of visual exploration of natural scenes. The ADJAR ("ADJAcent Response") algorithm is an algorithm for iterative estimation of temporal overlaps developed in 1993 by M. Woldorff. The results showed that the GLM model was more flexible and robust than the ADJAR algorithm in estimating Eye-Fixation Related Potentials. Further, two GLM configurations were compared in their estimation of evoked potential at the onset of the stimulus and the eye-fixation related potential at the beginning of the testing. Both configurations took into account the overlaps between evoked potentials, but one additionally distinguished the potential evoked by the first fixation of the exploration from the potential evoked by the following fixations. It became clear that the choice of the GLM configuration was a compromise between the estimation quality of the potentials and the assumptions about the underlying cognitive processes.Finally, we conducted an extensive joint EEG and eye-tracking experiment on the exploration of static and dynamic natural emotional facial expressions. We presented the first results for the static modality. After discussing the estimation method of the evoked potentials according to the impact of the ocular movements on their latency window, we studied the influence of the type of emotion. We found modulations of the differential EPN (Early Posterior Negativity) potential, between 230 and 350 ms after the stimulus onset and the Late Positivity Potential (LPP) , between 400 and 600 ms after the stimulus onset. We also observed variations for the Eye-Fixation Related Potentials. Regarding the LPP component, a marker of conscious recognition of emotion, we have shown that it is important to dissociate information that is immediately encoded at the onset of the emotional stimulus from information encoded at the first fixations. This shows a differentiated pattern of activation according to the emotional stimulus valence. This differentiation is in agreement with the hypothesis of a faster treatment of negative emotional stimuli than of positive emotional stimuli.

The aim of this study is to explore the patterns of emotion recognition in Mexican bilinguals using the JACFEE (Matsumoto & Ekman, 1988). Previous cross cultural research has documented high agreement in judgments of facial expressions of emotion, however, none of the previous studies has included data from Mexican culture. Participants were 229 Mexican college students (mean age 21.79). Results indicate that each of the seven universal emotions: anger, contempt, disgust, fear, happiness, sadness and surprise was recognized by the participants above chance levels (p < .001), regardless of the gender or ethnicity of the posers. These findings replicate reported data on the high cross cultural agreement in emo- tion recognition (Ekman, 1994) and contribute to the increasing body of evidence regardingthe universality of emotions. / Este estudio presenta un meta-análisis sobre la relación entre los valores de Schwartz y el bienestar subjetivo en distintos contextos culturales, con estudiantes, sus familiares e inmigrantes en España. Los resultados confirman una asociación significativa entre los valores y el bienestar. Auto trascendencia y apertura al cambio, y con menor intensidad, conservación, se asocian positivamente con mayor bienestar. Auto trascendencia se asocia con felicidad y satisfacción de forma positiva no homogénea, siendo los inmigrantes quienes presentan medias más bajas. Apertura al cambio se asocia con felicidad, siendo más fuerte la asociación en inmigrantes que en estudiantes. Los valores conservacionistas se asocian homogéneamente. Un segundo estudio sobre criterios de salud psicosocial y bienestar subjetivo -analizando un país sudamericano colectivista y jerárquico como Brasil, y otro europeo más individualista e igualitario como España- confirma que los valores conservacionistas, así como los de apertura al cambio y auto trascendencia, son deseables y favorecen el bienestar.

Magister Scientiae - MSc (Computer Science) / A machine translation system that can convert South African Sign Language (SASL) video to audio or text and vice versa would be bene cial to people who use SASL to communicate. Five fundamental parameters are associated with sign language gestures, these are: hand location; hand orientation; hand shape; hand movement and facial expressions. The aim of this research is to recognise facial expressions and to compare both feature descriptors and machine learning techniques. This research used the Design Science Research (DSR) methodology. A DSR artefact was built which consisted of two phases. The rst phase compared local binary patterns (LBP), compound local binary patterns (CLBP) and histogram of oriented gradients (HOG) using support vector machines (SVM). The second phase compared the SVM to arti cial neural networks (ANN) and random forests (RF) using the most promising feature descriptor|HOG|from the rst phase. The performance was evaluated in terms of accuracy, robustness to classes, robustness to subjects and ability to generalise on both the Binghamton University 3D facial expression (BU-3DFE) and Cohn Kanade (CK) datasets. The evaluation rst phase showed HOG to be the best feature descriptor followed by CLBP and LBP. The second showed ANN to be the best choice of machine learning technique closely followed by the SVM and RF.

Made available in DSpace on 2016-06-02T20:30:05Z (GMT). No. of bitstreams: 1 4479.pdf: 1460927 bytes, checksum: 8f694b1f5e420acf8e1ad43d8629b715 (MD5) Previous issue date: 2012-04-20 / Financiadora de Estudos e Projetos / The ability of expressing emotions via facial expressions is an indispensable component of some required childhood social skills. Therefore, facial expressions are crucial for successful social relations and the quality of life of both typically developing children and persons with special educational needs, such as visual impaired children. As facial expression of emotions and social skills are profoundly connected, there is the demand of programming interventions directed to maintaining, modulating and enhancing facial expressions topographically and functionally. In order to produce interventions socially valid and effective, planning programs which produce indicators of external and internal validity is of utmost importance. In other words, interventions must be carried out with reliable measures and well-delimited procedures so as the acquired repertoire may be generalized and maintained. In view of social, methodological and empirical issues that underlie those areas (facial expressions and social skills), the present study aimed at evaluating the impact of a program which trained the facial expression of emotions on the social skills repertoire of blind, low vision and sighted children in (1) acquiring, enhancing and maintaining the discrimination of characteristic facial signs of each basic emotion; (2) acquiring, enhancing and maintaining facial expression of basic emotions using photo and video registers; (3) the quality of facial expressions of basic emotions registered by photos; (4) the ability of emotionally express themselves through their face, actions and voice, evaluated by parents and teachers; (5) acquiring, enhancing and maintaining their social skills, according to their self-evaluation, as well as parents and teachers evaluation. A single-case research design with pretest and posttest, multiple probes and replications intra and inter subjects was adopted. Participants were 3 blind children, 3 children with low vision and 3 sighted children. The intervention program was carried out individually and lasted for 21 sessions. Moreover, the evaluation was carried out by 2 judges, the child s parents and teachers and the children themselves. The Social Skills Rating System (SSRS-BR), Checklist for Evaluation and Probe of Emotional Expressiveness, Checklist for Assessing Facial and Emotional Expression, Inventory of Facial Expression of Emotions by Pictures and Films, Protocol for Assessing the Quality of Facial Expression of Emotions, and, Protocol for Emotional Expressiveness Assessment by Facial Expressions and Non-Verbal Components of Emotions were the instruments used for the evaluation. Except for the SSRSBR, all of the instruments were especially created for the present study. Data analysis was carried out as the following: descriptive statistical analysis was performed for each individual (subjects as their own control) and JT Method (clinical significance and reliable change index) was used in order to assess SSRS-BR data. Results indicated that blind, low vision and typically developing children (ordered from the former to the latter) presented more difficulties in discriminating the facial signs characteristic of six basic emotions during the evaluation, which took place prior to the intervention. The percentage of correct answers of all children in the probes after the intervention was between 83,3% and 100%. In addition to that, parents, teachers and judges evaluated the facial expression repertoire of participants as having improved and maintained itself after the intervention, as well as the quality of facial expressiveness. All participants improved their general score in social skills, with some reliable positive changes (improvement) and clinically significant changes, evidencing the enhancement of the participants repertoire observed after the intervention. In summary, the intervention program was effective for improving and maintaining the facial expression of emotions and some classes of social skills, especially those related to emotional expressiveness. / A expressividade facial de emoções é considerada um dos componentes indispensáveis de algumas classes de habilidades sociais imprescindíveis na infância e, portanto, essenciais para a qualidade de vida e das relações sociais, seja das pessoas com necessidades educacionais especiais, como as crianças com deficiência visual, ou com desenvolvimento típico. Quando se considera esta relação entre a expressão facial de emoções e o repertório de habilidades sociais, torna-se necessário programar intervenções direcionadas para manutenção, modulação e aprimoramento topográfico e funcional da expressividade de emoções pela face, na sua relação com as diferentes classes de habilidades sociais. Para que estas intervenções sejam efetivas e socialmente válidas, é importante planejar programas que produzam indicadores de validade interna e externa, ou seja, com confiabilidade das medidas e com procedimentos bem delimitados para generalização e manutenção do repertório adquirido. Considerando as questões sociais, metodológicas e empíricas que permeiam estas duas áreas do conhecimento (expressões faciais de emoções e habilidades sociais), o presente estudo teve como objetivo avaliar o impacto de um programa de treinamento de expressão facial de emoções, na interface com as habilidades sociais, sobre o repertório de crianças cegas, com baixa visão e videntes na (1) aquisição, aprimoramento e manutenção da discriminação dos sinais faciais característicos de cada emoção básica; (2) aquisição, aprimoramento e manutenção da expressão facial de emoções básicas, registrada por meio de fotografias e filmagens; (3) qualidade das expressões faciais de emoções básicas, registradas por meio de fotografias; (4) expressividade emocional pela face, gestos e voz, avaliado pelos pais e professores; (5) aquisição, aprimoramento e manutenção das habilidades sociais, conforme a autoavaliação e avaliação pelos pais e professores. Adotando-se o delineamento pré e pósteste com sujeito único, com múltiplas sondagens e replicações intra e entre sujeitos com diferentes graus de comprometimento visual, o estudo foi conduzido com três crianças cegas, três com baixa visão e três videntes. O programa de intervenção foi em formato individual, com 21 sessões, que tinham uma estrutura semelhante, mas com flexibilidade para alterações de procedimentos e dos materiais diferenciados e adaptados às características, recursos, dificuldades e especificidades de cada criança. A avaliação foi realizada por dois juízes, além das próprias crianças, seus pais e professores, que avaliaram o repertório da criança por meio do Sistema de Avaliação de Habilidades Sociais (SSRS-BR); Roteiro de Sondagem e Avaliação da Expressividade Emocional; Roteiro de Avaliação das Expressões Faciais de Emoções; Ficha de Avaliação das Expressões Faciais de Emoções por Fotografias e Filmagens; Protocolo de Avaliação da Qualidade das Expressões Faciais de Emoções; e, Protocolo de Avaliação da Expressividade Facial de Emoção e dos demais Componentes Não- Verbais. O tratamento dos dados ocorreu por meio de estatística descritiva, para análises individuais (sujeito como próprio controle), e pelo Método JT (significância clínica e índice de mudança confiável) para os dados do SSRS-BR. Os dados do estudo apontaram que as crianças cegas, seguidas pelas com baixa visão e, depois, pelas videntes, apresentaram mais dificuldades em discriminar os sinais faciais característicos das seis emoções básicas nas avaliações que antecederam a intervenção. Nas sondagens que ocorreram após a intervenção, a porcentagem de acertos de todas as crianças foi entre 83,3% e 100%. Além disso, o repertório de expressão facial de emoções de todos os participantes, avaliado pelos pais, professoras e juízes, foi aprimorado e mantido após o programa de intervenção, assim como a qualidade da expressividade de emoções pela face. No caso do repertório de habilidades sociais, todos os participantes obtiveram ganhos na pontuação geral, com algumas mudanças positivas confiáveis (melhora) e mudanças clinicamente significativas, evidenciando o aprimoramento após a intervenção. Conclui-se, portanto, que o programa de intervenção foi efetivo para o aprimoramento e manutenção da expressão facial de emoções e de algumas classes de habilidades sociais, principalmente aquelas relacionadas a expressividade emocional.

De plus en plus d’applications ont pour objectif d’automatiser l’analyse des comportements humains afin d’aider les experts qui réalisent actuellement ces analyses. Cette thèse traite de l’analyse des expressions faciales qui fournissent des informations clefs sur ces comportements.Les travaux réalisés portent sur une solution innovante, basée sur l’organisation des expressions, permettant de définir efficacement une expression d’un visage.Nous montrons que l’organisation des expressions, telle que définie, est universelle : une expression est alors caractérisée par son intensité et sa position relative par rapport aux autres expressions. La solution est comparée aux méthodes classiques et montre une augmentation significative des résultats de reconnaissance sur 14 expressions non basiques. La méthode a été étendue à des sujets inconnus. L’idée principale est de créer un espace d’apparence plausible spécifique à la personne inconnue en synthétisant ses expressions basiques à partir de déformations apprises sur d’autres sujets et appliquées sur le neutre du sujet inconnu. La solution est aussi mise à l’épreuve dans un environnement multimodal dont l’objectif est la reconnaissance d’émotions lors de conversations spontanées. Notre méthode a été mise en œuvre dans le cadre du challenge international AVEC 2012 (Audio/Visual Emotion Challenge) où nous avons fini 2nd, avec des taux de reconnaissance très proches de ceux obtenus par les vainqueurs. La comparaison des deux méthodes (la nôtre et celles des vainqueurs) semble montrer que l’extraction des caractéristiques pertinentes est la clef de tels systèmes. / More and more applications aim at automating the analysis of human behavior to assist or replace the experts who are conducting these analyzes. This thesis deals with the analysis of facial expressions, which provide key information on these behaviors.Our work proposes an innovative solution to effectively define a facial expression, regardless of the morphology of the subject. The approach is based on the organization of expressions.We show that the organization of expressions, such as defined, is universal and can be effectively used to uniquely define an expression. One expression is given by its intensity and its relative position to the other expressions. The solution is compared with the conventional methods based on appearance data and shows a significant increase in recognition results of 14 non-basic expressions. The method has been extended to unknown subjects. The main idea is to create a plausible appearance space dedicated to the unknown person by synthesizing its basic expressions from deformations learned on other subjects and applied to the neutral face of the unknown subject. The solution is tested in a more comprehensive multimodal environment, whose aim is the recognition of emotions in spontaneous conversations. Our method has been implemented in the international challenge AVEC 2012 (Audio / Visual Emotion Challenge) where we finished 2nd, with recognition rates very close to the winners’ ones. Comparison of both methods (ours and the winners’ one) seems to show that the extraction of relevant features is the key to such systems.

L'analyse automatique des expressions faciales est une étape clef pour le développement d'interfaces intelligentes ou l'analyse de comportements. Toutefois, celle-ci est rendue difficile par un grand nombre de facteurs, pouvant être d'ordre morphologiques, liés à l'orientation du visage ou à la présence d'occultations. Nous proposons des adaptations des Random Forest permettant d' adresser ces problématiques:- Le développement des Pairwise Conditional Random Forest, consistant en l'apprentissage de modèles à partir de paires d'images expressives. Les arbres sont de plus conditionnés par rapport à l'expression de la première image afin de réduire la variabilité des transitions. De plus, il est possible de conditionner les arbres en rapport avec une estimation de la pose du visage afin de permettre la reconnaissance quel que soit le point de vue considéré.- L'utilisation de réseaux de neurones auto-associatifs pour modéliser localement l'apparence du visage. Ces réseaux fournissent une mesure de confiance qui peut être utilisée dans le but de pondérer des Random Forests définies sur des sous-espaces locaux du visage. Ce faisant, il est possible de fournir une prédiction d'expression robuste aux occultations partielles du visage.- Des améliorations du récemment proposé algorithme des Neural Decision Forests, lesquelles consistent en une procédure d'apprentissage simplifiée, ainsi qu'en une évaluation "greedy" permettant une évaluation plus rapide, avec des applications liées à l'apprentissage en ligne de représentations profondes pour la reconnaissance des expressions, ainsi que l'alignement de points caractéristiques. / Automatic face analysis is a key to the development of intelligent human-computer interaction systems and behavior understanding. However, there exist a number of factors that makes face analysis a difficult problem. This include morphological differences between different persons, head pose variations as well as the possibility of partial occlusions. In this PhD, we propose a number of adaptations of the so-called Random Forest algorithm to specifically adress those problems. Mainly, those improvements consist in:– The development of a Pairwise Conditional Random Forest framework, that consists in training Random Forests upon pairs of expressive images. Pairwise trees are conditionned on the expression label of the first frame of a pair to reduce the ongoing expression transition variability. Additionnally, trees can be conditionned upon a head pose estimate to peform facial expression recognition from an arbitrary viewpoint.– The design of a hierarchical autoencoder network to model the local face texture patterns. The reconstruction error of this network provides a confidence measurement that can be used to weight Randomized decision trees trained on spatially-defined local subspace of the face. Thus, we can provide an expression prediction that is robust to partial occlusions.– Improvements over the very recent Neural Decision Forests framework, that include both a simplified training procedure as well as a new greedy evaluation procedure, that allows to dramatically improve the evaluation runtime, with applications for online learning and, deep learning convolutional neural network-based features for facial expression recognition as well as feature point alignement.