Micro-expressions are brief facial expressions of emotion (40 to 500 ms) that are posited to represent true reflections of an individual’s emotional state that have 'leaked’ through voluntary attempts to neutralize or mask the involuntary expression. As such, correct recognition can have important benefits. The Japanese and Caucasian Brief Affect Recognition Task (JACBART) has been proposed as the standardized measure of affect recognition capabilities with micro-expression durations (i.e., facial expressions lasting less than 500 ms). In this paradigm target expressions of emotion are briefly presented between two neutral expressions. However, limited research has explored the temporal thresholds and the various factors that may influence performance in a JACBART paradigm. In three studies, the current thesis sought to determine the effects of a forward mask with a variable duration (Study 1), the inclusion/exclusion of a ‘neutral’ response category (Study 2), and expressions portrayed at lower intensities (Study 3). Although a variable-duration forward mask was found to have little effect on performance, significant effects were observed for the inclusion of a ‘neutral’ response option and when reducing the expression intensity. In addition, a trend was observed across all three studies that demonstrated a recognition advantage for expressions of happiness and surprise. Performances for these two expressions exceeded the psychometric threshold with durations of as little as 5 to 10 ms, whereas presentation times as long as 113 ms were necessary to elicit above-threshold recognition rates with negative emotions (i.e., anger, disgust, fear, and sadness). Altogether, the current findings present some methodological considerations for studies interested in measuring brief affect recognition with a JACBART paradigm. More generally, they expand our understanding of how various relevant factors affect the speed at which facial expressions can be processed.

Micro-expressions are brief facial expressions that last for 500 milliseconds or less and show the true emotional state of an individual when he or she is displaying a false emotional state. There are currently 2 different methods to train individuals to recognize micro-expressions-picture-based and video-based. Numerous organizations use micro-expression training as part of a deception detection program, but little research has been conducted on training outcomes, and no research has investigated the difference between the methods. In this quantitative study based on Darwin's theory of the universality of emotional expression, a control group experimental design was used to determine if there is a difference in training outcomes, as measured by post-training accuracy rates of overall and emotion-specific micro-expression identification, between the 2 current micro-expression training methods and no training. A total of 196 participants recruited from Amazon's Mechanical Turk community were randomly assigned to a picture-based training, video-based training, or no training control group. The online training and post-training test were delivered via a computer-based training platform. MANOVA, ANOVA and t-tests were run to determine the differences between the groups. Results indicated that participants in both picture-based and video-based training groups showed a significant increase in their ability to recognize micro-expressions compared to those in the no training group, but did not differ from each other. The study provides an increased understanding of micro-expression training outcomes that may contribute to the training of numerous law enforcement, security, and human resources professionals.

Therapeutic alliance and its relationship to client nonverbal behavior, specifically facial expressions, were examined. Therapist interpretation of the client nonverbal behavior, or affect, influences the therapeutic alliance and process. Based on a sample of clients from a graduate school therapy training facility, results suggest therapist training in facial expressions, and how they relate to client emotion, improve the therapeutic alliance between therapist and client. After a micro-expression training for therapists, clients reported higher life functioning on the Outcome Rating Scale (ORS) and an improved therapeutic alliance on the Session Rating Scale (SRS). Overall, these findings support the benefit of incorporating micro-expression training into therapy instruction.

Abstract The face plays an important role in our social interactions as it conveys rich sources of information. We can read a lot from one face image, but there is also information we cannot perceive without special devices. The thesis concerns using computer vision methodologies to analyse two kinds of subtle facial information that can hardly be perceived by naked eyes: the micro-expression (ME), and the heart rate (HR). MEs are rapid, involuntary facial expressions which reveal emotions people do not intend to show. It is difficult for people to perceive MEs as they are too fast and subtle, thus automatic ME analysis is valuable work which may lead to important applications. In the thesis, the progresses of ME studies are reviewed, and four parts of work are described. 1) We introduce the first spontaneous ME database, the SMIC. The lacking of data is hindering ME analysis research, as it is difficult to collect spontaneous MEs. The protocol for inducing and annotating SMIC is introduced to help future ME collections. 2) A framework including three features and a video magnification process is introduced for ME recognition, which outperforms other state-of-the-art methods on two ME databases. 3) An ME spotting method based on feature difference analysis is described, which can spot MEs from spontaneous long videos. 4) An automatic ME analysis system (MESR) was proposed for firstly spotting and then recognising MEs. The HR is an important indicator of our health and emotional status. Traditional HR measurements require skin-contact which cannot be applied remotely. We propose a method which can counter for illumination changes and head motions and measure HR remotely from color facial videos. We also apply the method for solving the face anti-spoofing problem. We show that the pulse-based feature is more robust than traditional texture-based features against unseen mask spoofs. We also show that the proposed pulse-based feature can be combined with other features to build a cascade system for detecting multiple types of attacks. At last, we summarize the contributions of the work, and propose future plans about ME and HR studies based on limitations of the current work. It is also planned to combine the ME and HR (maybe also other subtle signals from face) to build a multimodal system for affective status analysis. / Tiivistelmä Kasvot ovat monipuolinen informaatiolähde ja keskeinen ihmisten välisessä vuorovaikutuksessa. Pystymme päättelemään paljon yhdestäkin kasvokuvasta, mutta kasvoissa on paljon tietoa, jota ei pysty irrottamaan ilman erityiskeinoja. Tässä työssä analysoidaan konenäöllä ihmiselle vaikeasti havaittavaa tietoa: mikroilmeitä ja sydämen sykettä. Tahdosta riippumattomat mikroilmeet paljastavat tunteita, joita ihmiset pyrkivät piilottamaan. Mikroilmeiden havaitseminen on vaikeaa niiden nopeuden ja pienuuden vuoksi, joten automaattinen analyysi voi johtaa uusiin merkittäviin sovelluksiin. Tämä työ tarkastelee mikroilmetutkimuksen edistysaskeleita ja sisältää neljä uutta tulosta. 1) Spontaanien mikroilmeiden tietokanta (Spontaneous MIcroexpression Corpus, SMIC). Spontaanien mikroilmeiden aiheuttaminen datan saamiseksi on oma haasteensa. SMIC:n keräämisessä ja mikroilmeiden annotoinnissa käytetty menettely on kuvattu myöhemmän datan keruun ohjeistukseksi. 2) Aiempia mikroilmeiden tunnistusmenetelmiä paremmaksi kahden testitietokannan avulla todennettu ratkaisu, joka käyttää kolmea eri piirrettä ja videon suurennusta. 3) Piirre-eroanalyysiin perustuva mikroilmeiden havaitsemismenetelmä, joka havaitsee ne pitkistä realistisista videoista. 4) Automaattinen analyysijärjestelmä (Micro-Expression Spotting and Recognition, MESR), jossa mikroilmeet havaitaan ja tunnistetaan. Sydämen syke on tärkeä terveyden ja tunteiden indikoija. Perinteiset sykkeenmittausmenetelmät vaativat ihokontaktia, eivätkä siten toimii etäältä. Tässä työssä esitetään sykkeen videolta pienistä värimuutoksista mittaava menetelmä, joka sietää valaistusmuutoksia ja sallii pään liikkeet. Menetelmä on monikäyttöinen ja sen sovelluksena kuvataan todellisten kasvojen varmentaminen sykemittauksella. Tulokset osoittavat sykepiirteiden toimivan perinteisiä tekstuuripiirteitä paremmin uudenlaisia naamarihuijauksia vastaan. Syketietoa voidaan myös käyttää osana sarjatyyppisissä ratkaisuissa havaitsemaan useanlaisia huijausyrityksiä. Työn yhteenveto keskittyy suunnitelmiin parantaa mikroilmeiden ja sydämen sykkeen analyysimenetelmiä nykyisen tutkimuksen rajoitteiden pohjalta. Tavoitteena on yhdistää mikroilmeiden ja sydämen sykkeen analyysit, sekä mahdollisesti muuta kasvoista saatavaa tietoa, multimodaaliseksi affektiivisen tilan määrittäväksi ratkaisuksi.