Dans un monde de plus en plus ouvert, les flux de personnes sont amenés à exploser dans les prochaines années. Fluidifier et contrôler ces flux, tout en respectant de fortes contraintes sécuritaires, apparaît donc comme un élément clef pour favoriser le dynamisme économique mondial. Cette gestion des flux passe principalement par la connaissance et la vérification de l’identité des personnes. Pour son aspect pratique et a priori sécurisé, la biométrie, et en particulier celle des empreintes digitales, s’est imposée comme une solution efficace, et incontournable. Néanmoins, elle souffre de deux sévères limitations. La première concerne les mauvaises performances obtenues avec des doigts détériorés. Ces détériorations peuvent être involontaires (travailleurs manuels par exemple), ou bien volontaires, à des fins d’anonymisation. La deuxième concerne les failles de sécurité des capteurs. En particulier, ils sont vulnérables à des attaques avec de fausses empreintes, réalisées par des personnes mal intentionnées dans un but d’usurpation d’identité. D’après nous, ces limitations sont dues à la faible quantité d’information exploitée par les capteurs usuels. Elle se résume souvent à une simple image de la surface du doigt. Pourtant, la complexité biologique des tissus humains est telle qu’elle offre une information très riche, unique, et difficilement reproductible. Nous avons donc proposé une approche d’imagerie, basée sur la Tomographique par Cohérence Optique, un capteur 3D sans contact, permettant de mesurer finement cette information. L’idée majeure de la thèse consiste à étudier divers moyens de l’exploiter, afin de rendre la biométrie plus robuste et vraiment sécurisée / In an increasingly open world, the flows of people are brought to explode in the coming years. Facilitating, streamlining, and managing these flows, by maintaining strict security constraints, therefore represent a key element for the global socio-economic dynamism. This flows management is mainly based on knowledge and verification of person identity. For its practicality and a priori secured, biometrics, in particular fingerprints biometrics, has become an effective and unavoidable solution.Nevertheless, it still suffers from two severe limitations. The first one concerns the poor performances obtained with damaged fingers. This damage can be involuntary (e.g. manual workers) or volunteers, for purposes of anonymity. The second limitation consists in the vulnerability of the commonly used sensors. In particular, they are vulnerable to copies of stolen fingerprints, made by malicious persons for identity theft purpose. We believe that these limitations are due to the small amount of information brought by the usual biometric sensors. It often consists in a single print of the finger surface. However, the biological complexity of human tissue provides rich information, unique to each person, and very difficult to reproduce. We therefore proposed an imaging approach based on Optical Coherence Tomography (OCT), a 3D contactless optical sensor, to finely measure this information. The main idea of the thesis is therefore to explore novel ways to exploit this information in order to make biometrics more robust and truly secured. In particular, we have proposed and evaluated different fingerprint imaging methods, based on the phase of the OCT signal

Visage est l’une des meilleures biométries pour la reconnaissance de l’identité de personnes, car l’identification d’une personne par le visage est l’habitude instinctive humaine, et l’acquisition de données faciales est naturelle, non intrusive et bien acceptée par le public. Contrairement à la reconnaissance de visage par l’image 2D sur l’apparence, la reconnaissance de visage en 3D sur la forme est théoriquement plus stable et plus robuste à la variance d’éclairage, aux petits changements de pose de la tête et aux cosmétiques pour le visage. Spécifiquement, les courbures sont les plus importants attributs géométriques pour décrire la forme géométrique d’une surface. Elles sont bénéfiques à la caractérisation de la forme du visage qui permet de diminuer l’impact des variances environnementales. Cependant, les courbures traditionnelles ne sont définies que sur des surfaces lisses. Il est donc nécessaire de généraliser telles notions sur des surfaces discrètes, par exemple des visages 3D représenté par maillage triangulaire, et d’évaluer leurs performances en reconnaissance de visage 3D. En outre, même si un certain nombre d’algorithmes 3D FR avec une grande précision sont disponibles, le coût d’acquisition de telles données de haute résolution est difficilement acceptable pour les applications pratiques. Une question majeure est donc d’exploiter les algorithmes existants pour la reconnaissance de modèles à faible résolution collecté avec l’aide d’un nombre croissant de caméras consommateur de profondeur (Kinect). Le dernier problème, mais non le moindre, est la menace sur sécurité des systèmes de reconnaissance de visage 3D par les attaques de masque fabriqué. Cette thèse est consacrée à l’étude des attributs géométriques, des mesures de courbure principale, adaptées aux maillages triangulaires, et des schémas de reconnaissance de visage 3D impliquant des telles mesures de courbure principale. En plus, nous proposons aussi un schéma de vérification sur la reconnaissance de visage 3D collecté en comparant des modèles de résolutions hétérogènes équipement aux deux résolutions, et nous évaluons la performance anti-spoofing du système de RF 3D. Finalement, nous proposons une biométrie système complémentaire de reconnaissance veineuse de main basé sur la détection de vivacité et évaluons sa performance. Dans la reconnaissance de visage 3D par la forme géométrique, nous introduisons la généralisation des courbures principales conventionnelles et des directions principales aux cas des surfaces discrètes à maillage triangulaire, et présentons les concepts des mesures de courbure principale correspondants et des vecteurs de courbure principale. Utilisant ces courbures généralisées, nous élaborons deux descriptions de visage 3D et deux schémas de reconnaissance correspondent. Avec le premier descripteur de caractéristiques, appelé Local Principal Curvature Measures Pattern (LPCMP), nous générons trois images spéciales, appelée curvature faces, correspondant à trois mesures de courbure principale et encodons les curvature faces suivant la méthode de Local Binary Pattern. Il peut décrire la surface faciale de façon exhaustive par l’information de forme locale en concaténant un ensemble d’histogrammes calculés à partir de petits patchs dans les visages de courbure. Dans le deuxième système de reconnaissance de visage 3D sans enregistrement, appelée Principal Curvature Measures based meshSIFT descriptor (PCM-meshSIFT), les mesures de courbure principales sont d’abord calculées dans l’espace de l’échelle Gaussienne, et les extrèmes de la Différence de Courbure (DoC) sont définis comme les points de caractéristique. Ensuite, nous utilisons trois mesures de courbure principales et leurs vecteurs de courbure principaux correspondants pour construire trois descripteurs locaux pour chaque point caractéristique, qui sont invariants en rotation. [...] / Face is one of the best biometrics for person recognition related application, because identifying a person by face is human instinctive habit, and facial data acquisition is natural, non-intrusive, and socially well accepted. In contrast to traditional appearance-based 2D face recognition, shape-based 3D face recognition is theoretically more stable and robust to illumination variance, small head pose changes, and facial cosmetics. The curvatures are the most important geometric attributes to describe the shape of a smooth surface. They are beneficial to facial shape characterization which makes it possible to decrease the impact of environmental variances. However, exiting curvature measurements are only defined on smooth surface. It is required to generalize such notions to discrete meshed surface, e.g., 3D face scans, and to evaluate their performance in 3D face recognition. Furthermore, even though a number of 3D FR algorithms with high accuracy are available, they all require high-resolution 3D scans whose acquisition cost is too expensive to prevent them to be implemented in real-life applications. A major question is thus how to leverage the existing 3D FR algorithms and low-resolution 3D face scans which are readily available using an increasing number of depth-consumer cameras, e.g., Kinect. The last but not least problem is the security threat from spoofing attacks on 3D face recognition system. This thesis is dedicated to study the geometric attributes, principal curvature measures, suitable to triangle meshes, and the 3D face recognition schemes involving principal curvature measures. Meanwhile, based on these approaches, we propose a heterogeneous cross-resolution 3D FR scheme, evaluate the anti-spoofing performance of shape-analysis based 3D face recognition system, and design a supplementary hand-dorsa vein recognition system based on liveness detection with discriminative power. In 3D shape-based face recognition, we introduce the generalization of the conventional point-wise principal curvatures and principal directions for fitting triangle mesh case, and present the concepts of principal curvature measures and principal curvature vectors. Based on these generalized curvatures, we design two 3D face descriptions and recognition frameworks. With the first feature description, named as Local Principal Curvature Measures Pattern descriptor (LPCMP), we generate three curvature faces corresponding to three principal curvature measures, and encode the curvature faces following Local Binary Pattern method. It can comprehensively describe the local shape information of 3D facial surface by concatenating a set of histograms calculated from small patches in the encoded curvature faces. In the second registration-free feature description, named as Principal Curvature Measures based meshSIFT descriptor (PCM-meshSIFT), the principal curvature measures are firstly computed in the Gaussian scale space, and the extremum of Difference of Curvautre (DoC) is defined as keypoints. Then we employ three principal curvature measures and their corresponding principal curvature vectors to build three rotation-invariant local 3D shape descriptors for each keypoint, and adopt the sparse representation-based classifier for keypoint matching. The comprehensive experimental results based on FRGCv2 database and Bosphorus database demonstrate that our proposed 3D face recognition scheme are effective for face recognition and robust to poses and occlusions variations. Besides, the combination of the complementary shape-based information described by three principal curvature measures significantly improves the recognition ability of system. To deal with the problem towards heterogeneous cross-resolution 3D FR, we continuous to adopt the PCM-meshSIFT based feature descriptor to perform the related 3D face recognition. [...]

Abstract In the last decades, face recognition systems have evolved a lot in terms of performance. As a result, this technology is now considered as mature and is applied in many real world applications from border control to financial transactions and computer security. Yet, many studies show that these systems suffer from vulnerabilities to spoofing attacks, a weakness that may limit their usage in many cases. A face spoofing attack or presentation attack occurs when someone tries to masquerade as someone else by presenting a fake face in front of the face recognition camera. To protect the recognition systems against attacks of this kind, many face anti-spoofing methods have been proposed. These methods have shown good performances on the existing face anti-spoofing databases. However, their performances degrade drastically under real world variations (e.g., illumination and camera device variations). In this thesis, we concentrate on improving the generalization capabilities of the face anti-spoofing methods with a particular focus on the texture based techniques. In contrast to most existing texture based methods aiming at extracting texture features from gray-scale images, we propose a joint color-texture analysis. First, the face images are converted into different color spaces. Then, the feature histograms computed over each image band are concatenated and used for discriminating between real and fake face images. Our experiments conducted on three color spaces: RGB, HSV and YCbCr show that extracting the texture information from separated luminance chrominance color spaces (HSV and YCbCr) yields to better performances compared to gray-scale and RGB image representations. Moreover, to deal with the problem of illumination and image-resolution variations, we propose to extract this texture information from different scale images. In addition to representing the face images in different scales, the multi-scale filtering methods also act as pre-processing against factors such as noise and illumination. Although our obtained results are better than the state of the art, they are still far from the requirements of real world applications. Thus, to help in the development of robust face anti-spoofing methods, we collected a new challenging face anti-spoofing database using six camera devices in three different illumination and environmental conditions. Furthermore, we have organized a competition on the collected database where fourteen face anti-spoofing methods have been assessed and compared. / Tiivistelmä Kasvontunnistusjärjestelmien suorituskyky on parantunut huomattavasti viime vuosina. Tästä syystä tätä teknologiaa pidetään nykyisin riittävän kypsänä ja käytetään jo useissa käytännön sovelluksissa kuten rajatarkastuksissa, rahansiirroissa ja tietoturvasovelluksissa. Monissa tutkimuksissa on kuitenkin havaittu, että nämä järjestelmät ovat myös haavoittuvia huijausyrityksille, joissa joku yrittää esiintyä jonakin toisena henkilönä esittämällä kameralle jäljennöksen kohdehenkilön kasvoista. Tämä haavoittuvuus rajoittaa kasvontunnistuksen laajempaa käyttöä monissa sovelluksissa. Tunnistusjärjestelmien turvaamiseksi on kehitetty lukuisia menetelmiä tällaisten hyökkäysten torjumiseksi. Nämä menetelmät ovat toimineet hyvin tätä tarkoitusta varten kehitetyillä kasvotietokannoilla, mutta niiden suorituskyky huononee dramaattisesti todellisissa käytännön olosuhteissa, esim. valaistuksen ja käytetyn kuvantamistekniikan variaatioista johtuen. Tässä työssä yritämme parantaa kasvontunnistuksen huijauksen estomenetelmien yleistämiskykyä keskittyen erityisesti tekstuuripohjaisiin menetelmiin. Toisin kuin useimmat olemassa olevat tekstuuripohjaiset menetelmät, joissa tekstuuripiirteitä irrotetaan harmaasävykuvista, ehdotamme väritekstuurianalyysiin pohjautuvaa ratkaisua. Ensin kasvokuvat muutetaan erilaisiin väriavaruuksiin. Sen jälkeen kuvan jokaiselta kanavalta erikseen lasketut piirrehistogrammit yhdistetään ja käytetään erottamaan aidot ja väärät kasvokuvat toisistaan. Kolmeen eri väriavaruuteen, RGB, HSV ja YCbCr, perustuvat testimme osoittavat, että tekstuuri-informaation irrottaminen HSV- ja YCbCr-väriavaruuksien erillisistä luminanssi- ja krominanssikuvista parantaa suorituskykyä kuvien harmaasävy- ja RGB-esitystapoihin verrattuna. Valaistuksen ja kuvaresoluution variaation takia ehdotamme myös tämän tekstuuri-informaation irrottamista eri tavoin skaalatuista kuvista. Sen lisäksi, että itse kasvot esitetään eri skaaloissa, useaan skaalaan perustuvat suodatusmenetelmät toimivat myös esikäsittelynä sellaisia suorituskykyä heikentäviä tekijöitä vastaan kuten kohina ja valaistus. Vaikka tässä tutkimuksessa saavutetut tulokset ovat parempia kuin uusinta tekniikkaa edustavat tulokset, ne ovat kuitenkin vielä riittämättömiä reaalimaailman sovelluksissa tarvittavaan suorituskykyyn. Sen takia edistääksemme uusien robustien kasvontunnistuksen huijaamisen ilmaisumenetelmien kehittämistä kokosimme uuden, haasteellisen huijauksenestotietokannan käyttäen kuutta kameraa kolmessa erilaisessa valaistus- ja ympäristöolosuhteessa. Järjestimme keräämällämme tietokannalla myös kansainvälisen kilpailun, jossa arvioitiin ja verrattiin neljäätoista kasvontunnistuksen huijaamisen ilmaisumenetelmää.

Abstract The face plays an important role in our social interactions as it conveys rich sources of information. We can read a lot from one face image, but there is also information we cannot perceive without special devices. The thesis concerns using computer vision methodologies to analyse two kinds of subtle facial information that can hardly be perceived by naked eyes: the micro-expression (ME), and the heart rate (HR). MEs are rapid, involuntary facial expressions which reveal emotions people do not intend to show. It is difficult for people to perceive MEs as they are too fast and subtle, thus automatic ME analysis is valuable work which may lead to important applications. In the thesis, the progresses of ME studies are reviewed, and four parts of work are described. 1) We introduce the first spontaneous ME database, the SMIC. The lacking of data is hindering ME analysis research, as it is difficult to collect spontaneous MEs. The protocol for inducing and annotating SMIC is introduced to help future ME collections. 2) A framework including three features and a video magnification process is introduced for ME recognition, which outperforms other state-of-the-art methods on two ME databases. 3) An ME spotting method based on feature difference analysis is described, which can spot MEs from spontaneous long videos. 4) An automatic ME analysis system (MESR) was proposed for firstly spotting and then recognising MEs. The HR is an important indicator of our health and emotional status. Traditional HR measurements require skin-contact which cannot be applied remotely. We propose a method which can counter for illumination changes and head motions and measure HR remotely from color facial videos. We also apply the method for solving the face anti-spoofing problem. We show that the pulse-based feature is more robust than traditional texture-based features against unseen mask spoofs. We also show that the proposed pulse-based feature can be combined with other features to build a cascade system for detecting multiple types of attacks. At last, we summarize the contributions of the work, and propose future plans about ME and HR studies based on limitations of the current work. It is also planned to combine the ME and HR (maybe also other subtle signals from face) to build a multimodal system for affective status analysis. / Tiivistelmä Kasvot ovat monipuolinen informaatiolähde ja keskeinen ihmisten välisessä vuorovaikutuksessa. Pystymme päättelemään paljon yhdestäkin kasvokuvasta, mutta kasvoissa on paljon tietoa, jota ei pysty irrottamaan ilman erityiskeinoja. Tässä työssä analysoidaan konenäöllä ihmiselle vaikeasti havaittavaa tietoa: mikroilmeitä ja sydämen sykettä. Tahdosta riippumattomat mikroilmeet paljastavat tunteita, joita ihmiset pyrkivät piilottamaan. Mikroilmeiden havaitseminen on vaikeaa niiden nopeuden ja pienuuden vuoksi, joten automaattinen analyysi voi johtaa uusiin merkittäviin sovelluksiin. Tämä työ tarkastelee mikroilmetutkimuksen edistysaskeleita ja sisältää neljä uutta tulosta. 1) Spontaanien mikroilmeiden tietokanta (Spontaneous MIcroexpression Corpus, SMIC). Spontaanien mikroilmeiden aiheuttaminen datan saamiseksi on oma haasteensa. SMIC:n keräämisessä ja mikroilmeiden annotoinnissa käytetty menettely on kuvattu myöhemmän datan keruun ohjeistukseksi. 2) Aiempia mikroilmeiden tunnistusmenetelmiä paremmaksi kahden testitietokannan avulla todennettu ratkaisu, joka käyttää kolmea eri piirrettä ja videon suurennusta. 3) Piirre-eroanalyysiin perustuva mikroilmeiden havaitsemismenetelmä, joka havaitsee ne pitkistä realistisista videoista. 4) Automaattinen analyysijärjestelmä (Micro-Expression Spotting and Recognition, MESR), jossa mikroilmeet havaitaan ja tunnistetaan. Sydämen syke on tärkeä terveyden ja tunteiden indikoija. Perinteiset sykkeenmittausmenetelmät vaativat ihokontaktia, eivätkä siten toimii etäältä. Tässä työssä esitetään sykkeen videolta pienistä värimuutoksista mittaava menetelmä, joka sietää valaistusmuutoksia ja sallii pään liikkeet. Menetelmä on monikäyttöinen ja sen sovelluksena kuvataan todellisten kasvojen varmentaminen sykemittauksella. Tulokset osoittavat sykepiirteiden toimivan perinteisiä tekstuuripiirteitä paremmin uudenlaisia naamarihuijauksia vastaan. Syketietoa voidaan myös käyttää osana sarjatyyppisissä ratkaisuissa havaitsemaan useanlaisia huijausyrityksiä. Työn yhteenveto keskittyy suunnitelmiin parantaa mikroilmeiden ja sydämen sykkeen analyysimenetelmiä nykyisen tutkimuksen rajoitteiden pohjalta. Tavoitteena on yhdistää mikroilmeiden ja sydämen sykkeen analyysit, sekä mahdollisesti muuta kasvoista saatavaa tietoa, multimodaaliseksi affektiivisen tilan määrittäväksi ratkaisuksi.

The dissertation investigates the promises and challenges for application of cardiac signals in biometrics and affective computing, and noninvasive measurement of cardiac signals. We have mainly discussed two major cardiac signals: electrocardiogram (ECG), and photoplethysmogram (PPG). ECG and PPG signals hold strong potential for biometric authentications and identifications. We have shown that by mapping each cardiac beat from time domain to an angular domain using a limit cycle, intra-class variability can be significantly minimized. This is in contrary to conventional time domain analysis. Our experiments with both ECG and PPG signal shows that the proposed method eliminates the effect of instantaneous heart rate on the shape morphology and improves authentication accuracy. For noninvasive measurement of PPG beats, we have developed a systematic algorithm to extract pulse rate from face video in diverse situations using video magnification. We have extracted signals from skin patches and then used frequency domain correlation to filter out non-cardiac signals. We have developed a novel entropy based method to automatically select skin patches from face. We report beat-to-beat accuracy of remote PPG (rPPG) in comparison to conventional average heart rate. The beat-to-beat accuracy is required for applications related to heart rate variability (HRV) and affective computing. The algorithm has been tested on two datasets, one with static illumination condition and the other with unrestricted ambient illumination condition. Automatic skin detection is an intermediate step for rPPG. Existing methods always depend on color information to detect human skin. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. We have used LBP lacunarity based micro-textures features and a region growing algorithm to find skin pixels in an image. Our experiment shows that the proposed method is applicable universally to any image including near infra-red images. This finding helps to extend the domain of many application including rPPG. To the best of our knowledge, this is first such method that is independent of color cues. / Ph. D. / The heart is an integral part of the human body. With every beat, the heart continuously pumps oxygen-enriched blood to providing fuel to our cells and thus enabling life. The heartbeat is initiated by electrical signals generated in the heart muscles. This electrical activity, which are often governed by our autonomic nervous system, can be measured directly by electrocardiogram (ECG) using advanced and often obtrusive instrumentation. Photoplethysmogram (PPG), on the other hand, measures how the blood volume changes and can be readily measured with inexpensive instrumentation at certain locations (e.g. at the fingertip). The ECG and PPG are widely used cardiac signals in medical science for diagnosis and health monitoring. But, these signals hold greater potential than just its medical diagnostic applications. In this work, we have mainly investigated if these signals can be used to identify an individual. Every human heart differs by their size, shape, locations inside body, and internal structure. This motivated us to represent the signals using a mathematical model and use machine learning algorithm to identify individual persons. We have discussed how our method improves the identification accuracy and can be used with current biometric methods like fingerprint in our phone. The measurement procedures of cardiac signals are often cumbersome and need instruments which may not be available outside medical facilities. Therefore, we have investigated alternative method of remote photoplethysmography (rPPG) that are relatively inexpensive and unobtrusive. In this dissertation, we have used face video of an individual to extract the heart rate information. The flow of blood causes small changes in the color of face skin. This is not visible to human eyes without digital magnification, but we have shown how knowledge of distinct behavior of human heart rate and use of advanced computer vision algorithms helped us to extract vital signals like heart rate with a significant accuracy. In addition, to measure rPPG using face video, we integrated a method for automatic detection of skin from images and videos. Existing skin detection methods depended on color information which is not always available within available video sources. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. Our method relies on the context and the texture based appearance of skin. To the best of our knowledge, this is first such method that is independent of color cues. In summary, the dissertation investigates the promises and challenges for application of cardiac signals in biometrics and nonobtrusive measurement of cardiac signals using face video.

Abstract Because of its natural and non-intrusive interaction, identity verification and recognition using facial information is among the most active areas in computer vision research. Unfortunately, it has been shown that conventional 2D face recognition techniques are vulnerable to spoofing attacks, where a person tries to masquerade as another one by falsifying biometric data and thereby gaining an illegitimate advantage. This thesis explores different directions for software-based face anti-spoofing. The proposed approaches are divided into two categories: first, low-level feature descriptors are applied for describing the static and dynamic characteristic differences between genuine faces and fake ones in general, and second, complementary attack-specific countermeasures are investigated in order to overcome the limitations of generic spoof detection schemes. The static face representation is based on a set of well-known feature descriptors, including local binary patterns, Gabor wavelet features and histogram of oriented gradients. The key idea is to capture the differences in quality, light reflection and shading by analysing the texture and gradient structure of the input face images. The approach is then extended to the spatiotemporal domain when both facial appearance and dynamics are exploited for spoof detection using local binary patterns from three orthogonal planes. It is reasonable to assume that no generic spoof detection scheme is able to detect all known, let alone unseen, attacks scenarios. In order to find out well-generalizing countermeasures, the problem of anti-spoofing is broken into two attack-specific sub-problems based on whether the spoofing medium can be detected in the provided view or not. The spoofing medium detection is performed by describing the discontinuities in the gradient structures around the detected face. If the display medium is concealed outside the view, a combination of face and background motion correlation measurement and texture analysis is applied. Furthermore, an open-source anti-spoofing fusion framework is introduced and its system-level performance is investigated more closely in order to gain insight on how to combine different anti-spoofing modules. The proposed spoof detection schemes are evaluated on the latest benchmark datasets. The main findings of the experiments are discussed in the thesis. / Tiivistelmä Kasvokuvaan perustuvan henkilöllisyyden tunnistamisen etuja ovat luonnollinen vuorovaikutus ja etätunnistus, minkä takia aihe on ollut erittäin aktiivinen tutkimusalue konenäön tutkimuksessa. Valitettavasti tavanomaiset kasvontunnistustekniikat ovat osoittautuneet haavoittuvaisiksi hyökkäyksille, joissa kameralle esitetään jäljennös kohdehenkilön kasvoista positiivisen tunnistuksen toivossa. Tässä väitöskirjassa tutkitaan erilaisia ohjelmistopohjaisia ratkaisuja keinotekoisten kasvojen ilmaisuun petkuttamisen estämiseksi. Työn ensimmäisessä osassa käytetään erilaisia matalan tason piirteitä kuvaamaan aitojen ja keinotekoisten kasvojen luontaisia staattisia ja dynaamisia eroavaisuuksia. Työn toisessa osassa esitetään toisiaan täydentäviä hyökkäystyyppikohtaisia vastakeinoja, jotta yleispätevien menetelmien puutteet voitaisiin ratkaista ongelmaa rajaamalla. Kasvojen staattisten ominaisuuksien esitys perustuu yleisesti tunnettuihin matalan tason piirteisiin, kuten paikallisiin binäärikuvioihin, Gabor-tekstuureihin ja suunnattujen gradienttien histogrammeihin. Pääajatuksena on kuvata aitojen ja keinotekoisten kasvojen laadun, heijastumisen ja varjostumisen eroavaisuuksia tekstuuria ja gradienttirakenteita analysoimalla. Lähestymistapaa laajennetaan myös tila-aika-avaruuteen, jolloin hyödynnetään samanaikaisesti sekä kasvojen ulkonäköä ja dynamiikkaa irroittamalla paikallisia binäärikuvioita tila-aika-avaruuden kolmelta ortogonaaliselta tasolta. Voidaan olettaa, ettei ole olemassa yksittäistä yleispätevää vastakeinoa, joka kykenee ilmaisemaan jokaisen tunnetun hyökkäystyypin, saati tuntemattoman. Näin ollen työssä keskitytään tarkemmin kahteen hyökkäystilanteeseen. Ensimmäisessä tapauksessa huijausapuvälineen reunoja ilmaistaan analysoimalla gradienttirakenteiden epäjatkuvuuksia havaittujen kasvojen ympäristössä. Jos apuvälineen reunat on piilotettu kameran näkymän ulkopuolelle, petkuttamisen ilmaisu toteutetaan yhdistämällä kasvojen ja taustan liikkeen korrelaation mittausta ja kasvojen tekstuurianalyysiä. Lisäksi työssä esitellään vastakeinojen yhdistämiseen avoimen lähdekoodin ohjelmisto, jonka avulla tutkitaan lähemmin menetelmien fuusion vaikutuksia. Tutkimuksessa esitetyt menetelmät on kokeellisesti vahvistettu alan viimeisimmillä julkisesti saatavilla olevilla tietokannoilla. Tässä väitöskirjassa käydään läpi kokeiden päähavainnot.

<p dir="ltr">Several incidents report misuse of synthetic speech for impersonation attacks, spreading misinformation, and supporting financial frauds. To counter such misuse, this dissertation focuses on developing methods for speech forensics. First, we present a method to detect compressed synthetic speech. The method uses comparatively 33 times less information from compressed bit stream than used by existing methods and achieve high performance. Second, we present a transformer neural network method that uses 2D spectral representation of speech signals to detect synthetic speech. The method shows high performance on detecting both compressed and uncompressed synthetic speech. Third, we present a method using an interpretable machine learning approach known as disentangled representation learning for synthetic speech detection. Fourth, we present a method for synthetic speech attribution. It identifies the source of a speech signal. If the speech is spoken by a human, we classify it as authentic/bona fide. If the speech signal is synthetic, we identify the generation method used to create it. We examine both closed-set and open-set attribution scenarios. In a closed-set scenario, we evaluate our approach only on the speech generation methods present in the training set. In an open-set scenario, we also evaluate on methods which are not present in the training set. Fifth, we propose a multi-domain method for synthetic speech localization. It processes multi-domain features obtained from a transformer using a ResNet-style MLP. We show that with relatively less number of parameters, the proposed method performs better than existing methods. Finally, we present a new direction of research in speech forensics <i>i.e.</i>, bias and fairness of synthetic speech detectors. By bias, we refer to an action in which a detector unfairly targets a specific demographic group of individuals and falsely labels their bona fide speech as synthetic. We show that existing synthetic speech detectors are gender, age and accent biased. They also have bias against bona fide speech from people with speech impairments such as stuttering. We propose a set of augmentations that simulate stuttering in speech. We show that synthetic speech detectors trained with proposed augmentation have less bias relative to detector trained without it.</p>