Cette thèse s'intéresse aux biométries dites douces, et notamment à leurs utilisations en termes de sécurité, dans le cadre de différents scénarii commerciaux, incluant des aspects usage. L'accent sera ainsi porté sur les caractéristiques faciales qui constituent un jeu de traits significatifs de l'apparence physique mais aussi comportementale de l'utilisateur permettant de différencier, classer et identifier les individus. Ces traits, qui sont l'âge, le sexe, les cheveux, la peau et la couleur des yeux, mais aussi la présence de lunettes, de moustache ou de barbe, comportent plusieurs avantages notamment la facilité avec laquelle ils peuvent être acquis, mais également du fait qu'ils correspondent à la façon dont les êtres humains perçoivent leurs environnements. Plus précisément, les traits issus de la biométrie douce sont compatibles avec la manière dont l'humain tend à catégoriser son entourage, une démarche impliquant une structuration hiérarchique des différents traits. Cette thèse explore ces différents traits et leurs applications dans les systèmes de biométries douces (SBS), et met l'accent sur la manière dont de tels systèmes peuvent atteindre des buts différents, y compris la recherche accélérée dans des bases de données, l'identification et la ré-identification d'individus, mais également la prédiction et la quantification de l'esthétique d'un visage. Ce travail est motivé notamment par l'importance croissante de ces applications dans notre société en constante évolution, mais aussi par le côté peu contraignant du système. En effet, les SBS sont généralement non-intrusifs, et nécessitent le plus souvent de faibles temps de calculs, permettant ainsi une analyse biométrique rapide, sans imposer obligatoirement l'accord et la coopération de l'individu. Ces atouts rendent la biométrie douce indispensable dans les applications qui ont besoin de traitement d'images ou de vidéos en temps réel.

The relevance of faces in our daily lives is indisputable. We learn to recognize faces as newborns, and faces play a major role in interpersonal communication. The spectrum of computer vision research about face analysis includes, but is not limited to, face detection and facial attribute classification, which are the focus of this dissertation. The face is a primary biometric because by itself revels the subject's identity, while facial attributes (such as hair color and eye state) are soft biometrics because by themselves they do not reveal the subject's identity. In this dissertation, we proposed a real-time model for classifying 40 facial attributes, which preprocesses faces and then extracts 7 types of classical and deep features. These features were fused together to train 3 different classifiers. Our proposed model yielded 91.93% on the average accuracy outperforming 7 state-of-the-art models. We also developed a real-time model for classifying the states of human eyes and mouth (open/closed), and the presence/absence of eyeglasses in the wild. Our method begins by preprocessing a face by cropping the regions of interest (ROIs), and then describing them using RootSIFT features. These features were used to train a nonlinear support vector machine for each attribute. Our eye-state classifier achieved the top performance, while our mouth-state and glasses classifiers were tied as the top performers with deep learning classifiers. We also introduced a new facial attribute related to Middle Eastern headwear (called igal) along with its detector. Our proposed idea was to detect the igal using a linear multiscale SVM classifier with a HOG descriptor. Thereafter, false positives were discarded using dense SIFT filtering, bag-of-visual-words decomposition, and nonlinear SVM classification. Due to the similarity in real-life applications, we compared the igal detector with state-of-the-art face detectors, where the igal detector significantly outperformed the face detectors with the lowest false positives. We also fused the igal detector with a face detector to improve the detection performance. Face detection is the first process in any facial attribute classification pipeline. As a result, we reported a novel study that evaluates the robustness of current face detectors based on: (1) diffraction blur, (2) image scale, and (3) the IoU classification threshold. This study would enable users to pick the robust face detector for their intended applications. / Doctor of Philosophy / The relevance of faces in our daily lives is indisputable. We learn to recognize faces as newborns, and faces play a major role in interpersonal communication. Faces probably represent the most accurate biometric trait in our daily interactions. Thereby, it is not singular that so much effort from computer vision researchers have been invested in the analysis of faces. The automatic detection and analysis of faces within images has therefore received much attention in recent years. The spectrum of computer vision research about face analysis includes, but is not limited to, face detection and facial attribute classification, which are the focus of this dissertation. The face is a primary biometric because by itself revels the subject's identity, while facial attributes (such as hair color and eye state) are soft biometrics because by themselves they do not reveal the subject's identity. Soft biometrics have many uses in the field of biometrics such as (1) they can be utilized in a fusion framework to strengthen the performance of a primary biometric system. For example, fusing a face with voice accent information can boost the performance of the face recognition. (2) They also can be used to create qualitative descriptions about a person, such as being an "old bald male wearing a necktie and eyeglasses." Face detection and facial attribute classification are not easy problems because of many factors, such as image orientation, pose variation, clutter, facial expressions, occlusion, and illumination, among others. In this dissertation, we introduced novel techniques to classify more than 40 facial attributes in real-time. Our techniques followed the general facial attribute classification pipeline, which begins by detecting a face and ends by classifying facial attributes. We also introduced a new facial attribute related to Middle Eastern headwear along with its detector. The new facial attribute were fused with a face detector to improve the detection performance. In addition, we proposed a new method to evaluate the robustness of face detection, which is the first process in the facial attribute classification pipeline. Detecting the states of human facial attributes in real time is highly desired by many applications. For example, the real-time detection of a driver's eye state (open/closed) can prevent severe accidents. These systems are usually called driver drowsiness detection systems. For classifying 40 facial attributes, we proposed a real-time model that preprocesses faces by localizing facial landmarks to normalize faces, and then crop them based on the intended attribute. The face was cropped only if the intended attribute is inside the face region. After that, 7 types of classical and deep features were extracted from the preprocessed faces. Lastly, these 7 types of feature sets were fused together to train three different classifiers. Our proposed model yielded 91.93% on the average accuracy outperforming 7 state-of-the-art models. It also achieved state-of-the-art performance in classifying 14 out of 40 attributes. We also developed a real-time model that classifies the states of three human facial attributes: (1) eyes (open/closed), (2) mouth (open/closed), and (3) eyeglasses (present/absent). Our proposed method consisted of six main steps: (1) In the beginning, we detected the human face. (2) Then we extracted the facial landmarks. (3) Thereafter, we normalized the face, based on the eye location, to the full frontal view. (4) We then extracted the regions of interest (i.e., the regions of the mouth, left eye, right eye, and eyeglasses). (5) We extracted low-level features from each region and then described them. (6) Finally, we learned a binary classifier for each attribute to classify it using the extracted features. Our developed model achieved 30 FPS with a CPU-only implementation, and our eye-state classifier achieved the top performance, while our mouth-state and glasses classifiers were tied as the top performers with deep learning classifiers. We also introduced a new facial attribute related to Middle Eastern headwear along with its detector. After that, we fused it with a face detector to improve the detection performance. The traditional Middle Eastern headwear that men usually wear consists of two parts: (1) the shemagh or keffiyeh, which is a scarf that covers the head and usually has checkered and pure white patterns, and (2) the igal, which is a band or cord worn on top of the shemagh to hold it in place. The shemagh causes many unwanted effects on the face; for example, it usually occludes some parts of the face and adds dark shadows, especially near the eyes. These effects substantially degrade the performance of face detection. To improve the detection of people who wear the traditional Middle Eastern headwear, we developed a model that can be used as a head detector or combined with current face detectors to improve their performance. Our igal detector consists of two main steps: (1) learning a binary classifier to detect the igal and (2) refining the classier by removing false positives. Due to the similarity in real-life applications, we compared the igal detector with state-of-the-art face detectors, where the igal detector significantly outperformed the face detectors with the lowest false positives. We also fused the igal detector with a face detector to improve the detection performance. Face detection is the first process in any facial attribute classification pipeline. As a result, we reported a novel study that evaluates the robustness of current face detectors based on: (1) diffraction blur, (2) image scale, and (3) the IoU classification threshold. This study would enable users to pick the robust face detector for their intended applications. Biometric systems that use face detection suffer from huge performance fluctuation. For example, users of biometric surveillance systems that utilize face detection sometimes notice that state-of-the-art face detectors do not show good performance compared with outdated detectors. Although state-of-the-art face detectors are designed to work in the wild (i.e., no need to retrain, revalidate, and retest), they still heavily depend on the datasets they originally trained on. This condition in turn leads to variation in the detectors' performance when they are applied on a different dataset or environment. To overcome this problem, we developed a novel optics-based blur simulator that automatically introduces the diffraction blur at different image scales/magnifications. Then we evaluated different face detectors on the output images using different IoU thresholds. Users, in the beginning, choose their own values for these three settings and then run our model to produce the efficient face detector under the selected settings. That means our proposed model would enable users of biometric systems to pick the efficient face detector based on their system setup. Our results showed that sometimes outdated face detectors outperform state-of-the-art ones under certain settings and vice versa.

Puisque les traits de biométrie douce peuvent fournir des preuves supplémentaires pour aider à déterminer précisément l’identité de l’homme, il y a eu une attention croissante sur la reconnaissance faciale basée sur les biométrie douce ces dernières années. Parmi tous les biométries douces, le sexe et l’ethnicité sont les deux caractéristiques démographiques importantes pour les êtres humains et ils jouent un rôle très fondamental dans l’analyse de visage automatique. En attendant, la reconnaissance des expressions faciales est un autre challenge dans le domaine de l’analyse de visage en raison de la diversité et de l’hybridité des expressions humaines dans différentes cultures, genres et contextes. Ce thèse est dédié à combiner la texture du visage 2D et la morphologie du visage 3D pour estimer les biométries douces: le sexe, l’ethnicité, etc., et reconnaître les expressions faciales. Pour la reconnaissance du sexe et de l’ethnicité, nous présentons une approche efficace en combinant à la fois des textures locales et des caractéristiques de forme extraites à partir des modèles de visage 3D, contrairement aux méthodes existantes qui ne dépendent que des textures ou des caractéristiques de forme. Afin de souligne exhaustivement la différence entre les groupes sexuels et ethniques, nous proposons un nouveau descripteur, à savoir local circular patterns (LCP). Ce descripteur améliore Les motifs binaires locaux (LBP) et ses variantes en remplaçant la quantification binaire par une quantification basée sur le regroupement, entraînant d’une puissance plus discriminative et une meilleure résistance au bruit. En même temps, l’algorithme Adaboost est engagé à sélectionner les caractéristiques discriminatives fortement liés au sexe et à l’ethnicité. Les résultats expérimentaux obtenus sur les bases de données FRGC v2.0 et BU-3DFE démontrent clairement les avantages de la méthode proposée. Pour la reconnaissance des expressions faciales, nous présentons une méthode automatique basée sur les multi-modalité 2D + 3D et démontrons sa performance sur la base des données BU-3DFE. Notre méthode combine des textures locales et des descripteurs de formes pour atteindre l’efficacité et la robustesse. Tout d’abord, un grand ensemble des points des caractéristiques d’images 2D et de modèles 3D sont localisés à l’aide d’un nouvel algorithme, à savoir la cascade parallèle incrémentielle de régression linéaire (iPar-CLR). Ensuite, on utilise un nouveau descripteur basé sur les histogrammes des gradients d’ordre secondaire (HSOG) en conjonction avec le descripteur SIFT pour décrire la texture locale autour de chaque point de caractéristique 2D. De même, la géométrie locale autour de chaque point de caractéristique 3D est décrite par deux nouveaux descripteurs de forme construits à l’aide des quantités différentielle de géométries de la surface au premier ordre et au second ordre, à savoir meshHOG et meshHOS. Enfin, les résultats de reconnaissance des descripteurs 2D et 3D fournis par le classifier SVM sont fusionnés à la fois au niveau de fonctionnalité et de score pour améliorer la précision. Les expérimentaux résultats démontrent clairement qu’il existe des caractéristiques complémentaires entre les descripteurs 2D et 3D. Notre approche basée sur les multi-modalités surpasse les autres méthodes de l’état de l’art en obtenant une précision de reconnaissance 86,32%. De plus, une bonne capacité de généralisation est aussi présentée sur la base de données Bosphorus. / Since soft biometrics traits can provide sufficient evidence to precisely determine the identity of human, there has been increasing attention for face based soft biometrics identification in recent years. Among those face based soft biometrics, gender and ethnicity are both key demographic attributes of human beings and they play a very fundamental and important role in automatic machine based face analysis. Meanwhile, facial expression recognition is another challenge problem in face analysis because of the diversity and hybridity of human expressions among different subjects in different cultures, genders and contexts. This Ph.D thesis work is dedicated to combine 2D facial Texture and 3D face morphology for estimating people’s soft biometrics: gender, ethnicity, etc., and recognizing facial expression. For the gender and ethnicity recognition, we present an effective and efficient approach on this issue by combining both boosted local texture and shape features extracted from 3D face models, in contrast to the existing ones that only depend on either 2D texture or 3D shape of faces. In order to comprehensively represent the difference between different genders or ethnics groups, we propose a novel local descriptor, namely local circular patterns (LCP). LCP improves the widely utilized local binary patterns (LBP) and its variants by replacing the binary quantization with a clustering based one, resulting in higher discriminative power as well as better robustness to noise. Meanwhile, the following Adaboost based feature selection finds the most discriminative gender- and ethnic-related features and assigns them with different weights to highlight their importance in classification, which not only further raises the performance but reduces the time and memory cost as well. Experimental results achieved on the FRGC v2.0 and BU-3DFE data sets clearly demonstrate the advantages of the proposed method. For facial expression recognition, we present a fully automatic multi-modal 2D + 3D feature-based facial expression recognition approach and demonstrate its performance on the BU–3DFE database. Our approach combines multi-order gradientbased local texture and shape descriptors in order to achieve efficiency a nd robustness. First, a large set of fiducial facial landmarks of 2D face images along with their 3D face scans are localized using a novel algorithm namely incremental Parallel Cascade of Linear Regression (iPar–CLR). Then, a novel Histogram of Second Order Gradients (HSOG) based local image descriptor in conjunction with the widely used first-order gradient based SIFT descriptor are employed to describe the local texture around each 2D landmark. Similarly, the local geometry around each 3D landmark is described by two novel local shape descriptors constructed using the first-order and the second-order surface differential geometry quantities, i.e., Histogram of mesh Gradients (meshHOG) and Histogram of mesh Shape index (curvature quantization, meshHOS). Finally, the Support Vector Machine (SVM) based recognition results of all 2D and 3D descriptors are fused at both featurelevel and score-level to further improve the accuracy. Comprehensive experimental results demonstrate that there exist impressive complementary characteristics between the 2D and 3D descriptors. We use the BU–3DFE benchmark to compare our approach to the state-of-the-art ones. Our multi-modal feature-based approach outperforms the others by achieving an average recognition accuracy of 86,32%. Moreover, a good generalization ability is shown on the Bosphorus database.

Les progrès récents des réseaux de neurones artificiels (plus connus sous le nom d'apprentissage profond) ont permis d'améliorer l’état de l’art dans plusieurs domaines de la vision par ordinateur. Dans cette thèse, nous étudions des techniques d'apprentissage profond dans le cadre de l’analyse du genre et de l’âge à partir du visage humain. En particulier, deux problèmes complémentaires sont considérés : (1) la prédiction du genre et de l’âge, et (2) la synthèse et l’édition du genre et de l’âge.D’abord, nous effectuons une étude détaillée qui permet d’établir une liste de principes pour la conception et l’apprentissage des réseaux de neurones convolutifs (CNNs) pour la classification du genre et l’estimation de l’âge. Ainsi, nous obtenons les CNNs les plus performants de l’état de l’art. De plus, ces modèles nous ont permis de remporter une compétition internationale sur l’estimation de l’âge apparent. Nos meilleurs CNNs obtiennent une précision moyenne de 98.7% pour la classification du genre et une erreur moyenne de 4.26 ans pour l’estimation de l’âge sur un corpus interne particulièrement difficile.Ensuite, afin d’adresser le problème de la synthèse et de l’édition d’images de visages, nous concevons un modèle nommé GA-cGAN : le premier réseau de neurones génératif adversaire (GAN) qui produit des visages synthétiques réalistes avec le genre et l’âge souhaités. Enfin, nous proposons une nouvelle méthode permettant d’employer GA-cGAN pour le changement du genre et de l’âge tout en préservant l’identité dans les images synthétiques. Cette méthode permet d'améliorer la précision d’un logiciel sur étagère de vérification faciale en présence d’écarts d’âges importants. / The recent progress in artificial neural networks (rebranded as deep learning) has significantly boosted the state-of-the-art in numerous domains of computer vision. In this PhD study, we explore how deep learning techniques can help in the analysis of gender and age from a human face. In particular, two complementary problem settings are considered: (1) gender/age prediction from given face images, and (2) synthesis and editing of human faces with the required gender/age attributes.Firstly, we conduct a comprehensive study which results in an empirical formulation of a set of principles for optimal design and training of gender recognition and age estimation Convolutional Neural Networks (CNNs). As a result, we obtain the state-of-the-art CNNs for gender/age prediction according to the three most popular benchmarks, and win an international competition on apparent age estimation. On a very challenging internal dataset, our best models reach 98.7% of gender classification accuracy and an average age estimation error of 4.26 years.In order to address the problem of synthesis and editing of human faces, we design and train GA-cGAN, the first Generative Adversarial Network (GAN) which can generate synthetic faces of high visual fidelity within required gender and age categories. Moreover, we propose a novel method which allows employing GA-cGAN for gender swapping and aging/rejuvenation without losing the original identity in synthetic faces. Finally, in order to show the practical interest of the designed face editing method, we apply it to improve the accuracy of an off-the-shelf face verification software in a cross-age evaluation scenario.