Driving a vehicle involves a series of events, which are related to and evolve with the mental state (such as sleepiness, mental load, and stress) of the driv- er. These states are also identified as causal factors of critical situations that can lead to road accidents and vehicle crashes. These driver impairments need to be detected and predicted in order to reduce critical situations and road accidents. In the past years, physiological signals have become conven- tional measures in driver impairment research. Physiological signals have been applied in various studies to identify different levels of mental load, sleepiness, and stress during driving. This licentiate thesis work has investigated several artificial intelligence algorithms for developing an intelligent system to monitor driver mental state using physiological signals. The research aims to measure sleepiness and mental load using Electroencephalography (EEG). EEG signals, if pro- cessed correctly and efficiently, have potential to facilitate advanced moni- toring of sleepiness, mental load, fatigue, stress etc. However, EEG signals can be contaminated with unwanted signals, i.e., artifacts. These artifacts can lead to serious misinterpretation. Therefore, this work investigates EEG arti- fact handling methods and propose an automated approach for EEG artifact handling. Furthermore, this research has also investigated how several other physiological parameters (Heart Rate (HR) and Heart Rate Variability (HRV) from the Electrocardiogram (ECG), Respiration Rate, Finger Tem- perature (FT), and Skin Conductance (SC)) to quantify drivers’ stress. Dif- ferent signal processing methods have been investigated to extract features from these physiological signals. These features have been extracted in the time domain, in the frequency domain as well as in the joint time-frequency domain using wavelet analysis. Furthermore, data level signal fusion has been proposed using Multivariate Multiscale Entropy (MMSE) analysis by combining five physiological sensor signals. Primarily Case-Based Reason- ing (CBR) has been applied for drivers’ mental state classification, but other Artificial intelligence (AI) techniques such as Fuzzy Logic, Support Vector Machine (SVM) and Artificial Neural Network (ANN) have been investigat- ed as well. For drivers’ stress classification, using the CBR and MMSE approach, the system has achieved 83.33% classification accuracy compared to a human expert. Moreover, three classification algorithms i.e., CBR, an ANN, and a SVM were compared to classify drivers’ stress. The results show that CBR has achieved 80% and 86% accuracy to classify stress using finger tempera- ture and heart rate variability respectively, while ANN and SVM reached an accuracy of less than 80%. / Vehicle Driver Monitoring

Physiological sensor signals analysis is common practice in medical domain for diagnosis andclassification of various physiological conditions. Clinicians’ frequently use physiologicalsensor signals to diagnose individual’s psychophysiological parameters i.e., stress tiredness,and fatigue etc. However, parameters obtained from physiological sensors could vary becauseof individual’s age, gender, physical conditions etc. and analyzing data from a single sensorcould mislead the diagnosis result. Today, one proposition is that sensor signal fusion canprovide more reliable and efficient outcome than using data from single sensor and it is alsobecoming significant in numerous diagnosis fields including medical diagnosis andclassification. Case-Based Reasoning (CBR) is another well established and recognizedmethod in health sciences. Here, an entropy based algorithm, “Multivariate MultiscaleEntropy analysis” has been selected to fuse multiple sensor signals. Other physiologicalsensor signals measurements are also taken into consideration for system evaluation. A CBRsystem is proposed to classify ‘healthy’ and ‘stressed’ persons using both fused features andother physiological i.e. Heart Rate Variability (HRV), Respiratory Sinus Arrhythmia (RSA),Finger Temperature (FT) features. The evaluation and performance analysis of the system have been done and the results ofthe classification based on data fusion and physiological measurements are presented in thisthesis work.

Cette thèse étudie la reconnaissance des émotions à partir de signaux physiologiques dans le contexte des jeux vidéo et la faisabilité de sa mise en œuvre sur un système embarqué. Les défis suivants sont abordés : la relation entre les états émotionnels et les réponses physiologiques dans le contexte du jeu, les variabilités individuelles des réponses psycho-physiologiques et les problèmes de mise en œuvre sur un système embarqué. Les contributions majeures de cette thèse sont les suivantes. Premièrement, nous construisons une base de données multimodale dans le cadre de l'Affective Gaming (DAG). Cette base de données contient plusieurs mesures concernant les modalités objectives telles que les signaux physiologiques de joueurs et des évaluations subjectives sur des phases de jeu. A l'aide de cette base, nous présentons une série d'analyses effectuées pour la détection des moments marquant émotionnellement et la classification des émotions à l'aide de diverses méthodes d'apprentissage automatique. Deuxièmement, nous étudions la variabilité individuelle de la réponse émotionnelle et proposons un modèle basé sur un groupe de joueurs déterminé par un clustering selon un ensemble de traits physiologiques pertinents. Nos travaux mettent en avant le fait que le modèle proposé, basé sur un tel groupe personnalisé, est plus performant qu'un modèle général ou qu'un modèle spécifique à un utilisateur. Troisièmement, nous appliquons la méthode proposée sur un système ARM A9 et montrons que la méthode proposée peut répondre à l'exigence de temps de calcul. / This thesis investigates physiological-based emotion recognition in a digital game context and the feasibility of implementing the model on an embedded system. The following chanllenges are addressed: the relationship between emotional states and physiological responses in the game context, individual variabilities of the pschophysiological responses and issues of implementation on an embedded system. The major contributions of this thesis are : Firstly, we construct a multi-modal Database for Affective Gaming (DAG). This database contains multiple measurements concerning objective modalities: physiological signals (ECG, EDA, EMG, Respiration), screen recording, and player's face recording, as well as subjective assessments on both game event and match level. We presented statistics of the database and run a series of analysis on issues such as emotional moment detection and emotion classification, influencing factors of the overall game experience using various machine learning methods. Secondly, we investigate the individual variability in the collected data by creating an user-specific model and analyzing the optimal feature set for each individual. We proposed a personalized group-based model created the similar user groups by using the clustering techniques based on physiological traits deduced from optimal feature set. We showed that the proposed personalized group-based model performs better than the general model and user-specific model. Thirdly, we implemente the proposed method on an ARM A9 system and showed that the proposed method can meet the requirement of computation time.

在心理與認知科學領域中常使用生理訊號來測量受試者的反應，並反映出人們的心理狀態起伏。本研究探討應用生理訊號識別情緒之可能性，以及將生理訊號與其他情緒辨識結果整合之方法。 在過去的研究中，生理與心理的對應關係，並無太多著墨，可稱為一黑盒子(black-box)的方式。並因上述類別式實驗長時間收集的生理訊號，對於誘發特定情緒反應之因果(cause-effect)並未進行深入的討論。本研究由於實驗的設計與選用材料之故，可一探純粹由刺激引發的情緒下情緒在生理與心理之因果關係，在輸入輸出對應間能有較明確的解釋。 本研究中嘗試監測較短時間(<10sec)的生理資訊，期望以一近乎即時的方式判讀並回饋使用者適當的資訊，對於生理訊號與情緒狀態的關聯性研究，將以IAPS(International Affective Picture System) 素材為來源，進行較過去嚴謹的實驗設計與程序，以探究生理訊號特徵如何應用於情緒分類。 雖然本研究以維度式情緒學說為理論基礎，然考慮到實際應用情境，若有其他以類別式的理論為基礎之系統，如何整合維度式與類別式兩類的資訊，提出可行的轉換方式，亦是本研究的主要課題。 / Physiological signals can be used to measure a subject’s response to a particular stimulus, and infer the emotional status accordingly. This research investigates the feasibility of emotion recognition using physiological measurements in a smart living space. It also addresses important issues regarding the integration of classification results from multiple modalities. Most past research regarded the recognition of emotion from physiological data as a mapping mechanism which can be learned from training data. These data were collected over a long period of time, and can not model the immediate cause-effect relationship effectively. Our research employs a more rigorous experiment design to study the relationship between a specific physiological signal and the emotion status. The newly designed procedure will enable us to identify and validate the discriminating power of each type of physiological signal in recognizing emotion. Our research monitors short term (< 10s) physiological signals. We use the IAPS (International Affective Picture System) as our experiment material. Physiological data were collected during the presentation of various genres of pictures. With such controlled experiments, we expect the cause-effect relation to be better explained than previous black-box approaches. Our research employs dimensional approach for emotion modeling. However, emotion recognition based on audio and/or visual clues mostly adopt categorical method (or basic emotion types). It becomes necessary to integrate results from these different modalities. Toward this end, we have also developed a mapping process to convert the result encoded in dimensional format into categorical data.

Dans la société moderne, le stress s’avère un problème omniprésent. Un stress permanent peut entraîner divers problèmes mentaux et physiques notamment pour des personnes confrontées à des situations d'urgence comme par exemple des pompiers en intervention: il peut modifier leurs actions et les mettre en danger. Par conséquent, dans ce contexte, il est pertinent de chercher à évaluer le stress de la personne. Sur la base de cette idée, a été proposé le projet Psypocket qui vise à concevoir un système portable capable d'analyser précisément l'état de stress d'une personne en fonction de ses modifications physiologiques, psychologiques et comportementales, puis de proposer des solutions de rétroaction pour réguler cet état. Cette thèse s’inscrit dans le cadre du projet Psypocket. Nous y discutons de la faisabilité et de l'intérêt de la reconnaissance du stress à partir de données hétérogènes. Non seulement les signaux physiologiques, tels que l'électrocardiographie (ECG), l'élecchtromyographie (EMG) et l'activité électrodermale (EDA), mais aussi le temps de réaction (RT) sont adoptés pour discriminer différents états de stress d'une personne. Pour cela nous proposons une approche basée sur un classifieur SVM (Machine à Vecteurs de Support). Les résultats obtenus montrent que le temps de réaction peut-être un moyen d’estimation du niveau de stress de l’individu en complément ou non des signaux physiologiques. En outre, nous discutons de la faisabilité d'un système embarqué à même de réaliser la chaîne globale de traitement des signaux. Cette thèse contribue donc à la conception d’un système portable de reconnaissance du stress d'une personne en temps réel en adoptant des données hétérogènes, en l’occurrence les signaux physiologiques et le temps de réaction / In modern society, the stress of an individual has been found to be a common problem. Continuous stress can lead to various mental and physical problems and especially for the people who always face emergency situations (e.g., fireman): it may alter their actions and put them in danger. Therefore, it is meaningful to provide the assessment of the stress of an individual. Based on this idea, the Psypocket project is proposed which is aimed at making a portable system able to analyze accurately the stress state of an individual based on his physiological, psychological and behavioural modifications. It should then offer solutions for feedback to regulate this state.The research of this thesis is an essential part of the Psypocket project. In this thesis, we discuss the feasibility and the interest of stress recognition from heterogeneous data. Not only physiological signals, such as Electrocardiography (ECG), Electromyography (EMG) and Electrodermal activity (EDA), but also reaction time (RT) are adopted to recognize different stress states of an individual. For the stress recognition, we propose an approach based on a SVM classifier (Support Vector Machine). The results obtained show that the reaction time can be used to estimate the level of stress of an individual in addition or not to the physiological signals. Besides, we discuss the feasibility of an embedded system which would realize the complete data processing. Therefore, the study of this thesis can contribute to make a portable system to recognize the stress of an individual in real time by adopting heterogeneous data like physiological signals and RT

No description available.

Mechanomyography (MMG) has been proposed as a control modality for alternative access technologies for individuals with disabilities. However, MMG recordings are highly susceptible to contamination from limb movements. Pressure-based transducers are touted to be the most robust to external movement although there is some debate about their optimal chamber geometry, in terms of low frequency gain and spectral flatness. To investigate the question of preferred geometry, transducers with varying chamber shapes were designed, manufactured and tested. The best performance was achieved with a transducer consisting of a low-frequency MEMS microphone, a 4 micron thick aluminized mylar membrane and a rigid conical chamber 7 mm in diameter and 5 mm in height. Furthermore, microphone-derived MMG spectra were found to be less influenced by motion artifact than corresponding accelerometer-derived spectra. However, artifact harmonics were present in both spectra, suggesting that bandpass filtering may not remove artifact influences permeating into MMG frequency bands.

Mechanomyography (MMG) has been proposed as a control modality for alternative access technologies for individuals with disabilities. However, MMG recordings are highly susceptible to contamination from limb movements. Pressure-based transducers are touted to be the most robust to external movement although there is some debate about their optimal chamber geometry, in terms of low frequency gain and spectral flatness. To investigate the question of preferred geometry, transducers with varying chamber shapes were designed, manufactured and tested. The best performance was achieved with a transducer consisting of a low-frequency MEMS microphone, a 4 micron thick aluminized mylar membrane and a rigid conical chamber 7 mm in diameter and 5 mm in height. Furthermore, microphone-derived MMG spectra were found to be less influenced by motion artifact than corresponding accelerometer-derived spectra. However, artifact harmonics were present in both spectra, suggesting that bandpass filtering may not remove artifact influences permeating into MMG frequency bands.