This study was performed with the goal of delineating how drivers' interactions with an Automated Driving System were affected by a Driver Monitoring System (DMS), which provided alerts to the driver when he or she became inattentive to the driving environment. There were two specific research questions. The first was centered on addressing how drivers' trust and satisfaction with an Automated Driving System was affected by a DMS. The second was centered on addressing how drivers' abilities to detect changes in the driving environment that required intervention were affected by the presence of a DMS. Data were collected from fifty-six drivers during a test-track experiment with an Automated Driving System prototype that was equipped with a DMS. DMS attention prompt conditions were treated as the independent variable and trust, satisfaction, and driver performance during the experimenter triggered lane drifts were treated as dependent variables. The findings of this investigation suggested that drivers who receive attention prompts from a DMS have lower levels of trust and satisfaction with the Automated Driving System compared to drivers who do not receive attention prompts from a DMS. While the DMS may result in lower levels of trust and satisfaction, the DMS may help drivers detect changes in the driving environment that require attention. Specifically, drivers who received attention prompts after 7 consecutive seconds of inattention were 5 times more likely to react to a lane drift with no alert compared to drivers who did not receive attention prompts at all. / Master of Science

Driving a vehicle involves a series of events, which are related to and evolve with the mental state (such as sleepiness, mental load, and stress) of the driv- er. These states are also identified as causal factors of critical situations that can lead to road accidents and vehicle crashes. These driver impairments need to be detected and predicted in order to reduce critical situations and road accidents. In the past years, physiological signals have become conven- tional measures in driver impairment research. Physiological signals have been applied in various studies to identify different levels of mental load, sleepiness, and stress during driving. This licentiate thesis work has investigated several artificial intelligence algorithms for developing an intelligent system to monitor driver mental state using physiological signals. The research aims to measure sleepiness and mental load using Electroencephalography (EEG). EEG signals, if pro- cessed correctly and efficiently, have potential to facilitate advanced moni- toring of sleepiness, mental load, fatigue, stress etc. However, EEG signals can be contaminated with unwanted signals, i.e., artifacts. These artifacts can lead to serious misinterpretation. Therefore, this work investigates EEG arti- fact handling methods and propose an automated approach for EEG artifact handling. Furthermore, this research has also investigated how several other physiological parameters (Heart Rate (HR) and Heart Rate Variability (HRV) from the Electrocardiogram (ECG), Respiration Rate, Finger Tem- perature (FT), and Skin Conductance (SC)) to quantify drivers’ stress. Dif- ferent signal processing methods have been investigated to extract features from these physiological signals. These features have been extracted in the time domain, in the frequency domain as well as in the joint time-frequency domain using wavelet analysis. Furthermore, data level signal fusion has been proposed using Multivariate Multiscale Entropy (MMSE) analysis by combining five physiological sensor signals. Primarily Case-Based Reason- ing (CBR) has been applied for drivers’ mental state classification, but other Artificial intelligence (AI) techniques such as Fuzzy Logic, Support Vector Machine (SVM) and Artificial Neural Network (ANN) have been investigat- ed as well. For drivers’ stress classification, using the CBR and MMSE approach, the system has achieved 83.33% classification accuracy compared to a human expert. Moreover, three classification algorithms i.e., CBR, an ANN, and a SVM were compared to classify drivers’ stress. The results show that CBR has achieved 80% and 86% accuracy to classify stress using finger tempera- ture and heart rate variability respectively, while ANN and SVM reached an accuracy of less than 80%. / Vehicle Driver Monitoring

Since forward collision is one of the most common and dangerous types of traffic accidents, many studies and researches have been conducted to develop forward collision avoidance system. To facilitate the tradeoff between comfort and safety for forward collision avoidance, the driver's state needs to be monitored and estimated. Such support is necessary for Forward Collision Warning (FCW) system given human-involved control. Due to the advances of Driver Monitoring System (DMS), the demand for camera-based driver's state estimation has increased. This master thesis project, conducted at Zenuity AB, investigates a method to estimate driver's awareness based on DMS. The estimation of a driver's awareness is expected to help adapt FCW system based on visual attention when facing the unpredictable braking of the leading vehicle. The project consists of three tasks: gaze estimation, Gaze-to-Object Mapping (GTOM), and awareness estimation. A combined Kalman Filter was developed in gaze estimation for compensation of missing data and outliers and reducing the difference to “ground truth” data. The uncertainty matrix from gaze estimation was utilized to extract a gaze-to-object probability signal in GTOM, while the corresponding fixation duration was also obtained in GTOM. The two extracted new features were used in awareness estimation with two methods: Logistic Regression and two-Hidden Markov Model. The comparison between the two methods reveals whether a complex method is preferred or not. Based on the results of this project, Logistic Regression seems to perform better in driver's state estimation, with 92.0% accuracy and 76.3% True Negative rate. However, further research and improvements on the two-Hidden Markov Model are needed to reach a more comprehensive conclusion. The main contribution of this project is an investigation of an end-to-end method for driver's awareness estimation and thereby an identification of challenges for further studies. / Frontkollision (forward collision) är en av de vanligaste och farligaste typerna av trafikolyckor. Många studier och undersökningar har genomförts för att utveckla system för att undvika kollisioner. För att underlätta avvägningar mellan komfort och säkerhet för att undvika Frontkollision måste förarens tillstånd övervakas och skattas. Ett sådant stöd är nödvändigt för Forward Collision Warning (FCW) systemet, som involverar interaktion med människor. Efterfrågan på kamerabaserad uppskattning för föraren har ökat på grund av framsteg Driver Monitoring System (DMS). Det här examensarbete genomfördes på Zenuity AB och undersökte en metod för att skatta förarens medvetenhet baserad på Driver Monitoring System. Uppskattningen av förarens medvetenhet förväntas bidra till att anpassa FCW-systemet. Detta FCW-system är baserat på visuell uppmärksamhet om när oförutsägbar bromsning av det framförvarande fordonet sker. Examensarbetet består av tre uppgifter: blickuppskattning, Gaze-to-Object Mapping (GTOM), och medventenhetsuppskattning. Ett kombinerat Kalman-filter har utvecklats i gaze uppskattning för att kompensera saknade data och outliers samt reducera skillnaden till “ground truth” data. Osäkerhetesmatrisen från gaze uppskattningen användes för att extrahera en gaze-to-object sannolikhetssignal i GTOM. Den motsvarande fixeringsvaraktigheten erhålls också i GTOM. De två extraherade nya egenskaperna användes i medvetenhetsanalys med hjälp av två metoder: logistic regression och two-Hidden Markov Model. Jämförelsen mellan de två metoderna avslöjar om en komplex metod är att föredra eller inte. Resultatet av detta examensarbet visar att logistic regression fungerar bättre i förarens statusuppskattning med 92% noggrannhet och 76.3% True Negative rate. Vidare forskning och förbättringar av den two-hidden Markov modell behövs för att dra en mer omfattande slutsats. Det huvudsakliga bidraget av examensarbetet är en utforskning av en end-to-end metod för att uppskatta förarens medvetenhet och därmed kunna identifiera utmaningar för framtid studie.

Thesis (M.S.)--Linköping University, 2004. / Includes bibliographical references (p. 63-64). Also available online via the VTI web site (www.vti.se).

The dissertation investigates the promises and challenges for application of cardiac signals in biometrics and affective computing, and noninvasive measurement of cardiac signals. We have mainly discussed two major cardiac signals: electrocardiogram (ECG), and photoplethysmogram (PPG). ECG and PPG signals hold strong potential for biometric authentications and identifications. We have shown that by mapping each cardiac beat from time domain to an angular domain using a limit cycle, intra-class variability can be significantly minimized. This is in contrary to conventional time domain analysis. Our experiments with both ECG and PPG signal shows that the proposed method eliminates the effect of instantaneous heart rate on the shape morphology and improves authentication accuracy. For noninvasive measurement of PPG beats, we have developed a systematic algorithm to extract pulse rate from face video in diverse situations using video magnification. We have extracted signals from skin patches and then used frequency domain correlation to filter out non-cardiac signals. We have developed a novel entropy based method to automatically select skin patches from face. We report beat-to-beat accuracy of remote PPG (rPPG) in comparison to conventional average heart rate. The beat-to-beat accuracy is required for applications related to heart rate variability (HRV) and affective computing. The algorithm has been tested on two datasets, one with static illumination condition and the other with unrestricted ambient illumination condition. Automatic skin detection is an intermediate step for rPPG. Existing methods always depend on color information to detect human skin. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. We have used LBP lacunarity based micro-textures features and a region growing algorithm to find skin pixels in an image. Our experiment shows that the proposed method is applicable universally to any image including near infra-red images. This finding helps to extend the domain of many application including rPPG. To the best of our knowledge, this is first such method that is independent of color cues. / Ph. D. / The heart is an integral part of the human body. With every beat, the heart continuously pumps oxygen-enriched blood to providing fuel to our cells and thus enabling life. The heartbeat is initiated by electrical signals generated in the heart muscles. This electrical activity, which are often governed by our autonomic nervous system, can be measured directly by electrocardiogram (ECG) using advanced and often obtrusive instrumentation. Photoplethysmogram (PPG), on the other hand, measures how the blood volume changes and can be readily measured with inexpensive instrumentation at certain locations (e.g. at the fingertip). The ECG and PPG are widely used cardiac signals in medical science for diagnosis and health monitoring. But, these signals hold greater potential than just its medical diagnostic applications. In this work, we have mainly investigated if these signals can be used to identify an individual. Every human heart differs by their size, shape, locations inside body, and internal structure. This motivated us to represent the signals using a mathematical model and use machine learning algorithm to identify individual persons. We have discussed how our method improves the identification accuracy and can be used with current biometric methods like fingerprint in our phone. The measurement procedures of cardiac signals are often cumbersome and need instruments which may not be available outside medical facilities. Therefore, we have investigated alternative method of remote photoplethysmography (rPPG) that are relatively inexpensive and unobtrusive. In this dissertation, we have used face video of an individual to extract the heart rate information. The flow of blood causes small changes in the color of face skin. This is not visible to human eyes without digital magnification, but we have shown how knowledge of distinct behavior of human heart rate and use of advanced computer vision algorithms helped us to extract vital signals like heart rate with a significant accuracy. In addition, to measure rPPG using face video, we integrated a method for automatic detection of skin from images and videos. Existing skin detection methods depended on color information which is not always available within available video sources. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. Our method relies on the context and the texture based appearance of skin. To the best of our knowledge, this is first such method that is independent of color cues. In summary, the dissertation investigates the promises and challenges for application of cardiac signals in biometrics and nonobtrusive measurement of cardiac signals using face video.

En del förare ägnar sig åt sekundära aktiviteter som bidrar till en minskad trafiksäkerhet när de kör. User Experience kan informera och hjälpa människor till att göra säkrare och mer medvetna val i trafiken. Forskning från självrapporterad mobiltelefonanvändning visar att ett förbud mot mobilanvändning under bilkörning inte har lett till att människor helt upphört från beteendet. Arbetet undersökte om ett förarövervakningssystem bidrar till säkrare bilkörning när bilförare är distraherade av mobiltelefoner och om systemet påverkade deras känsla av säkerhet. En ytterligare frågeställning undersökte om det fanns någon skillnad mellan kvinnor och mäns förarbeteende och uppfattning av förarövervakningssystem. Detta undersöktes meden experimentellt mixad metoddesign med tester i simulator, enkäter och intervjuer. Simulatorstudien hade en mellangruppsdesign där 16 deltagare delades upp i tre grupper och körde tre banor i tätort och stadstrafik: deltagarna delades in i en kontrollgrupp (grupp A) utan mobil eller förarövervakningssystem, grupp B fick distraherande SMS och grupp C fick distraherande SMS och varnades av förarövervakningssystemet AIS12. Deltagarna intervjuades och fick fylla i en utvärderande enkät efter simulatorn. En enkät skickades ut online för att ta reda på förarbeteende och åsikter om förarövervakningssystem. Trots att resultaten från simulatorn ej var signifikanta så framkom åsikter om för- och nackdelar med förarövervakningssystem från intervjudeltagarna. Simulatorexperimentet hade ett litet stickprov, vilket kan bidragit till att resultatet inte visade sig vara signifikant. / Some drivers engage in secondary activities that contribute to decreased traffic safety. User Experience can inform and help people make safer and more conscious choices in traffic. Previous research on self-reported mobile phone use shows that a ban on mobile phone use while driving has not led people to completely cease the behavior. The study investigated whether a driver monitoring system contributes to safer driving when drivers are distracted by mobile phones and whether the system affected their sense of safety. An additional question examined whether there was a difference between male and female drivers' behavior and perception of the driver monitoring system. This was investigated using an experimental mixed-method design with a simulator, surveys, and an interview. The simulator study had a between-groups design where 16 participants were divided into three groups and drove three tracks in urban and city traffic: group A was a control group without mobile or driver monitoring system, group B received distracting SMS messages, and group C received distracting SMS messages and were warned by the driver monitoring system AIS12. Participants were interviewed and filled out an evaluative survey after the simulator. Another survey was sent out online to investigate driver behavior and opinions about driver monitoring systems. Although the results from the simulator were not significant, opinions about the pros and cons of the driver monitoring system emerged from the interview participants. The simulator experiment had a small sample size, which may have contributed to the result not being significant.