Driver sleepiness is a cause for crashes and it is estimated that 3.9 to 33 % of all crashes might be related to sleepiness at the wheel. It is desirable to get an objective measurement of driver sleepiness for reduced sensitivity to subjective variations. Using deep learning for classification of driver sleepiness could be a step toward this objective. In this master thesis, deep learning was used for investigating classification of electrophysiological data, electroencephalogram (EEG) and electrooculogram (EOG), from drivers into levels of sleepiness. The EOG reflects eye position and EEG reflects brain activity. Initially, the intention was to include electrocardiogram (ECG), which reflects heart activity, in the research but this data were later excluded. Both raw time series data and data transformed into time-frequency domain representations were fed into the developed neural networks and for comparison manually extracted features were used in a shallow neural network architecture. Investigation of using EOG and EEG data separately as input was performed as well as a combination as input. The data were labeled using the Karolinska Sleepiness Scale, and the scale was divided into two labels "fatigue" and "alert" for binary classification or in five labels for comparison of classification and regression. The effect of example length was investigated using 150 seconds, 60 seconds and 30 seconds data. Different variations of the main network architecture were used depending on the data representation and the best result was given when using a combination of a convolutional neural network (CNN) and a long short-term memory (LSTM) network with time distributed 150 seconds EOG data as input. The accuracy was in this case 80.4 % and the majority of both alert and fatigue epochs were classified correctly with 85.7 % and 66.7 % respectively. Using the optimal threshold from the created receiver operating characteristics (ROC) curve resulted in a more balanced classifier with 76.3 % correctly classified alert examples and 79.2 % correctly classified fatigue examples. The results from the EEG data, both in terms of accuracy and distribution of correctly classified examples, were shown to be less promising compared to EOG data. Combining EOG and EEG signals was shown to slightly increase the proportion of correctly classified fatigue examples. However, more promising results were obtained when balancing the classifier for solely EOG signals. The overall result from this project shows that there are patterns in the data connected to sleepiness that the neural network can find which makes further work on applying deep learning to the area of driver sleepiness interesting.

Approximately 1.35 million people die each year in car accidents and it is the most common cause of death for people aged 5-29. Because of this it is of large interest to be able to detect when a driver enters a sleepy state and to be able to alert the driver. This thesis aims to develop and evaluate a video-based system for driver sleepiness classification, and also to investigate personalised classification. The dataset used in this work consisted of face videos of 89 drivers captured using an IR camera. All drivers drove on a highway in real traffic in four different driving sessions, two at night time while being sleep deprived and two at day time while being alert. The drivers estimated their sleepiness level every 5th minute according to the 9-point Karolinska Sleepiness Scale (KSS).  The proposed driver sleepiness detection system consists of two steps. First, facial features are extracted from videos. Facial features corresponding to 60 seconds of data were used as input to a 1D-convolutional network. The network architecture consisted of five convolution blocks follower by a fully connected (dense) layer and an output layer with linear activation. The KSS-ratings were used as target values. For classification, the output was rounded to the closest KSS-level. KSS-levels 1-4 were merged into a single class, giving a total of 6 classes. The hyperparameters of the developed models were set by finding the best performing model during a grid search. Personalised models were developed by leaving out data from one of the drivers during training, and then finetuning the model parameters of the last dense layer by training on three of the driving sessions from the left out driver and evaluating on the fourth session. On average, the generic models of the developed architecture obtained an accuracy of 22%, a mean absolute error (MAE) of 1.49 and macro-F1 of 0.16. For the personalised models the mean results were an accuracy of 21%, MAE of 1.50 and macro-F1 of 0.14. The generic model showed signs of bias towards the majority classes (low KSS-levels) despite using weighting to compensate for class imbalance. Further, the results did not improve when personalising the model to compensate for individual differences. Future work should investigate if a larger and more balanced dataset is the only remedy, or if algorithm-level methods can be used to get around the problem.

Driver sleepiness is known to be a major contributor to road traffic incidents (RTIs). An initial literature review identified many studies reporting untreated obstructive sleep apnoea (OSA) sufferers as having impaired driving performance and increased RTI risk. It is consistently reported that treatment with continuous positive air pressure (CPAP) improves driving performance and decreases RTI risk, although most of these studies are conducted less than one year after starting treatment. UK law allows treated OSA patients to continue driving if their doctor states that treatment has been successful. Despite the wealth of publications surrounding OSA and driving, 6 key areas were identified from the literature review as not fully investigated, the: (i) prevalence of undiagnosed OSA in heavy goods vehicle (HGV) drivers in the UK; (ii) impact of sleep restriction on long term CPAP treated OSA compared with healthy controls; (iii) ability of treated OSA participants to identify sleepiness when driving; (iv) impact of one night CPAP withdrawal on driving performance; (v) individual difference in driving performance of long term CPAP treated OSA participants; (vi) choice of countermeasures to driver sleepiness by two groups susceptible to driver sleepiness, OSA and HGV drivers. Key areas (i) and (vi) were assessed using questionnaires. 148 HGV drivers were surveyed to assess OSA symptoms and preference of countermeasures to driver sleepiness. All participants completing the driving simulator study were also surveyed. 9.5% of HGV drivers were found to have symptoms of suspected undiagnosed OSA. Additionally the OSA risk factors were more prevalent for HGV drivers than reported in national statistics reports for the general population. The most effective countermeasures to driver sleepiness (caffeine and a nap) were not the most popular. Being part of a susceptible group (OSA or HGV driver) and prior experience of driver sleepiness did not promote effective choice of countermeasure. Key areas (ii) to (v) were assessed using a driving simulator. Driving simulators present a safe environment to test participants in a scenario where they may experience sleepiness without endangering other road users.

Studies have reported that around 20 % of all traffic accidents are caused by a sleepy driver. Sleepy driving has been compared to drunk driving. A sleepy driver is also an issue in the case of automated vehicles in the future. Handing back the control of the vehicle to a sleepy driver is a serious risk. This has increased the need for a sleepiness estimation system that can be used in the car to warn the driver when driving is not recommended. One commonly used method to estimate sleepiness is to study the heart rate variability, HRV, which is said to reflect the activity of the autonomous nervous system, the ANS. The HRV can be expressed through different measures obtained from a signal of RR-intervals. The aim with the thesis is to investigate how well the HRV translates into sleepiness estimation and how the experimental setup might affect the results. In this study, HRV data from 85 sleep deprived drivers was collected together with the drivers’ own ratings of their sleepiness according to the nine graded Karolinska sleepiness scale, KSS. An ANOVA test showed statistical significance for almost all of the used HRV measures when the Driver ID was set as random variable. In order to reduce the number of HRV measures, a feature selection step was performed before training a Support Vector Machine (SVM) used for classification of the data. SVM classifiers are trained to use the input features describing the data to optimize hyperplanes separating the discrete set of classes. Previous research has shown good results in using HRV for sleepiness detection, but common issues are the small data sets used and that most experiments are performed in a simulator instead of at real roads. In some cases, no sleep deprivation is used. The result from the classification in this study is a mean accuracy of around 58-59 %, mean sensitivity of 50-51 %, mean specificity of 75-76 % and mean F1 score of 50-51 % over the three classes ’Alert’, ’Getting sleepy’ and ’Sleepy’. This together with the results of the ANOVA test shows that the HRV measures performed relatively poor when used for classification of the data and that there are large inter-individual differences. This suggests the use of personalized algorithms when developing a sleepiness estimation system and an investigation regarding how other confounding factors could affect the estimation is also motivated.

Falling asleep while operating a moving vehicle is a contributing factor to the statistics of road related accidents. It has been estimated that 20% of all accidents where a vehicle has been involved are due to sleepiness behind the wheel. To prevent accidents and to save lives are of uttermost importance. In this thesis, given the world’s largest dataset of driver participants, two methods of evaluating driver sleepiness have been evaluated. The first method was based on the creation of epochs from lane departures and KSS, whilst the second method was based solely on the creation of epochs based on KSS. From the epochs, a number of features were extracted from both physiological signals and the car’s controller area network. The most important features were selected via a feature selection step, using sequential forward floating selection. The selected features were trained and evaluated on linear SVM, Gaussian SVM, KNN, random forest and adaboost. The random forest classifier was chosen in all cases when classifying previously unseen data.The results shows that method 1 was prone to overfit. Method 2 proved to be considerably better, and did not suffer from overfitting. The test results regarding method 2 were as follows; sensitivity = 80.3%, specificity = 96.3% and accuracy = 93.5%.The most prominent features overall were found in the EEG and EOG domain together with the sleep/wake predictor feature. However indications have been made that complexities might contribute to the detection of sleepiness as well, especially the Higuchi’s fractal dimension.

Driving sleepiness is a serious problem worldwide. It is of interest to develop reliable sleepiness detection systems to implement in vehicles, and for such a system both physi-ological data and driver performance data can be used. The reasons for driver sleepiness can be many, where an interesting factor to consider is the light condition of the environment, specifically daylight and darkness. Daylight and darkness has shown to affect human sleepiness in general and it is therefore of importance to investigate the effect of it on driver sleepiness independent of other factors. This thesis aimed to investigate whether light condition is a parameter that should be considered when developing a sleepiness detection system in a vehicle. This was done by investigating if the course of sleepiness would be affected by daylight and darkness, and if adding light condition information as a parameter to a classification model improved the performance of the sleepiness classification. To achieve this, the study was based upon data collected from driving simulator tests conducted by the Swedish National Road and Transport Research Institute (VTI). Test subjects drove in simulated daylight and darkness during both daytime while rested and nighttime while sleep-deprived. An exploratory and statistical analysis was conducted of several sleepiness indicators extracted from physio-logical data and simulator data. Three different classification models were implemented. The indicators pointed to a higher level of driver sleepiness during night compared to during day, as well as an increase with time on task. However, no clear trends pointed to daylight and darkness having affected the sleepiness of the driver. The classification models showed a marginal improvement when including light condition as a feature, however not large enough to draw any specific conclusion regarding the effect. The conclusion was that an effect of daylight and darkness on the course of driver sleepiness could not be seen in this thesis. The adding of light and dark as a feature did not significantly improve the classification models’ performances. In summary, further investigations of the effect of daylight and darkness in relation to driver sleepiness are needed.