Microsleeps and other lapses of responsiveness can have severe, or even fatal, consequences for people who must maintain high levels of attention on monotonous tasks for long periods of time, e.g., commercial vehicle drivers, pilots, and air-traffic controllers. This thesis describes a head-mounted system which is the first prototype in the process of creating a system that can detect (and possibly predict) these lapses in real time. The system consists of a wearable device which captures multiple physiological signals from the wearer and an extensible software framework for imple- menting signal processing algorithms. Proof-of-concept algorithms are implemented and used to demonstrate that the system can detect simulated microsleeps in real time. The device has three sensing modalities in order to get a better estimate of the user＇s cognitive state than by any one alone. Firstly, it has 16 channels of EEG (8 currently in use) captured by 24-bit ADCs sampling at 250 Hz. The EEG is acquired by custom-built dry electrodes consisting of spring-loaded, gold-plated pins. Secondly, the device has a miniature video camera mounted below one eye, providing 320 x 240 px greyscale video of the eye at 60 fps. The camera module includes infrared illumination so that it can operate in the dark. Thirdly, the device has a six-axis IMU to measure the orientation and movement of the head. These sensors are connected to a Gumstix computer-on-module which transmits the captured data to a remote computer via Wi-Fi. The device has a battery life of about 7.4 h. In addition to this hardware, software to receive and analyse data from the head-mounted device was developed. The software is built around a signal processing pipeline that has been designed to encapsulate a wide variety of signal processing algorithms; feature extractors calculate salient properties of the input data and a classifier fuses these features to determine the user＇s cognitive state. A plug-in system is provided which allows users to write their own signal processing algorithms and to experiment with different combinations of feature extractors and classifiers. Because of this flexible modular design, the system could also be used for applications other than lapse detection‒any application which monitors EEG, eye video, and head movement can be implemented by writing appropriate signal processing plug-ins, e.g., augmented cognition or passive BCIs. The software also provides the ability to configure the device＇s hardware, to save data to disk, and to monitor the system in real time. Plug-ins can be implemented in C++ or Python. A series of validation tests were carried out to confirm that the system operates as intended. Most of the measured parameters were within the expected ranges: EEG amplifier noise = 0.14 μVRMS input-referred, EEG pass band = DC to 47 Hz, camera focus = 2.4 lp/mm at 40 mm, and total latency < 100 ms. Some parameters were worse than expected but still sufficient for effective operation: EEG amplifier CMRR ≥ 82 dB, EEG cross-talk = -17.4 dB, and IMU sampling rate = 10 Hz. The contact impedance of the dry electrodes, measured to be several hundred kilohms, was too high to obtain clean EEG. Three small-scale experiments were done to test the performance of the device in operation on people. The first two demonstrated that the pupil localization algorithm produces PERCLOS values close to those from a manually-rated gold standard and is robust to changes in ambient light levels, iris colour, and the presence of glasses. The final experiment demonstrated that the system is capable of capturing all three physiological signals, transmitting them to the remote computer in real time, extracting features from each signal, and classifying simulated microsleeps from the extracted features. However, this test was successful only when using conventional wet EEG electrodes instead of the dry electrodes built into the device; it will be necessary to find replacement dry electrodes for the device to be useful. The device and associated software form a platform which other researchers can use to develop algorithms for lapse detection. This platform provides data capture hardware and abstracts away the low-level software details so that other researchers are free to focus solely on developing signal processing techniques. In this way, we hope to enable progress towards a practical real-time, real-world lapse detection system.

A device capable of continuously monitoring an individual’s levels of alertness in real-time is highly desirable for preventing drowsiness and lapse related accidents. This thesis presents the development of a non-intrusive and light-insensitive video-based system that uses computer-vision methods to localize face, eyes, and eyelids positions to measure level of eye closure within an image, which, in turn, can be used to identify visible facial signs associated with drowsiness and behavioural microsleeps. The system was developed to be non-intrusive and light-insensitive to make it practical and end-user compliant. To non-intrusively monitor the subject without constraining their movement, the video was collected by placing a camera, a near-infrared (NIR) illumination source, and an NIR-pass optical filter at an eye-to-camera distance of 60 cm from the subject. The NIR-illumination source and filter make the system insensitive to lighting conditions, allowing it to operate in both ambient light and complete darkness without visually distracting the subject. To determine the image characteristics and to quantitatively evaluate the developed methods, reference videos of nine subjects were recorded under four different lighting conditions with the subjects exhibiting several levels of eye closure, head orientations, and eye gaze. For each subject, a set of 66 frontal face reference images was selected and manually annotated with multiple face and eye features. The eye-closure measurement system was developed using a top-down passive feature-detection approach, in which the face region of interest (fROI), eye regions of interests (eROIs), eyes, and eyelid positions were sequentially localized. The fROI was localized using an existing Haar-object detection algorithm. In addition, a Kalman filter was used to stabilize and track the fROI in the video. The left and the right eROIs were localized by scaling the fROI with corresponding proportional anthropometric constants. The position of an eye within each eROI was detected by applying a template-matching method in which a pre-formed eye-template image was cross-correlated with the sub-images derived from the eROI. Once the eye position was determined, the positions of the upper and lower eyelids were detected using a vertical integral-projection of the eROI. The detected positions of the eyelids were then used to measure eye closure. The detection of fROI and eROI was very reliable for frontal-face images, which was considered sufficient for an alertness monitoring system as subjects are most likely facing straight ahead when they are drowsy or about to have microsleep. Estimation of the y- coordinates of the eye, upper eyelid, and lower eyelid positions showed average median errors of 1.7, 1.4, and 2.1 pixels and average 90th percentile (worst-case) errors of 3.2, 2.7, and 6.9 pixels, respectively (1 pixel 1.3 mm in reference images). The average height of a fully open eye in the reference database was 14.2 pixels. The average median and 90th percentile errors of the eye and eyelid detection methods were reasonably low except for the 90th percentile error of the lower eyelid detection method. Poor estimation of the lower eyelid was the primary limitation for accurate eye-closure measurement. The median error of fractional eye-closure (EC) estimation (i.e., the ratio of closed portions of an eye to average height when the eye is fully open) was 0.15, which was sufficient to distinguish between the eyes being fully open, half closed, or fully closed. However, compounding errors in the facial-feature detection methods resulted in a 90th percentile EC estimation error of 0.42, which was too high to reliably determine extent of eye-closure. The eye-closure measurement system was relatively robust to variation in facial-features except for spectacles, for which reflections can saturate much of the eye-image. Therefore, in its current state, the eye-closure measurement system requires further development before it could be used with confidence for monitoring drowsiness and detecting microsleeps.

This master thesis deals with detection of microsleep on the basis of the changes in power spectrum of EEG signal. The results of time-frequency analysis are input values for the classifikation. Proposed classification method uses fuzzy logic. Four classifiers were designed, which are based on a fuzzy inference systems, that are differ in rule base. The results of fuzzy clustering are used for the design of rule premises membership functions. The two classifiers microsleep detection use only alpha band of the EEG signal’s spectrogram then allows the detection of the relaxation state of a person. Unlike to first and second classifiers, the third classifier is supplemented with rules for the delta band, which makes it possible to distinguish the 3 states: vigilance, relaxation and somnolence. The fourth classifier inference system includes the rules for the whole spectrum band. The method was implemented by computer. The program with a graphical user interface was created.