Wearable sensors can be used to monitor opioid use and other key behaviors of interest, and to prompt interventions that promote behavioral change. The effectiveness of such systems is threatened by the potential of a subject's deliberate non-compliance (DNC) to the monitoring. We define deliberate non-compliance as the process of giving one's device to someone else when surveillance is on-going. The principal aim of this thesis is to develop an approach to leverage movement and cardiac features from a wearable sensor to detect such deliberate non-compliance by individuals under surveillance for opioid use. Data from 11 participants who presented to the Emergency Department following an opioid overdose was analyzed. Using a personalized machine learning classifier (model), we evaluated if a snippet of blood volume pulse (BVP) and accelerometer data received is coming from the expected participant or an alternate person. Analysis of our classier shows the viability of this approach, as we were able to detect DNC (or compliance) with over 90% accuracy within 3 seconds of its occurrence.

Effective game music is typically adaptive, interactive, or both. Changes in game music are usually influenced by the current state of the game or the actions of the player. To provide another dimension of interactivity, it would be useful to know the affective state of the human player. Biosignals are continuous signals generated by a person that can be measured over time, and have been shown to reflect affective state. This project demonstrates that control signals can be gathered from the player and mapped to musical parameters. Using a heart rate sensor and galvanic skin response sensor built from open source designs, we have used biosignals to control music playback while playing four games from different genres. A system for controlling game music with biosignals is computationally cheap, and can provide data that is useful to other game systems. The prototype developed for this project is basic, but with further research and development, we believe such a system will greatly improve the immersive experience of video games by involving the player on a new level. / text

Recording and monitoring vital signs is an essential aspect of home-based healthcare. Using contact sensors to record physiological signals can cause discomfort to patients, especially after prolonged use. Hence, remote physiological measurement approaches have attracted considerable attention as they do not require physical contact with the patient’s skin. Several studies proposed techniques to measure Heart Rate (HR) and Heart Rate Variability (HRV) by detecting the Blood Volume Pulse (BVP) from human facial video recordings while the subject is in a resting condition. In this thesis, we focus on the measurement of HR. We adopt an algorithm that uses the Independent Component Analysis (ICA) to separate the source (physiological) signal from noise in the RGB channels of a facial video. We generalize existing methods to support subject movement during video recording. When a subject is moving, the face may be turned away from the camera. We utilize multiple cameras to enable the algorithm to monitor the vital sign continuously, even if the subject leaves the frame or turns away from a subset of the system’s cameras. Furthermore, we improve the accuracy of existing methods by implementing a light equalization scheme to reduce the effect of shadows and unequal facial light on the HR estimation, a machine learning method to select the most accurate channel outputted by the ICA module, and a regression technique to adjust the initial HR estimate. We systematically test our method on eleven subjects using four cameras. The proposed method decreases the RMSE by 27% compared to the state of the art in the rest condition. When the subject is in motion, the proposed method achieves a RMSE of 1.12 bpm using a single camera and RMSE of 0.96 bpm using multiple camera.

Heart Rate Variability (HRV) analysis has been garnering attention from researchers due to its wide range of applications. Medical researchers have always been interested in Heart Rate (HR) and HRV analysis, but nowadays, investigators from variety of other fields are also probing the subject. For instance, variation in HR and HRV is connected to emotional arousal. Therefore, knowledge from the fields of affective computing and psychology, can be employed to devise machines that understand the emotional states of humans. Recent advancements in non-contact HR and HRV measurement techniques will likely further boost interest in emotional estimation through . Such measurement methods involve the extraction of the photoplethysmography (PPG) signal from the human's face through a camera. The latest approaches apply Independent Component Analysis (ICA) on the color channels of video recordings to extract a PPG signal. Other investigated methods rely on Eulerian Video Magnification (EVM) to detect subtle changes in skin color associated with PPG. The effectiveness of the EVM in HR estimation has well been established. However, to the best of our knowledge, EVM has not been successfully employed to extract HRV feature from a video of a human face. In contrast, ICA based methods have been successfully used for HRV analysis. As we demonstrate in this thesis, these two approaches for HRV feature extraction are highly sensitive to noise. Hence, when we evaluated them in indoor settings, we obtained mean absolute error in the range of 0.012 and 28.4. Therefore, in this thesis, we present two approaches to minimize the error rate when estimating physiological measurements from recorded facial videos using a standard camera. In our first approach which is based on the EVM method, we succeeded in extracting HRV measurements but we could not get rid of high frequency noise, which resulted in a high error percentage for the result of the High frequency (HF) component. Our second proposed approach solved this issue by applying ICA on the red, green and blue (RGB) colors channels and we were able to achieve lower error rates and less noisy signal as compared to previous related works. This was done by using a Buterworth filter with the subject's specific HR range as its Cut-Off. The methods were tested with 12 subjects from the DISCOVER lab at the University of Ottawa, using artificial lights as the only source of illumination. This made it a challenge for us because artificial light produces HF signals which can interfere with the PPG signal. The final results show that our proposed ICA based method has a mean absolute error (MAE) of 0.006, 0.005, 0.34, 0.57 and 0.419 for the mean HR, mean RR, LF, HF and LF/HF respectively. This approach also shows that these physiological parameters are highly correlated with the results taken from the electrocardiography (ECG).

The dissertation investigates the promises and challenges for application of cardiac signals in biometrics and affective computing, and noninvasive measurement of cardiac signals. We have mainly discussed two major cardiac signals: electrocardiogram (ECG), and photoplethysmogram (PPG). ECG and PPG signals hold strong potential for biometric authentications and identifications. We have shown that by mapping each cardiac beat from time domain to an angular domain using a limit cycle, intra-class variability can be significantly minimized. This is in contrary to conventional time domain analysis. Our experiments with both ECG and PPG signal shows that the proposed method eliminates the effect of instantaneous heart rate on the shape morphology and improves authentication accuracy. For noninvasive measurement of PPG beats, we have developed a systematic algorithm to extract pulse rate from face video in diverse situations using video magnification. We have extracted signals from skin patches and then used frequency domain correlation to filter out non-cardiac signals. We have developed a novel entropy based method to automatically select skin patches from face. We report beat-to-beat accuracy of remote PPG (rPPG) in comparison to conventional average heart rate. The beat-to-beat accuracy is required for applications related to heart rate variability (HRV) and affective computing. The algorithm has been tested on two datasets, one with static illumination condition and the other with unrestricted ambient illumination condition. Automatic skin detection is an intermediate step for rPPG. Existing methods always depend on color information to detect human skin. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. We have used LBP lacunarity based micro-textures features and a region growing algorithm to find skin pixels in an image. Our experiment shows that the proposed method is applicable universally to any image including near infra-red images. This finding helps to extend the domain of many application including rPPG. To the best of our knowledge, this is first such method that is independent of color cues. / Ph. D. / The heart is an integral part of the human body. With every beat, the heart continuously pumps oxygen-enriched blood to providing fuel to our cells and thus enabling life. The heartbeat is initiated by electrical signals generated in the heart muscles. This electrical activity, which are often governed by our autonomic nervous system, can be measured directly by electrocardiogram (ECG) using advanced and often obtrusive instrumentation. Photoplethysmogram (PPG), on the other hand, measures how the blood volume changes and can be readily measured with inexpensive instrumentation at certain locations (e.g. at the fingertip). The ECG and PPG are widely used cardiac signals in medical science for diagnosis and health monitoring. But, these signals hold greater potential than just its medical diagnostic applications. In this work, we have mainly investigated if these signals can be used to identify an individual. Every human heart differs by their size, shape, locations inside body, and internal structure. This motivated us to represent the signals using a mathematical model and use machine learning algorithm to identify individual persons. We have discussed how our method improves the identification accuracy and can be used with current biometric methods like fingerprint in our phone. The measurement procedures of cardiac signals are often cumbersome and need instruments which may not be available outside medical facilities. Therefore, we have investigated alternative method of remote photoplethysmography (rPPG) that are relatively inexpensive and unobtrusive. In this dissertation, we have used face video of an individual to extract the heart rate information. The flow of blood causes small changes in the color of face skin. This is not visible to human eyes without digital magnification, but we have shown how knowledge of distinct behavior of human heart rate and use of advanced computer vision algorithms helped us to extract vital signals like heart rate with a significant accuracy. In addition, to measure rPPG using face video, we integrated a method for automatic detection of skin from images and videos. Existing skin detection methods depended on color information which is not always available within available video sources. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. Our method relies on the context and the texture based appearance of skin. To the best of our knowledge, this is first such method that is independent of color cues. In summary, the dissertation investigates the promises and challenges for application of cardiac signals in biometrics and nonobtrusive measurement of cardiac signals using face video.

La détection et l’évaluation des émotions sont des domaines qui suscitent un grand intérêt par de nombreuses communautés tant au niveau des sciences humaines que des sciences exactes. Dans cette thèse nous nous intéressons à la reconnaissance de l’anxiété sociale qui est une peur irrationnelle ressentie par une personne lors de toute forme de relation sociale. L’anxiété peut être révélée par un ensemble de traits physiques et physiologiques tels que l’intonation de la voix, les mimiques faciales, l’augmentation du rythme cardiaque, le rougissement… etc. L’avantage de l’utilisation des mesures physiologiques est que les individus ne peuvent pas les manipuler, c’est une source continue de données et chaque émotion est caractérisée par une variation physiologique particulière. Dans ce travail, nous proposons un système de mesure d’anxiété basé sur l’utilisation d’un seul signal physiologique « signal de pression sanguine volumique (Blood volume pulse BVP)». Le choix d’un seul capteur limite la gêne des sujets due au nombre de capteurs. De ce signal nous avons sélectionné des paramètres pertinents représentant au mieux les relations étroites du signal BVP avec le processus émotionnel de l’anxiété. Cet ensemble de paramètres est classé en utilisant les séparateurs à vastes marges SVM. Les travaux engagés dans le domaine de la reconnaissance des émotions utilisent fréquemment, pour support d’information, des données peu fiables ne correspondant pas toujours aux situations envisagées. Ce manque de fiabilité peut être dû à plusieurs paramètres parmi eux la subjectivité de la méthode d’évaluation utilisée (questionnaire, auto-évaluation des sujets, …etc.). Nous avons développé une approche d’évaluation objective des données basée sur les dynamiques des paramètres sélectionnés. La base de données utilisée a été enregistrée dans notre laboratoire dans des conditions réelles acquises sur des sujets présentant un niveau d’anxiété face aux situations sociales et qui ne sont pas sous traitement psychologique. L’inducteur utilisé est l’exposition à des environnements virtuels représentant quelques situations sociales redoutées. L’étape d’évaluation, nous a permis d’obtenir un modèle de données fiable pour la reconnaissance de deux niveaux d’anxiété. Ce modèle a été testé dans une clinique spécialisée dans les thérapies cognitives comportementales (TCC) sur des sujets phobiques. Les résultats obtenus mettent en lumière la fiabilité du modèle construit notamment pour la reconnaissance des niveaux d’anxiété sur des sujets sains ou sur des sujets phobiques ce qui constitue une solution au manque de données dont souffrent les différents domaines de reconnaissances / Detection and evaluation of emotions are areas of great interest in many communities both in terms of human and exact sciences. In this thesis we focus on social anxiety recognition, which is an irrational fear felt by a person during any form of social relationship. Anxiety can be revealed by a set of physical and physiological traits such as tone of voice, facial expressions, increased heart rate, flushing ... etc. The interest to the physiological measures is motivated by them robustness to avoid the artifacts created by human social masking, they are a continuous source of data and each emotion is characterized by a particular physiological variation. In this work, we propose a measurement system based on the use of a single physiological signal "Blood volume pulse BVP". The use of a single sensor limits the subjects’ discomfort. From the BVP signal we selected three relevant features which best represents the close relationship between this signal and anxiety status. This features set is classified using support vector machine SVM. The work undertaken in the field of emotion recognition frequently use, for information support, unreliable data do not always correspond to the situations envisaged. This lack of reliability may be due to several parameters among them the subjectivity of the evaluation method used (self-evaluation questionnaire, subjects…etc.). We have developed an approach to objective assessment of data based on the dynamics of selected features. The used database was recorded in our laboratory under real conditions acquired in subjects with a level of anxiety during social situations and who are not under psychological treatment. The used stimulus is the exposition to virtual environments representing some feared social situations. After the evaluation stage, we obtained a reliable model for the recognition of two levels of anxiety. The latter was tested in a clinic specializing in cognitive behavioral therapy (CBT) on phobic subjects. The results highlight the reliability of the built model specifically for the recognition of anxiety levels in healthy subjects or of phobic subjects, what constitutes a solution to the lack of data affecting different areas of recognition