This work addresses the problem of recovering the morphology of blood volume pulse (BVP) information from a video of a person's face. Video-based remote plethysmography methods have shown promising results in estimating vital signs such as heart rate and breathing rate. However, recovering the instantaneous pulse rate signals is still a challenge for the community. This is due to the fact that most of the previous methods concentrate on capturing the temporal average of the cardiovascular signals. In contrast, we present an approach in which BVP signals are extracted with a focus on the recovery of the signal morphology as a generalized form for the computation of physiological metrics. We also place emphasis on allowing natural movements by the subject. Furthermore, our system is capable of extracting individual BVP instances with sufficient signal detail to facilitate candidate re-identification. These improvements have resulted in part from the incorporation of a robust skin-detection module into the overall imaging-based photoplethysmography (iPPG) framework. We present extensive experimental results using the challenging UBFC-Phys dataset and the well-known COHFACE dataset. The source code is available at https://github.com/yogeshd21/CVPM-2023-iPPG-Paper. / Master of Science / In this work we are trying to study and recover human health related metrics and the physiological signals which are at the core for the derivation of such metrics. A well know form of physiological signals is ECG (Electrocardiogram) signals and for our research we work with BVP (Blood Volume Pulse) signals. With this work we are proposing a Deep Learning based model for non-invasive retrieval of human physiological signals from human face videos. Most of the state of the art models as well as researchers try to recover averaged cardiac pulse based metrics like heart rate, breathing rate, etc. without focusing on the details of the recovered physiological signal. Physiological signals like BVP have details like systolic peak, diastolic peak and dicrotic notch, and these signals also have applications in various domains like human mental health study, emotional stimuli study, etc. Hence with this work we focus on retrieval of the morphology of such physiological signals and present a quantitative as well as qualitative results for the same. An efficient attention based deep learning model is presented and scope of reidentification using the retrieved signals is also explored. Along with significant implementations like skin detection model our proposed architecture also shows better performance than state of the art models for two very challenging datasets UBFC-Phys as well as COHFACE. The source code is available at https://github.com/yogeshd21/CVPM-2023-iPPG-Paper.

This thesis considers methods for extracting blood volume pulse (BVP) representations from video of the human face. Whereas most previous systems have been concerned with estimating vital signs such as average heart rate, this thesis addresses the more difficult problem of recovering BVP signal morphology. We present a new approach that is inspired by temporal encoder-decoder architectures that have been used for audio signal separation. As input, this system accepts a temporal sequence of RGB (red, green, blue) values that have been spatially averaged over a small portion of the face. The output of the system is a temporal sequence that approximates a BVP signal. In order to reduce noise in the recovered signal, a separate processing step extracts individual pulses and performs normalization and outlier removal. After these steps, individual pulse shapes have been extracted that are sufficiently distinct to support biometric authentication. Our findings demonstrate the effectiveness of our approach in extracting BVP signal morphology from facial videos, which presents exciting opportunities for further research in this area. The source code is available at https://github.com/Adleof/CVPM-2023-Temporal-Encoder-Decoder-iPPG / Master of Science / This thesis considers methods for extracting blood volume pulse (BVP) representations from video of the human face. We present a new approach that is inspired by the method that has been used for audio signal separation. The output of our system is an approximation of the BVP signal of the person in the video. Our method can extract a signal that is sufficiently distinct to support biometric authentication. Our findings demonstrate the effectiveness of our approach in extracting BVP signal morphology from facial videos, which presents exciting opportunities for further research in this area.

Acute mental stress is an everyday phenomenon that has evidently intensified over the past decades and poses significant health risks. Conventional methods for stress assessment are not suitable for everyday use. They are suitable only for clinical and laboratory assessment because they require full attention, limit the freedom of movement (sensors, cables), often require trained personnel or special equipment, and thus are cost-intensive. This work investigates camera-based photoplethysmography (cbPPG), a non-contact technique for the monitoring of cardiovascular vital signs, as an alternative for the assessment of acute mental stress that is suitable for everyday use. As a non-contact technique cbPPG is considered susceptible to artifacts. To overcome limitations of existing cbPPG methods, this work covers essential developments for the robust extraction of non-contact vital signs in addition to the assessment of acute mental stress. An experimental study was designed and conducted with 65 healthy participants to gain a database for cbPPG including synchronized reference measurements (e.g. electrocardiography, skin conductance, salivary cortisol concentration). The experimental study resulted in the „Dresden Multimodal Biosignal Dataset for the Mannheim Multi-component Stress Test“ (DMBD). In addition, the „Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database“ (BP4D+) was utilized. For robust extraction of non-contact vital signs measured with cbPPG, a novel method for the extraction of cbPPG signals was developed: O3C. O3C optimizes the combination of the color channels of RGB cameras with an evaluation metric in a specialized, systematic grid search. Several investigations on properties of the novel method revealed that the grid search always identified a global optimum. O3C was independent of different skin tones and the choice of evaluation metric. Temporal normalization of the RGB color channels improved the transferability of O3C between datasets (DMBD, BP4D+). At the example of breath rate measurement, it was shown that the method behind O3C is transferable from pulse rate to other vital signs. In addition, a novel method for automatic, reference-free identification of erroneous measurements was developed on the basis of signal quality indexes (SQIs). The developments on robust extraction of non-contact vital signs contribute to the fundamentals of cardiovascular monitoring that is suitable for everyday use. Among other aspects, this forms the basis for non-contact assessment of acute mental stress with cbPPG. In the experimental study (DMBD), conventional reference methods showed distinct changes in psychometric variables, chemical biomarkers, and contact-based vital signs during acute mental stress. The results are widely in line with existing literature and indicated successful activation of the hypothalamic-pituitary-adrenal axis (HPA axis) as well as sympathetic activation of the autonomic nervous system. A special characteristic of this investigation on stress assessment resides in the large variety of synchronized reference parameters, which allows a side-by-side comparison of the effectiveness of different measurement techniques. To assess the physiological reaction to acute mental stress with non-contact technique, ten vital signs derived with cbPPG were analyzed. The cbPPG vital signs registered positive chronotropy, peripheral vasoconstriction, and altered respiration in accordance with reference measurements. Thus, they also successfully indicated sympathetic activation of the autonomic nervous system. In a machine learning approach, the cbPPG vital signs were effective in detecting the immediate stress response with a fairly high temporal resolution of 30 s. These investigations are unique in terms of their extent and the possibility to adduce diverse synchronized reference measurements for comparison. They provide valuable insights into capabilities and effectiveness of cbPPG for non-contact assessment of acute mental stress. The findings of this work pave the way for robust non-contact monitoring with cbPPG. At the example of acute mental stress, a method for physiological assessment of the human state that is suitable for everyday use has been presented. This provides new opportunities to make use of the great potential that cbPPG offers for numerous everyday applications (e.g. telemedical video consultations, adaptive human-machine interfaces).:1 Introduction .. 1.1 Relevance .. 1.2 Scope .. 1.3 Outline .. 1.4 Delineation 2 Physiological Fundamentals .. 2.1 Stress and Strain .. .. 2.1.1 Historical Development .. .. 2.1.2 Definition .. 2.2 Endocrine System .. 2.3 Autonomic Nervous System .. 2.4 Cardiovascular System .. .. 2.4.1 Heart .. .. 2.4.2 Vascular System .. .. 2.4.3 Facial Vasculature .. 2.5 Skin 3 Methods to Assess the Human Response to Acute Mental Stress .. 3.1 Clinical and Laboratory Procedures .. .. 3.1.1 Stress Induction .. .. 3.1.2 Stress Response Assessment .. 3.2 Biomedical Engineering Techniques .. .. 3.2.1 Conventional Techniques .. .. .. 3.2.1.1 Electrocardiography .. .. .. 3.2.1.2 Photoplethysmography .. .. .. 3.2.1.3 Blood Pressure Measurement .. .. .. 3.2.1.4 Electrodermal Activity .. .. .. 3.2.1.5 Vital Signs of Conventional Techniques .. .. 3.2.2 Non-contact Techniques .. .. .. 3.2.2.1 Overview .. .. .. 3.2.2.2 Comparison .. 3.3 Summary 4 Camera-based Photoplethysmography .. 4.1 Functional Principle .. 4.2 Measurement Technology .. 4.3 Pulse Rate Measurement .. 4.4 Algorithms for Signal Extraction .. .. 4.4.1 Image Processing .. .. 4.4.2 Channel Combination .. .. 4.4.3 Signal Processing .. .. 4.4.4 Excursus: A Note on Deep Learning .. .. 4.4.5 Summary .. 4.5 Application to Stress Assessment 5 Study Design .. 5.1 Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database .. 5.2 Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. .. 5.2.1 Protocol .. .. 5.2.2 Setup .. .. 5.2.3 Annotations .. .. 5.2.4 Cohort Summary 6 Investigations on Robust Extraction of Non-contact Vital Signs .. 6.1 Color Space Transformations .. 6.2 Novel Method for the Optimization of Color Channel Combinations .. 6.3 Impact of Skin Tone on the Optimal Color Channel Combination .. 6.4 Impact of Normalization on the Optimal Color Channel Combination .. 6.5 Impact of Evaluation Metric on the Optimal Color Channel Combination .. 6.6 Optimal Color Channel Combination for Breath Rate Measurement .. 6.7 Signal Quality Index Filtering .. 6.8 Summary 7 Investigations on the Assessment of Acute Mental Stress .. 7.1 Examination of Reference Parameters .. 7.2 Examination of Camera-based Vital Signs .. 7.3 Prediction from Camera-based Vital Signs .. 7.4 Summary 8 Conclusion .. 8.1 Summary .. 8.2 Outlook References Appendix .. A Schematic Structure of the Autonomic Nervous System .. B Other Conventional Techniques for Biosignal Acquisition .. C Recording and Synchronization of the Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. D Definition of Regions of Interest From Facial Landmarks .. E Definition of Color Space Transformations .. F Extended Results of Camera-based Pulse Rate Measurement With Different Color Spaces and Regions of Interest .. G Level-Set Regions of Interest in the Experimental Study .. H Relative Accuracy Differences Across the Hemispherical Surface Grid for Multiple Settings .. I Descriptive Statistics for the Reference Vital Signs of the Experimental Study .. J Insignificant Reference Vital Signs of the Experimental Study .. K Statistics for the Binary Logistic Regression with Forward Selection .. .. K.1 Omnibus Tests of Model Coefficients .. .. K.2 Model Summary .. .. K.3 Hosmer and Lemeshow Test .. .. K.4 Classification Table .. .. K.5 Equation Variables / Akuter mentaler Stress ist ein alltägliches Phänomen, dass sich im Laufe der vergangenen Jahrzehnte nachweislich intensiviert hat und ein Risiko für die Gesundheit darstellt. Herkömmliche Methoden zur Stressbewertung sind nicht alltagstauglich. Sie eignen sich nur für Klinik und Labor, da sie volle Aufmerksamkeit erfordern, Bewegungsfreiheit einschränken (Sensoren, Kabel), zumeist Fachpersonal oder Spezialausrüstung voraussetzen und entsprechend kostenintensiv sind. Diese Arbeit beschäftigt sich mit der kamerabasierten Photoplethysmographie (cbPPG), einer kontaktlosen Technik zum Monitoring kardiovaskulärer Vitalparameter, als alltagstaugliche Alternative zur Bewertung der physiologischen Reaktion auf akuten mentalen Stress. Als kontaktlose Technologie gilt cbPPG allerdings als artefaktanfällig. Um Limitationen bestehender Methoden zu überwinden, umfasst diese Arbeit neben der Stressbewertung mit cbPPG essenzielle Weiterentwicklungen zur robusten Extraktion kontaktloser Vitalparameter. Um eine Datenbasis für cbPPG mit zahlreichen Referenzmessverfahren (z. B. Elektrokardiografie, Hautleitfähigkeit, Speichelkortisolkonzentration) zu schaffen, wurde eine Experimentalstudie mit 65 gesunden Probanden aufgesetzt. Daraus resultierte das „Dresden Multimodal Biosignal Dataset for the Mannheim Multi-component Stress Test“ (DMBD). Zusätzlich fand die „Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database“ (BP4D+) Anwendung. Für die robuste Extraktion von Vitalparametern mit cbPPG wurde eine neuartige Methodik zur Signalextraktion entwickelt: O3C. O3C optimiert die Kombination der Farbkanäle einer RGB-Kamera in einer spezialisierten, systematischen Rastersuche anhand einer Evaluationsmetrik. Die Untersuchung zentraler Eigenschaften von O3C zeigte, dass stets ein globales Optimum der Rastersuche existiert und die neue Methode robust gegenüber verschiedenen Hauttönen und Evaluationsmetriken ist. Zeitliche Normalisierung der RGB-Farbkanäle verbesserte die Übertragbarkeit von O3C zwischen verschiedenen Datensätzen (DMBD, BP4D+). Am Beispiel der Atemratenmessung wurde gezeigt, dass die Methodik von O3C auf andere Vitalparameter übertragbar ist. Darüber hinaus wurde eine neue Methode zur referenzfreien Identifikation fehlerhafter Messungen mittels Signalqualitätsindizes (SQIs) entwickelt. Die Entwicklungen zur robusten Extraktion von Vitalparametern leisten einen grundlegenden Beitrag für das alltagstaugliche kardiovaskuläre Monitoring mit cbPPG. Damit schaffen sie unter anderem die Voraussetzung für die kontaktlose Stressbewertung mit cbPPG. Die Referenzmessverfahren der Experimentalstudie (DMBD) zeigten bei akutem mentalem Stress deutliche Veränderungen psychometrischer Variablen, chemischer Biomarker und kontaktbasiert erfasster Vitalparameter. Die Ergebnisse stehen in weitreichender Übereinstimmung mit bisheriger Literatur und wiesen die erfolgreiche Aktivierung der Hypothalamus-Hypophysen-Nebennierenrinden-Achse und die sympathische Aktivierung des autonomen Nervensystems aus. Eine Besonderheit dieser Untersuchung zur Stressbewertung liegt in der Vielfalt synchronisierter Referenzparameter, mit der sich die Effektivität verschiedener Referenzmessverfahren direkt gegenüberstellen lässt. Für die kontaktlose Bewertung der physiologischen Reaktion auf akuten mentalen Stress wurden zehn cbPPG Vitalparameter analysiert. Die cbPPG Vitalparameter erfassten positive Chronotropie, periphere Vasokonstriktion und veränderte Atmung, und zeigten damit ebenfalls die sympathische Aktivierung des autonomen Nervensystems erfolgreich an. Die cbPPG Vitalparameter eigneten sich darüber hinaus zur zuverlässigen automatisierten Detektion der unmittelbaren Stressreaktion mit einer hohen zeitlichen Auflösung von 30 s. Die Untersuchungen sind einzigartig in ihrem Umfang und der Möglichkeit, diverse Referenzmessverfahren zum Vergleich heranzuziehen. Sie liefern damit wertvolle Erkenntnisse über Möglichkeiten und Leistungsfähigkeit von cbPPG zur kontaktlosen Stressbewertung. Die Ergebnisse dieser Arbeit ebnen den Weg für ein robustes kontaktloses Monitoring mittels cbPPG. Am Beispiel akuten mentalen Stresses wurde eine Methode zur alltagstauglichen Bewertung physiologischer Zustände aufgezeigt. Damit eröffnen sich neue Möglichkeiten, das große Potenzial von cbPPG für zahlreiche Anwendungsfälle (z. B. adaptive Mensch-Maschine-Schnittstellen, telemedizinische Videokonsultationen) alltagstauglich zu erschließen.:1 Introduction .. 1.1 Relevance .. 1.2 Scope .. 1.3 Outline .. 1.4 Delineation 2 Physiological Fundamentals .. 2.1 Stress and Strain .. .. 2.1.1 Historical Development .. .. 2.1.2 Definition .. 2.2 Endocrine System .. 2.3 Autonomic Nervous System .. 2.4 Cardiovascular System .. .. 2.4.1 Heart .. .. 2.4.2 Vascular System .. .. 2.4.3 Facial Vasculature .. 2.5 Skin 3 Methods to Assess the Human Response to Acute Mental Stress .. 3.1 Clinical and Laboratory Procedures .. .. 3.1.1 Stress Induction .. .. 3.1.2 Stress Response Assessment .. 3.2 Biomedical Engineering Techniques .. .. 3.2.1 Conventional Techniques .. .. .. 3.2.1.1 Electrocardiography .. .. .. 3.2.1.2 Photoplethysmography .. .. .. 3.2.1.3 Blood Pressure Measurement .. .. .. 3.2.1.4 Electrodermal Activity .. .. .. 3.2.1.5 Vital Signs of Conventional Techniques .. .. 3.2.2 Non-contact Techniques .. .. .. 3.2.2.1 Overview .. .. .. 3.2.2.2 Comparison .. 3.3 Summary 4 Camera-based Photoplethysmography .. 4.1 Functional Principle .. 4.2 Measurement Technology .. 4.3 Pulse Rate Measurement .. 4.4 Algorithms for Signal Extraction .. .. 4.4.1 Image Processing .. .. 4.4.2 Channel Combination .. .. 4.4.3 Signal Processing .. .. 4.4.4 Excursus: A Note on Deep Learning .. .. 4.4.5 Summary .. 4.5 Application to Stress Assessment 5 Study Design .. 5.1 Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database .. 5.2 Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. .. 5.2.1 Protocol .. .. 5.2.2 Setup .. .. 5.2.3 Annotations .. .. 5.2.4 Cohort Summary 6 Investigations on Robust Extraction of Non-contact Vital Signs .. 6.1 Color Space Transformations .. 6.2 Novel Method for the Optimization of Color Channel Combinations .. 6.3 Impact of Skin Tone on the Optimal Color Channel Combination .. 6.4 Impact of Normalization on the Optimal Color Channel Combination .. 6.5 Impact of Evaluation Metric on the Optimal Color Channel Combination .. 6.6 Optimal Color Channel Combination for Breath Rate Measurement .. 6.7 Signal Quality Index Filtering .. 6.8 Summary 7 Investigations on the Assessment of Acute Mental Stress .. 7.1 Examination of Reference Parameters .. 7.2 Examination of Camera-based Vital Signs .. 7.3 Prediction from Camera-based Vital Signs .. 7.4 Summary 8 Conclusion .. 8.1 Summary .. 8.2 Outlook References Appendix .. A Schematic Structure of the Autonomic Nervous System .. B Other Conventional Techniques for Biosignal Acquisition .. C Recording and Synchronization of the Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. D Definition of Regions of Interest From Facial Landmarks .. E Definition of Color Space Transformations .. F Extended Results of Camera-based Pulse Rate Measurement With Different Color Spaces and Regions of Interest .. G Level-Set Regions of Interest in the Experimental Study .. H Relative Accuracy Differences Across the Hemispherical Surface Grid for Multiple Settings .. I Descriptive Statistics for the Reference Vital Signs of the Experimental Study .. J Insignificant Reference Vital Signs of the Experimental Study .. K Statistics for the Binary Logistic Regression with Forward Selection .. .. K.1 Omnibus Tests of Model Coefficients .. .. K.2 Model Summary .. .. K.3 Hosmer and Lemeshow Test .. .. K.4 Classification Table .. .. K.5 Equation Variables