Camera-based photoplethysmography (cbPPG) is a novel measurement technique that allows the continuous monitoring of vital signs by using common video cameras. In the last decade, the technology has attracted a lot of attention as it is easy to set up, operates remotely, and offers new diagnostic opportunities. Despite the growing interest, cbPPG is not completely established yet and is still primarily the object of research. There are a variety of reasons for this lack of development including that reliable and autonomous hardware setups are missing, that robust processing algorithms are needed, that application fields are still limited, and that it is not completely understood which physiological factors impact the captured signal. In this thesis, these issues will be addressed. A new and innovative measuring system for cbPPG was developed. In the course of three large studies conducted in clinical and non-clinical environments, the system’s great flexibility, autonomy, user-friendliness, and integrability could be successfully proven. Furthermore, it was investigated what value optical polarization filtration adds to cbPPG. The results show that a perpendicular filter setting can significantly enhance the signal quality. In addition, the performed analyses were used to draw conclusions about the origin of cbPPG signals: Blood volume changes are most likely the defining element for the signal's modulation. Besides the hardware-related topics, the software topic was addressed. A new method for the selection of regions of interest (ROIs) in cbPPG videos was developed. Choosing valid ROIs is one of the most important steps in the processing chain of cbPPG software. The new method has the advantage of being fully automated, more independent, and universally applicable. Moreover, it suppresses ballistocardiographic artifacts by utilizing a level-set-based approach. The suitability of the ROI selection method was demonstrated on a large and challenging data set. In the last part of the work, a potentially new application field for cbPPG was explored. It was investigated how cbPPG can be used to assess autonomic reactions of the nervous system at the cutaneous vasculature. The results show that changes in the vasomotor tone, i.e. vasodilation and vasoconstriction, reflect in the pulsation strength of cbPPG signals. These characteristics also shed more light on the origin problem. Similar to the polarization analyses, they support the classic blood volume theory. In conclusion, this thesis tackles relevant issues regarding the application of cbPPG. The proposed solutions pave the way for cbPPG to become an established and widely accepted technology.

Objective. After skin flap transplants, perfusion strength monitoring is essential for the early detection of tissue perfusion disorders and thus to ensure the survival of skin flaps. Camera-based photoplethysmography (cbPPG) is a non-contact measurement method, using video cameras and ambient light, which provides spatially resolved information about tissue perfusion. It has not been researched yet whether the measurement depth of cbPPG, which is limited by the penetration depth of ambient light, is sufficient to reach pulsatile vessels and thus to measure the perfusion strength in regions that are relevant for skin flap transplants. Approach. We applied constant negative pressure (compared to ambient pressure) to the anterior thighs of 40 healthy subjects. Seven measurements (two before and five up to 90 min after the intervention) were acquired using an RGB video camera and photospectrometry simultaneously. We investigated the performance of different algorithmic approaches for perfusion strength assessment, including the signal-to-noise ratio (SNR), its logarithmic components logS and logN, amplitude maps, and the amplitude height of alternating and direct signal components. Main results. We found strong correlations of up to r = 0.694 (p < 0.001) between photospectrometric measurements and all cbPPG parameters except SNR when using the green color channel. The transfer of cbPPG signals to POS, CHROM, and O3C did not lead to systematic improvements. However, for direct signal components, the transformation to O3C led to correlations of up to r = 0.744 (p < 0.001) with photospectrometric measurements. Significance. Our results indicate that a camera-based perfusion strength assessment in tissue with deep-seated pulsatile vessels is possible.

Background Camera-based photoplethysmography (cbPPG) is a measurement technique which enables remote vital sign monitoring by using cameras. To obtain valid plethysmograms, proper regions of interest (ROIs) have to be selected in the video data. Most automated selection methods rely on specific spatial or temporal features limiting a broader application. In this work, we present a new method which overcomes those drawbacks and, therefore, allows cbPPG to be applied in an intraoperative environment. Methods We recorded 41 patients during surgery using an RGB and a near-infrared (NIR) camera. A Bayesian skin classifier was employed to detect suitable regions, and a level set segmentation approach to define and track ROIs based on spatial homogeneity. Results The results show stable and homogeneously illuminated ROIs. We further evaluated their quality with regards to extracted cbPPG signals. The green channel provided the best results where heart rates could be correctly estimated in 95.6% of cases. The NIR channel yielded the highest contribution in compensating false estimations. Conclusions The proposed method proved that cbPPG is applicable in intraoperative environments. It can be easily transferred to other settings regardless of which body site is considered.

Background Camera-based photoplethysmography (cbPPG) is a measurement technique which enables remote vital sign monitoring by using cameras. To obtain valid plethysmograms, proper regions of interest (ROIs) have to be selected in the video data. Most automated selection methods rely on specific spatial or temporal features limiting a broader application. In this work, we present a new method which overcomes those drawbacks and, therefore, allows cbPPG to be applied in an intraoperative environment. Methods We recorded 41 patients during surgery using an RGB and a near-infrared (NIR) camera. A Bayesian skin classifier was employed to detect suitable regions, and a level set segmentation approach to define and track ROIs based on spatial homogeneity. Results The results show stable and homogeneously illuminated ROIs. We further evaluated their quality with regards to extracted cbPPG signals. The green channel provided the best results where heart rates could be correctly estimated in 95.6% of cases. The NIR channel yielded the highest contribution in compensating false estimations. Conclusions The proposed method proved that cbPPG is applicable in intraoperative environments. It can be easily transferred to other settings regardless of which body site is considered.

Acute mental stress is an everyday phenomenon that has evidently intensified over the past decades and poses significant health risks. Conventional methods for stress assessment are not suitable for everyday use. They are suitable only for clinical and laboratory assessment because they require full attention, limit the freedom of movement (sensors, cables), often require trained personnel or special equipment, and thus are cost-intensive. This work investigates camera-based photoplethysmography (cbPPG), a non-contact technique for the monitoring of cardiovascular vital signs, as an alternative for the assessment of acute mental stress that is suitable for everyday use. As a non-contact technique cbPPG is considered susceptible to artifacts. To overcome limitations of existing cbPPG methods, this work covers essential developments for the robust extraction of non-contact vital signs in addition to the assessment of acute mental stress. An experimental study was designed and conducted with 65 healthy participants to gain a database for cbPPG including synchronized reference measurements (e.g. electrocardiography, skin conductance, salivary cortisol concentration). The experimental study resulted in the „Dresden Multimodal Biosignal Dataset for the Mannheim Multi-component Stress Test“ (DMBD). In addition, the „Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database“ (BP4D+) was utilized. For robust extraction of non-contact vital signs measured with cbPPG, a novel method for the extraction of cbPPG signals was developed: O3C. O3C optimizes the combination of the color channels of RGB cameras with an evaluation metric in a specialized, systematic grid search. Several investigations on properties of the novel method revealed that the grid search always identified a global optimum. O3C was independent of different skin tones and the choice of evaluation metric. Temporal normalization of the RGB color channels improved the transferability of O3C between datasets (DMBD, BP4D+). At the example of breath rate measurement, it was shown that the method behind O3C is transferable from pulse rate to other vital signs. In addition, a novel method for automatic, reference-free identification of erroneous measurements was developed on the basis of signal quality indexes (SQIs). The developments on robust extraction of non-contact vital signs contribute to the fundamentals of cardiovascular monitoring that is suitable for everyday use. Among other aspects, this forms the basis for non-contact assessment of acute mental stress with cbPPG. In the experimental study (DMBD), conventional reference methods showed distinct changes in psychometric variables, chemical biomarkers, and contact-based vital signs during acute mental stress. The results are widely in line with existing literature and indicated successful activation of the hypothalamic-pituitary-adrenal axis (HPA axis) as well as sympathetic activation of the autonomic nervous system. A special characteristic of this investigation on stress assessment resides in the large variety of synchronized reference parameters, which allows a side-by-side comparison of the effectiveness of different measurement techniques. To assess the physiological reaction to acute mental stress with non-contact technique, ten vital signs derived with cbPPG were analyzed. The cbPPG vital signs registered positive chronotropy, peripheral vasoconstriction, and altered respiration in accordance with reference measurements. Thus, they also successfully indicated sympathetic activation of the autonomic nervous system. In a machine learning approach, the cbPPG vital signs were effective in detecting the immediate stress response with a fairly high temporal resolution of 30 s. These investigations are unique in terms of their extent and the possibility to adduce diverse synchronized reference measurements for comparison. They provide valuable insights into capabilities and effectiveness of cbPPG for non-contact assessment of acute mental stress. The findings of this work pave the way for robust non-contact monitoring with cbPPG. At the example of acute mental stress, a method for physiological assessment of the human state that is suitable for everyday use has been presented. This provides new opportunities to make use of the great potential that cbPPG offers for numerous everyday applications (e.g. telemedical video consultations, adaptive human-machine interfaces).:1 Introduction .. 1.1 Relevance .. 1.2 Scope .. 1.3 Outline .. 1.4 Delineation 2 Physiological Fundamentals .. 2.1 Stress and Strain .. .. 2.1.1 Historical Development .. .. 2.1.2 Definition .. 2.2 Endocrine System .. 2.3 Autonomic Nervous System .. 2.4 Cardiovascular System .. .. 2.4.1 Heart .. .. 2.4.2 Vascular System .. .. 2.4.3 Facial Vasculature .. 2.5 Skin 3 Methods to Assess the Human Response to Acute Mental Stress .. 3.1 Clinical and Laboratory Procedures .. .. 3.1.1 Stress Induction .. .. 3.1.2 Stress Response Assessment .. 3.2 Biomedical Engineering Techniques .. .. 3.2.1 Conventional Techniques .. .. .. 3.2.1.1 Electrocardiography .. .. .. 3.2.1.2 Photoplethysmography .. .. .. 3.2.1.3 Blood Pressure Measurement .. .. .. 3.2.1.4 Electrodermal Activity .. .. .. 3.2.1.5 Vital Signs of Conventional Techniques .. .. 3.2.2 Non-contact Techniques .. .. .. 3.2.2.1 Overview .. .. .. 3.2.2.2 Comparison .. 3.3 Summary 4 Camera-based Photoplethysmography .. 4.1 Functional Principle .. 4.2 Measurement Technology .. 4.3 Pulse Rate Measurement .. 4.4 Algorithms for Signal Extraction .. .. 4.4.1 Image Processing .. .. 4.4.2 Channel Combination .. .. 4.4.3 Signal Processing .. .. 4.4.4 Excursus: A Note on Deep Learning .. .. 4.4.5 Summary .. 4.5 Application to Stress Assessment 5 Study Design .. 5.1 Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database .. 5.2 Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. .. 5.2.1 Protocol .. .. 5.2.2 Setup .. .. 5.2.3 Annotations .. .. 5.2.4 Cohort Summary 6 Investigations on Robust Extraction of Non-contact Vital Signs .. 6.1 Color Space Transformations .. 6.2 Novel Method for the Optimization of Color Channel Combinations .. 6.3 Impact of Skin Tone on the Optimal Color Channel Combination .. 6.4 Impact of Normalization on the Optimal Color Channel Combination .. 6.5 Impact of Evaluation Metric on the Optimal Color Channel Combination .. 6.6 Optimal Color Channel Combination for Breath Rate Measurement .. 6.7 Signal Quality Index Filtering .. 6.8 Summary 7 Investigations on the Assessment of Acute Mental Stress .. 7.1 Examination of Reference Parameters .. 7.2 Examination of Camera-based Vital Signs .. 7.3 Prediction from Camera-based Vital Signs .. 7.4 Summary 8 Conclusion .. 8.1 Summary .. 8.2 Outlook References Appendix .. A Schematic Structure of the Autonomic Nervous System .. B Other Conventional Techniques for Biosignal Acquisition .. C Recording and Synchronization of the Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. D Definition of Regions of Interest From Facial Landmarks .. E Definition of Color Space Transformations .. F Extended Results of Camera-based Pulse Rate Measurement With Different Color Spaces and Regions of Interest .. G Level-Set Regions of Interest in the Experimental Study .. H Relative Accuracy Differences Across the Hemispherical Surface Grid for Multiple Settings .. I Descriptive Statistics for the Reference Vital Signs of the Experimental Study .. J Insignificant Reference Vital Signs of the Experimental Study .. K Statistics for the Binary Logistic Regression with Forward Selection .. .. K.1 Omnibus Tests of Model Coefficients .. .. K.2 Model Summary .. .. K.3 Hosmer and Lemeshow Test .. .. K.4 Classification Table .. .. K.5 Equation Variables / Akuter mentaler Stress ist ein alltägliches Phänomen, dass sich im Laufe der vergangenen Jahrzehnte nachweislich intensiviert hat und ein Risiko für die Gesundheit darstellt. Herkömmliche Methoden zur Stressbewertung sind nicht alltagstauglich. Sie eignen sich nur für Klinik und Labor, da sie volle Aufmerksamkeit erfordern, Bewegungsfreiheit einschränken (Sensoren, Kabel), zumeist Fachpersonal oder Spezialausrüstung voraussetzen und entsprechend kostenintensiv sind. Diese Arbeit beschäftigt sich mit der kamerabasierten Photoplethysmographie (cbPPG), einer kontaktlosen Technik zum Monitoring kardiovaskulärer Vitalparameter, als alltagstaugliche Alternative zur Bewertung der physiologischen Reaktion auf akuten mentalen Stress. Als kontaktlose Technologie gilt cbPPG allerdings als artefaktanfällig. Um Limitationen bestehender Methoden zu überwinden, umfasst diese Arbeit neben der Stressbewertung mit cbPPG essenzielle Weiterentwicklungen zur robusten Extraktion kontaktloser Vitalparameter. Um eine Datenbasis für cbPPG mit zahlreichen Referenzmessverfahren (z. B. Elektrokardiografie, Hautleitfähigkeit, Speichelkortisolkonzentration) zu schaffen, wurde eine Experimentalstudie mit 65 gesunden Probanden aufgesetzt. Daraus resultierte das „Dresden Multimodal Biosignal Dataset for the Mannheim Multi-component Stress Test“ (DMBD). Zusätzlich fand die „Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database“ (BP4D+) Anwendung. Für die robuste Extraktion von Vitalparametern mit cbPPG wurde eine neuartige Methodik zur Signalextraktion entwickelt: O3C. O3C optimiert die Kombination der Farbkanäle einer RGB-Kamera in einer spezialisierten, systematischen Rastersuche anhand einer Evaluationsmetrik. Die Untersuchung zentraler Eigenschaften von O3C zeigte, dass stets ein globales Optimum der Rastersuche existiert und die neue Methode robust gegenüber verschiedenen Hauttönen und Evaluationsmetriken ist. Zeitliche Normalisierung der RGB-Farbkanäle verbesserte die Übertragbarkeit von O3C zwischen verschiedenen Datensätzen (DMBD, BP4D+). Am Beispiel der Atemratenmessung wurde gezeigt, dass die Methodik von O3C auf andere Vitalparameter übertragbar ist. Darüber hinaus wurde eine neue Methode zur referenzfreien Identifikation fehlerhafter Messungen mittels Signalqualitätsindizes (SQIs) entwickelt. Die Entwicklungen zur robusten Extraktion von Vitalparametern leisten einen grundlegenden Beitrag für das alltagstaugliche kardiovaskuläre Monitoring mit cbPPG. Damit schaffen sie unter anderem die Voraussetzung für die kontaktlose Stressbewertung mit cbPPG. Die Referenzmessverfahren der Experimentalstudie (DMBD) zeigten bei akutem mentalem Stress deutliche Veränderungen psychometrischer Variablen, chemischer Biomarker und kontaktbasiert erfasster Vitalparameter. Die Ergebnisse stehen in weitreichender Übereinstimmung mit bisheriger Literatur und wiesen die erfolgreiche Aktivierung der Hypothalamus-Hypophysen-Nebennierenrinden-Achse und die sympathische Aktivierung des autonomen Nervensystems aus. Eine Besonderheit dieser Untersuchung zur Stressbewertung liegt in der Vielfalt synchronisierter Referenzparameter, mit der sich die Effektivität verschiedener Referenzmessverfahren direkt gegenüberstellen lässt. Für die kontaktlose Bewertung der physiologischen Reaktion auf akuten mentalen Stress wurden zehn cbPPG Vitalparameter analysiert. Die cbPPG Vitalparameter erfassten positive Chronotropie, periphere Vasokonstriktion und veränderte Atmung, und zeigten damit ebenfalls die sympathische Aktivierung des autonomen Nervensystems erfolgreich an. Die cbPPG Vitalparameter eigneten sich darüber hinaus zur zuverlässigen automatisierten Detektion der unmittelbaren Stressreaktion mit einer hohen zeitlichen Auflösung von 30 s. Die Untersuchungen sind einzigartig in ihrem Umfang und der Möglichkeit, diverse Referenzmessverfahren zum Vergleich heranzuziehen. Sie liefern damit wertvolle Erkenntnisse über Möglichkeiten und Leistungsfähigkeit von cbPPG zur kontaktlosen Stressbewertung. Die Ergebnisse dieser Arbeit ebnen den Weg für ein robustes kontaktloses Monitoring mittels cbPPG. Am Beispiel akuten mentalen Stresses wurde eine Methode zur alltagstauglichen Bewertung physiologischer Zustände aufgezeigt. Damit eröffnen sich neue Möglichkeiten, das große Potenzial von cbPPG für zahlreiche Anwendungsfälle (z. B. adaptive Mensch-Maschine-Schnittstellen, telemedizinische Videokonsultationen) alltagstauglich zu erschließen.:1 Introduction .. 1.1 Relevance .. 1.2 Scope .. 1.3 Outline .. 1.4 Delineation 2 Physiological Fundamentals .. 2.1 Stress and Strain .. .. 2.1.1 Historical Development .. .. 2.1.2 Definition .. 2.2 Endocrine System .. 2.3 Autonomic Nervous System .. 2.4 Cardiovascular System .. .. 2.4.1 Heart .. .. 2.4.2 Vascular System .. .. 2.4.3 Facial Vasculature .. 2.5 Skin 3 Methods to Assess the Human Response to Acute Mental Stress .. 3.1 Clinical and Laboratory Procedures .. .. 3.1.1 Stress Induction .. .. 3.1.2 Stress Response Assessment .. 3.2 Biomedical Engineering Techniques .. .. 3.2.1 Conventional Techniques .. .. .. 3.2.1.1 Electrocardiography .. .. .. 3.2.1.2 Photoplethysmography .. .. .. 3.2.1.3 Blood Pressure Measurement .. .. .. 3.2.1.4 Electrodermal Activity .. .. .. 3.2.1.5 Vital Signs of Conventional Techniques .. .. 3.2.2 Non-contact Techniques .. .. .. 3.2.2.1 Overview .. .. .. 3.2.2.2 Comparison .. 3.3 Summary 4 Camera-based Photoplethysmography .. 4.1 Functional Principle .. 4.2 Measurement Technology .. 4.3 Pulse Rate Measurement .. 4.4 Algorithms for Signal Extraction .. .. 4.4.1 Image Processing .. .. 4.4.2 Channel Combination .. .. 4.4.3 Signal Processing .. .. 4.4.4 Excursus: A Note on Deep Learning .. .. 4.4.5 Summary .. 4.5 Application to Stress Assessment 5 Study Design .. 5.1 Binghamton-Pittsburgh-RPI Multimodal Spontaneous Emotion Database .. 5.2 Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. .. 5.2.1 Protocol .. .. 5.2.2 Setup .. .. 5.2.3 Annotations .. .. 5.2.4 Cohort Summary 6 Investigations on Robust Extraction of Non-contact Vital Signs .. 6.1 Color Space Transformations .. 6.2 Novel Method for the Optimization of Color Channel Combinations .. 6.3 Impact of Skin Tone on the Optimal Color Channel Combination .. 6.4 Impact of Normalization on the Optimal Color Channel Combination .. 6.5 Impact of Evaluation Metric on the Optimal Color Channel Combination .. 6.6 Optimal Color Channel Combination for Breath Rate Measurement .. 6.7 Signal Quality Index Filtering .. 6.8 Summary 7 Investigations on the Assessment of Acute Mental Stress .. 7.1 Examination of Reference Parameters .. 7.2 Examination of Camera-based Vital Signs .. 7.3 Prediction from Camera-based Vital Signs .. 7.4 Summary 8 Conclusion .. 8.1 Summary .. 8.2 Outlook References Appendix .. A Schematic Structure of the Autonomic Nervous System .. B Other Conventional Techniques for Biosignal Acquisition .. C Recording and Synchronization of the Dresden Multimodal Biosignal Dataset for the Mannheim Multicomponent Stress Test .. D Definition of Regions of Interest From Facial Landmarks .. E Definition of Color Space Transformations .. F Extended Results of Camera-based Pulse Rate Measurement With Different Color Spaces and Regions of Interest .. G Level-Set Regions of Interest in the Experimental Study .. H Relative Accuracy Differences Across the Hemispherical Surface Grid for Multiple Settings .. I Descriptive Statistics for the Reference Vital Signs of the Experimental Study .. J Insignificant Reference Vital Signs of the Experimental Study .. K Statistics for the Binary Logistic Regression with Forward Selection .. .. K.1 Omnibus Tests of Model Coefficients .. .. K.2 Model Summary .. .. K.3 Hosmer and Lemeshow Test .. .. K.4 Classification Table .. .. K.5 Equation Variables

Superpixels are the results of an image oversegmentation. They are an established intermediate level image representation and used for various applications including object detection, 3d reconstruction and semantic segmentation. While there are various approaches to create such segmentations, there is a lack of knowledge about their properties. In particular, there are contradicting results published in the literature. This thesis identifies segmentation quality, stability, compactness and runtime to be important properties of superpixel segmentation algorithms. While for some of these properties there are established evaluation methodologies available, this is not the case for segmentation stability and compactness. Therefore, this thesis presents two novel metrics for their evaluation based on ground truth optical flow. These two metrics are used together with other novel and existing measures to create a standardized benchmark for superpixel algorithms. This benchmark is used for extensive comparison of available algorithms. The evaluation results motivate two novel segmentation algorithms that better balance trade-offs of existing algorithms: The proposed Preemptive SLIC algorithm incorporates a local preemption criterion in the established SLIC algorithm and saves about 80 % of the runtime. The proposed Compact Watershed algorithm combines Seeded Watershed segmentation with compactness constraints to create regularly shaped, compact superpixels at the even higher speed of the plain watershed transformation. Operating autonomous systems over the course of days, weeks or months, based on visual navigation, requires repeated recognition of places despite severe appearance changes as they are for example induced by illumination changes, day-night cycles, changing weather or seasons - a severe problem for existing methods. Therefore, the second part of this thesis presents two novel approaches that incorporate superpixel segmentations in place recognition in changing environments. The first novel approach is the learning of systematic appearance changes. Instead of matching images between, for example, summer and winter directly, an additional prediction step is proposed. Based on superpixel vocabularies, a predicted image is generated that shows, how the summer scene could look like in winter or vice versa. The presented results show that, if certain assumptions on the appearance changes and the available training data are met, existing holistic place recognition approaches can benefit from this additional prediction step. Holistic approaches to place recognition are known to fail in presence of viewpoint changes. Therefore, this thesis presents a new place recognition system based on local landmarks and Star-Hough. Star-Hough is a novel approach to incorporate the spatial arrangement of local image features in the computation of image similarities. It is based on star graph models and Hough voting and particularly suited for local features with low spatial precision and high outlier rates as they are expected in the presence of appearance changes. The novel landmarks are a combination of local region detectors and descriptors based on convolutional neural networks. This thesis presents and evaluates several new approaches to incorporate superpixel segmentations in local region detection. While the proposed system can be used with different types of local regions, in particular the combination with regions obtained from the novel multiscale superpixel grid shows to perform superior to the state of the art methods - a promising basis for practical applications.

Superpixels are the results of an image oversegmentation. They are an established intermediate level image representation and used for various applications including object detection, 3d reconstruction and semantic segmentation. While there are various approaches to create such segmentations, there is a lack of knowledge about their properties. In particular, there are contradicting results published in the literature. This thesis identifies segmentation quality, stability, compactness and runtime to be important properties of superpixel segmentation algorithms. While for some of these properties there are established evaluation methodologies available, this is not the case for segmentation stability and compactness. Therefore, this thesis presents two novel metrics for their evaluation based on ground truth optical flow. These two metrics are used together with other novel and existing measures to create a standardized benchmark for superpixel algorithms. This benchmark is used for extensive comparison of available algorithms. The evaluation results motivate two novel segmentation algorithms that better balance trade-offs of existing algorithms: The proposed Preemptive SLIC algorithm incorporates a local preemption criterion in the established SLIC algorithm and saves about 80 % of the runtime. The proposed Compact Watershed algorithm combines Seeded Watershed segmentation with compactness constraints to create regularly shaped, compact superpixels at the even higher speed of the plain watershed transformation. Operating autonomous systems over the course of days, weeks or months, based on visual navigation, requires repeated recognition of places despite severe appearance changes as they are for example induced by illumination changes, day-night cycles, changing weather or seasons - a severe problem for existing methods. Therefore, the second part of this thesis presents two novel approaches that incorporate superpixel segmentations in place recognition in changing environments. The first novel approach is the learning of systematic appearance changes. Instead of matching images between, for example, summer and winter directly, an additional prediction step is proposed. Based on superpixel vocabularies, a predicted image is generated that shows, how the summer scene could look like in winter or vice versa. The presented results show that, if certain assumptions on the appearance changes and the available training data are met, existing holistic place recognition approaches can benefit from this additional prediction step. Holistic approaches to place recognition are known to fail in presence of viewpoint changes. Therefore, this thesis presents a new place recognition system based on local landmarks and Star-Hough. Star-Hough is a novel approach to incorporate the spatial arrangement of local image features in the computation of image similarities. It is based on star graph models and Hough voting and particularly suited for local features with low spatial precision and high outlier rates as they are expected in the presence of appearance changes. The novel landmarks are a combination of local region detectors and descriptors based on convolutional neural networks. This thesis presents and evaluates several new approaches to incorporate superpixel segmentations in local region detection. While the proposed system can be used with different types of local regions, in particular the combination with regions obtained from the novel multiscale superpixel grid shows to perform superior to the state of the art methods - a promising basis for practical applications.

Visual place recognition is the task of finding same places in a set of database images for a given set of query images. This becomes particularly challenging for long-term applications when the environmental condition changes between or within the database and query set, e.g., from day to night. Visual place recognition in changing environments can be used if global position data like GPS is not available or very inaccurate, or for redundancy. It is required for tasks like loop closure detection in SLAM, candidate selection for global localization, or multi-robot/multi-session mapping and map merging. In contrast to pure image retrieval, visual place recognition can often build upon additional information and data for improvements in performance, runtime, or memory usage. This includes additional data-inherent knowledge about information that is contained in the image sets themselves because of the way they were recorded. Using data-inherent knowledge avoids the dependency on other sensors, which increases the generality of methods for an integration into many existing place recognition pipelines. This thesis focuses on the usage of additional data-inherent knowledge. After the discussion of basics about visual place recognition, the thesis gives a systematic overview of existing data-inherent knowledge and corresponding methods. Subsequently, the thesis concentrates on a deeper consideration and exploitation of four different types of additional data-inherent knowledge. This includes 1) sequences, i.e., the database and query set are recorded as spatio-temporal sequences so that consecutive images are also adjacent in the world, 2) knowledge of whether the environmental conditions within the database and query set are constant or continuously changing, 3) intra-database similarities between the database images, and 4) intra-query similarities between the query images. Except for sequences, all types have received only little attention in the literature so far. For the exploitation of knowledge about constant conditions within the database and query set (e.g., database: summer, query: winter), the thesis evaluates different descriptor standardization techniques. For the alternative scenario of continuous condition changes (e.g., database: sunny to rainy, query: sunny to cloudy), the thesis first investigates the qualitative and quantitative impact on the performance of image descriptors. It then proposes and evaluates four unsupervised learning methods, including our novel clustering-based descriptor standardization method K-STD and three PCA-based methods from the literature. To address the high computational effort of descriptor comparisons during place recognition, our novel method EPR for efficient place recognition is proposed. Given a query descriptor, EPR uses sequence information and intra-database similarities to identify nearly all matching descriptors in the database. For a structured combination of several sources of additional knowledge in a single graph, the thesis presents our novel graphical framework for place recognition. After the minimization of the graph's error with our proposed ICM-based optimization, the place recognition performance can be significantly improved. For an extensive experimental evaluation of all methods in this thesis and beyond, a benchmark for visual place recognition in changing environments is presented, which is composed of six datasets with thirty sequence combinations.