Background. Stress can profoundly affect human behavior. Critical-infrastructure operators (e.g., at nuclear power plants) may make more errors when overstressed; malicious insiders may experience stress while engaging in rogue behavior; and chronic stress has deleterious effects on mental and physical health. If stress could be detected unobtrusively, without requiring special equipment, remedies to these situations could be undertaken. In this study a common computer keyboard and everyday typing are the primary instruments for detecting stress. Aim. The goal of this dissertation is to detect stress via keystroke dynamics – the analysis of a user’s typing rhythms – and to detect the changes to those rhythms concomitant with stress. Additionally, we pinpoint markers for stress (e.g., a 10% increase in typing speed), analogous to the antigens used as markers for blood type. We seek markers that are universal across all typists, as well as markers that apply only to groups or clusters of typists, or even only to individual typists. Data. Five types of data were collected from 116 subjects: (1) demographic data, which can reveal factors (e.g., gender) that influence subjects’ reactions to stress; (2) psychological data, which capture a subject’s general susceptibility to stress and anxiety, as well as his/her current stress state; (3) physiological data (e.g., heart-rate variability and blood pressure) that permit an objective and independent assessment of a subject’s stress level; (4) self-report data, consisting of subjective self-reports regarding the subject’s stress, anxiety, and workload levels; and (5) typing data from subjects, in both neutral and stressed states, measured in terms of keystroke timings – hold and latency times – and typographical errors. Differences in typing rhythms between neutral and stressed states were examined to seek specific markers for stress. Method. An ABA, single-subject design was used, in which subjects act as their own controls. Each subject provided 80 typing samples in each of three conditions: (A) baseline/neutral, (B) induced stress, and (A) post-stress return/recovery-to-baseline. Physiological measures were analyzed to ascertain the subject’s stress level when providing each sample. Typing data were analyzed, using a variety of statistical and machine learning techniques, to elucidate markers of stress. Clustering techniques (e.g., K-means) were also employed to detect groups of users whose responses to stress are similar. Results. Our stressor paradigm was effective for all 116 subjects, as confirmed through analysis of physiological and self-report data. We were able to identify markers for stress within each subject; i.e., we can discriminate between neutral and stressed typing when examining any subject individually. However, despite our best attempts, and the use of state-of-the-art machine learning techniques, we were not able to identify universal markers for stress, across subjects, nor were we able to identify clusters of subjects whose stress responses were similar. Subjects’ stress responses, in typing data, appear to be highly individualized. Consequently, effective deployment in a realworld environment may require an approach similar to that taken in personalized medicine.

Emotional intelligence has a prominent role in education, health care, and day to day interaction. With the increasing use of computer technology, computers are interacting with more and more individuals. This interaction provides an opportunity to increase knowledge about human emotion for human consumption, well-being, and improved computer adaptation. This thesis explores the efficacy of using up to four different sensors in three domains for computational affect detection. We first consider computer-based education, where a collection of four sensors is used to detect student emotions relevant to learning, such as frustration, confidence, excitement and interest while students use a computer geometry tutor. The best classier of each emotion in terms of accuracy ranges from 78% to 87.5%. We then use voice data collected in a clinical setting to differentiate both gender and culture of the speaker. We produce classifiers with accuracies between 84% and 94% for gender, and between 58% and 70% for American vs. Asian culture, and we find that classifiers for distinguishing between four cultures do not perform better than chance. Finally, we use video and audio in a health care education scenario to detect students' emotions during a clinical simulation evaluation. The video data provides classifiers with accuracies between 63% and 88% for the emotions of confident, anxious, frustrated, excited, and interested. We find the audio data to be too complex to single out the voice source of the student by automatic means. In total, this work is a step forward in the automatic computational detection of affect in realistic settings.

abstract: Traditional usability methods in Human-Computer Interaction (HCI) have been extensively used to understand the usability of products. Measurements of user experience (UX) in traditional HCI studies mostly rely on task performance and observable user interactions with the product or services, such as usability tests, contextual inquiry, and subjective self-report data, including questionnaires, interviews, and usability tests. However, these studies fail to directly reflect a user’s psychological involvement and further fail to explain the cognitive processing and the related emotional arousal. Thus, capturing how users think and feel when they are using a product remains a vital challenge of user experience evaluation studies. Conversely, recent research has revealed that sensor-based affect detection technologies, such as eye tracking, electroencephalography (EEG), galvanic skin response (GSR), and facial expression analysis, effectively capture affective states and physiological responses. These methods are efficient indicators of cognitive involvement and emotional arousal and constitute effective strategies for a comprehensive measurement of UX. The literature review shows that the impacts of sensor-based affect detection systems to the UX can be categorized in two groups: (1) confirmatory to validate the results obtained from the traditional usability methods in UX evaluations; and (2) complementary to enhance the findings or provide more precise and valid evidence. Both provided comprehensive findings to uncover the issues related to mental and physiological pathways to enhance the design of product and services. Therefore, this dissertation claims that it can be efficient to integrate sensor-based affect detection technologies to solve the current gaps or weaknesses of traditional usability methods. The dissertation revealed that the multi-sensor-based UX evaluation approach through biometrics tools and software corroborated user experience identified by traditional UX methods during an online purchasing task. The use these systems enhanced the findings and provided more precise and valid evidence to predict the consumer purchasing preferences. Thus, their impact was “complementary” on overall UX evaluation. The dissertation also provided information of the unique contributions of each tool and recommended some ways user experience researchers can combine both sensor-based and traditional UX approaches to explain consumer purchasing preferences. / Dissertation/Thesis / Doctoral Dissertation Human Systems Engineering 2018