{"id":{"repo_id":"usf","oai_identifier":"oai:digitalcommons.usf.edu:etd-12310"},"canonical_url":"https://search.ndltd.org/etd/usf/oai:digitalcommons.usf.edu:etd-12310","repository":{"repo_id":"usf","name":"University of South Florida","base_url":"https://digitalcommons.usf.edu/do/oai/"},"display":{"title":"Context and Interpretability in Affective Computing Applications","abstract":"Affective Computing (AC) is a subdomain of AI that primarily deals with recognizing and interpreting human emotions. This field inherently intersects with psychological studies, as the comprehension of human emotions and behaviors necessitates an understanding of their underlying cognitive processes. One such concepts that lends itself from psychology is context. Roughly speaking, context in AC is defined as any meta information (e.g., environment) that can be utilized to describe the interaction between a user and a model to solve a particular application (e.g., emotion recognition). This doctoral dissertation comprises of two distinct yet interconnected components (Part I and II), the first of which seeks to analyze and study the effect of integrating such contexts in designing AC applications. In the first part, we investigate the impact of context across diverse affect-based systems, including those designed for reflective thinking, continuous authentication, and post-traumatic stress disorder (PTSD) analysis. Initially, we examine the effect of task-specific classifiers in a unimodal system for reflective thinking prediction and showcase the performance improvement on inclusion of the tasks as context. Next, we shift our focus to multimodal systems for continuous authentication, where we delve into the effects of multiple emotional responses (context) as an indicator for identifying a particular subject across different sessions. We demonstrate that although context inclusion yields improved performance, its impact is superseded by model's capability to perform continuous authentication in a single-session environment. Finally, we illustrate the influence of clinician-focused sessions as context to classify children videos with PTSD. we further extend our study of context by evaluating the inter-dependencies across context and its contribution towards the classification. Concluding with the release of a new context-integrated dataset to further expand the resources available for research in this domain. The significance of context-aware affect systems is paralleled by the critical role of model interpretability. The aforementioned applications of affect extends to healthcare, biometrics, and education which are considered highly sensitive fields and the interpretation of machine learning model output is of paramount importance due to the potential consequences of erroneous conclusion. To this end, the second part of the dissertation focuses on novel representation learning approaches to interpretable predictions. More specifically, we employ probabilistic likelihood models to learn class-specific distribution to provide exact likelihood values of a data point belonging to a particular class, as opposed to softmax psuedo-probabilities. For instance, our initial work proposes a facial expression recognition (FER) model using normalizing flows to learn latent representations per expression. We showcase that while this approach produces latent representations consistent with model output, it exhibits limitations in real-world classification scenarios. To overcome these constraints, we introduce an innovative loss function that leverages supervised contrastive learning (SCL) to capture class-specific distributions rather than representations. We find that although this novel method enhances classification accuracy, its primary contribution lies in its capacity to detect out-of-distribution data, a crucial yet under-explored application in affective computing. We provide extensive quantitative and qualitative analysis to validate our approach.","abstract_html":"Affective Computing (AC) is a subdomain of AI that primarily deals with recognizing and interpreting human emotions. This field inherently intersects with psychological studies, as the comprehension of human emotions and behaviors necessitates an understanding of their underlying cognitive processes. One such concepts that lends itself from psychology is context. Roughly speaking, context in AC is defined as any meta information (e.g., environment) that can be utilized to describe the interaction between a user and a model to solve a particular application (e.g., emotion recognition). This doctoral dissertation comprises of two distinct yet interconnected components (Part I and II), the first of which seeks to analyze and study the effect of integrating such contexts in designing AC applications. In the first part, we investigate the impact of context across diverse affect-based systems, including those designed for reflective thinking, continuous authentication, and post-traumatic stress disorder (PTSD) analysis. Initially, we examine the effect of task-specific classifiers in a unimodal system for reflective thinking prediction and showcase the performance improvement on inclusion of the tasks as context. Next, we shift our focus to multimodal systems for continuous authentication, where we delve into the effects of multiple emotional responses (context) as an indicator for identifying a particular subject across different sessions. We demonstrate that although context inclusion yields improved performance, its impact is superseded by model&#x27;s capability to perform continuous authentication in a single-session environment. Finally, we illustrate the influence of clinician-focused sessions as context to classify children videos with PTSD. we further extend our study of context by evaluating the inter-dependencies across context and its contribution towards the classification. Concluding with the release of a new context-integrated dataset to further expand the resources available for research in this domain. The significance of context-aware affect systems is paralleled by the critical role of model interpretability. The aforementioned applications of affect extends to healthcare, biometrics, and education which are considered highly sensitive fields and the interpretation of machine learning model output is of paramount importance due to the potential consequences of erroneous conclusion. To this end, the second part of the dissertation focuses on novel representation learning approaches to interpretable predictions. More specifically, we employ probabilistic likelihood models to learn class-specific distribution to provide exact likelihood values of a data point belonging to a particular class, as opposed to softmax psuedo-probabilities. For instance, our initial work proposes a facial expression recognition (FER) model using normalizing flows to learn latent representations per expression. We showcase that while this approach produces latent representations consistent with model output, it exhibits limitations in real-world classification scenarios. To overcome these constraints, we introduce an innovative loss function that leverages supervised contrastive learning (SCL) to capture class-specific distributions rather than representations. We find that although this novel method enhances classification accuracy, its primary contribution lies in its capacity to detect out-of-distribution data, a crucial yet under-explored application in affective computing. We provide extensive quantitative and qualitative analysis to validate our approach.","abstract_has_math":false,"creators":["Sidhapur Lakshminarayan, Saandeep Aathreya"],"institution":"University of South Florida","degree_name":"Doctor of Philosophy (Ph.D.)","degree_level":"Dissertation","degree_discipline":"Computer Science and Engineering","degree_department":null,"school":null,"contributors":["Shaun Canavan, Ph.D.","Sudeep Sarkar, Ph.D.","Paul Rosen, Ph.D.","Mauricio Pamplona Segundo, Ph.D.","Alison Salloum, Ph.D.","Achilleas Kourtellis, Ph.D."],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-03-26T07:00:00Z","date_published":"2025-03-26T07:00:00Z","updated_at":"2026-11-19T08:49:40Z","subjects":["Facial Expression Recognition","Multimodal","Normalizing Flows","Out-of-Distribution","Representation Learning","Computer Sciences"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.usf.edu/etd/11007","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.usf.edu/etd/11007","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.usf.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.usf.edu%3Aetd-12310","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Shaun Canavan, Ph.D.","Sudeep Sarkar, Ph.D.","Paul Rosen, Ph.D.","Mauricio Pamplona Segundo, Ph.D.","Alison Salloum, Ph.D.","Achilleas Kourtellis, Ph.D."]},{"key":"dc:creator","label":"Author","values":["Sidhapur Lakshminarayan, Saandeep Aathreya"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025-03-26T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science and Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation","Ph.D."]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy (Ph.D.)"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of South Florida"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Facial Expression Recognition","Multimodal","Normalizing Flows","Out-of-Distribution","Representation Learning","Computer Sciences"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.usf.edu/etd/11007"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Affective Computing (AC) is a subdomain of AI that primarily deals with recognizing and interpreting human emotions. This field inherently intersects with psychological studies, as the comprehension of human emotions and behaviors necessitates an understanding of their underlying cognitive processes. One such concepts that lends itself from psychology is context. Roughly speaking, context in AC is defined as any meta information (e.g., environment) that can be utilized to describe the interaction between a user and a model to solve a particular application (e.g., emotion recognition). This doctoral dissertation comprises of two distinct yet interconnected components (Part I and II), the first of which seeks to analyze and study the effect of integrating such contexts in designing AC applications. In the first part, we investigate the impact of context across diverse affect-based systems, including those designed for reflective thinking, continuous authentication, and post-traumatic stress disorder (PTSD) analysis. Initially, we examine the effect of task-specific classifiers in a unimodal system for reflective thinking prediction and showcase the performance improvement on inclusion of the tasks as context. Next, we shift our focus to multimodal systems for continuous authentication, where we delve into the effects of multiple emotional responses (context) as an indicator for identifying a particular subject across different sessions. We demonstrate that although context inclusion yields improved performance, its impact is superseded by model's capability to perform continuous authentication in a single-session environment. Finally, we illustrate the influence of clinician-focused sessions as context to classify children videos with PTSD. we further extend our study of context by evaluating the inter-dependencies across context and its contribution towards the classification. Concluding with the release of a new context-integrated dataset to further expand the resources available for research in this domain. The significance of context-aware affect systems is paralleled by the critical role of model interpretability. The aforementioned applications of affect extends to healthcare, biometrics, and education which are considered highly sensitive fields and the interpretation of machine learning model output is of paramount importance due to the potential consequences of erroneous conclusion. To this end, the second part of the dissertation focuses on novel representation learning approaches to interpretable predictions. More specifically, we employ probabilistic likelihood models to learn class-specific distribution to provide exact likelihood values of a data point belonging to a particular class, as opposed to softmax psuedo-probabilities. For instance, our initial work proposes a facial expression recognition (FER) model using normalizing flows to learn latent representations per expression. We showcase that while this approach produces latent representations consistent with model output, it exhibits limitations in real-world classification scenarios. To overcome these constraints, we introduce an innovative loss function that leverages supervised contrastive learning (SCL) to capture class-specific distributions rather than representations. We find that although this novel method enhances classification accuracy, its primary contribution lies in its capacity to detect out-of-distribution data, a crucial yet under-explored application in affective computing. We provide extensive quantitative and qualitative analysis to validate our approach."]},{"key":"dc:title","label":"Title","values":["Context and Interpretability in Affective Computing Applications"]}]}],"canonical_facts":{"dc:contributor":["Shaun Canavan, Ph.D.","Sudeep Sarkar, Ph.D.","Paul Rosen, Ph.D.","Mauricio Pamplona Segundo, Ph.D.","Alison Salloum, Ph.D.","Achilleas Kourtellis, Ph.D."],"dc:creator":["Sidhapur Lakshminarayan, Saandeep Aathreya"],"dc:date":["2025-03-26T07:00:00Z"],"dc:description":["Affective Computing (AC) is a subdomain of AI that primarily deals with recognizing and interpreting human emotions. This field inherently intersects with psychological studies, as the comprehension of human emotions and behaviors necessitates an understanding of their underlying cognitive processes. One such concepts that lends itself from psychology is context. Roughly speaking, context in AC is defined as any meta information (e.g., environment) that can be utilized to describe the interaction between a user and a model to solve a particular application (e.g., emotion recognition). This doctoral dissertation comprises of two distinct yet interconnected components (Part I and II), the first of which seeks to analyze and study the effect of integrating such contexts in designing AC applications. In the first part, we investigate the impact of context across diverse affect-based systems, including those designed for reflective thinking, continuous authentication, and post-traumatic stress disorder (PTSD) analysis. Initially, we examine the effect of task-specific classifiers in a unimodal system for reflective thinking prediction and showcase the performance improvement on inclusion of the tasks as context. Next, we shift our focus to multimodal systems for continuous authentication, where we delve into the effects of multiple emotional responses (context) as an indicator for identifying a particular subject across different sessions. We demonstrate that although context inclusion yields improved performance, its impact is superseded by model's capability to perform continuous authentication in a single-session environment. Finally, we illustrate the influence of clinician-focused sessions as context to classify children videos with PTSD. we further extend our study of context by evaluating the inter-dependencies across context and its contribution towards the classification. Concluding with the release of a new context-integrated dataset to further expand the resources available for research in this domain. The significance of context-aware affect systems is paralleled by the critical role of model interpretability. The aforementioned applications of affect extends to healthcare, biometrics, and education which are considered highly sensitive fields and the interpretation of machine learning model output is of paramount importance due to the potential consequences of erroneous conclusion. To this end, the second part of the dissertation focuses on novel representation learning approaches to interpretable predictions. More specifically, we employ probabilistic likelihood models to learn class-specific distribution to provide exact likelihood values of a data point belonging to a particular class, as opposed to softmax psuedo-probabilities. For instance, our initial work proposes a facial expression recognition (FER) model using normalizing flows to learn latent representations per expression. We showcase that while this approach produces latent representations consistent with model output, it exhibits limitations in real-world classification scenarios. To overcome these constraints, we introduce an innovative loss function that leverages supervised contrastive learning (SCL) to capture class-specific distributions rather than representations. We find that although this novel method enhances classification accuracy, its primary contribution lies in its capacity to detect out-of-distribution data, a crucial yet under-explored application in affective computing. We provide extensive quantitative and qualitative analysis to validate our approach."],"dc:identifier":["https://digitalcommons.usf.edu/etd/11007"],"dc:subject":["Facial Expression Recognition","Multimodal","Normalizing Flows","Out-of-Distribution","Representation Learning","Computer Sciences"],"dc:title":["Context and Interpretability in Affective Computing Applications"],"thesis:degree_discipline":["Computer Science and Engineering"],"thesis:degree_level":["Dissertation","Ph.D."],"thesis:degree_name":["Doctor of Philosophy (Ph.D.)"],"thesis:institution_name":["University of South Florida"]},"updated_at":"2026-11-19T08:49:40Z"}