Back to search

University of South Florida

Context and Interpretability in Affective Computing Applications

Abstract

dc:description

Affective Computing (AC) is a subdomain of AI that primarily deals with recognizing and interpreting human emotions. This field inherently intersects with psychological studies, as the comprehension of human emotions and behaviors necessitates an understanding of their underlying cognitive processes. One such concepts that lends itself from psychology is context. Roughly speaking, context in AC is defined as any meta information (e.g., environment) that can be utilized to describe the interaction between a user and a model to solve a particular application (e.g., emotion recognition). This doctoral dissertation comprises of two distinct yet interconnected components (Part I and II), the first of which seeks to analyze and study the effect of integrating such contexts in designing AC applications. In the first part, we investigate the impact of context across diverse affect-based systems, including those designed for reflective thinking, continuous authentication, and post-traumatic stress disorder (PTSD) analysis. Initially, we examine the effect of task-specific classifiers in a unimodal system for reflective thinking prediction and showcase the performance improvement on inclusion of the tasks as context. Next, we shift our focus to multimodal systems for continuous authentication, where we delve into the effects of multiple emotional responses (context) as an indicator for identifying a particular subject across different sessions. We demonstrate that although context inclusion yields improved performance, its impact is superseded by model's capability to perform continuous authentication in a single-session environment. Finally, we illustrate the influence of clinician-focused sessions as context to classify children videos with PTSD. we further extend our study of context by evaluating the inter-dependencies across context and its contribution towards the classification. Concluding with the release of a new context-integrated dataset to further expand the resources available for research in this domain. The significance of context-aware affect systems is paralleled by the critical role of model interpretability. The aforementioned applications of affect extends to healthcare, biometrics, and education which are considered highly sensitive fields and the interpretation of machine learning model output is of paramount importance due to the potential consequences of erroneous conclusion. To this end, the second part of the dissertation focuses on novel representation learning approaches to interpretable predictions. More specifically, we employ probabilistic likelihood models to learn class-specific distribution to provide exact likelihood values of a data point belonging to a particular class, as opposed to softmax psuedo-probabilities. For instance, our initial work proposes a facial expression recognition (FER) model using normalizing flows to learn latent representations per expression. We showcase that while this approach produces latent representations consistent with model output, it exhibits limitations in real-world classification scenarios. To overcome these constraints, we introduce an innovative loss function that leverages supervised contrastive learning (SCL) to capture class-specific distributions rather than representations. We find that although this novel method enhances classification accuracy, its primary contribution lies in its capacity to detect out-of-distribution data, a crucial yet under-explored application in affective computing. We provide extensive quantitative and qualitative analysis to validate our approach.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy (Ph.D.)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Computer Science and Engineering
Grantor
University of South Florida
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sidhapur Lakshminarayan, Saandeep Aathreya
Contributors dc:contributor
  • Shaun Canavan, Ph.D.
  • Sudeep Sarkar, Ph.D.
  • Paul Rosen, Ph.D.
  • Mauricio Pamplona Segundo, Ph.D.
  • Alison Salloum, Ph.D.
  • Achilleas Kourtellis, Ph.D.

Subjects

dc:subject × 6

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.usf.edu/etd/11007

Chain of custody

source
Harvested from
University of South Florida
Base URL
digitalcommons.usf.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Sidhapur Lakshminarayan, Saandeep Aathreya. Context and Interpretability in Affective Computing Applications. Dissertation thesis, University of South Florida, 2025. https://digitalcommons.usf.edu/etd/11007