Back to search

University of South Florida

Analysis of Contextual Emotions Using Multimodal Data

Abstract

dc:description

Affective computing builds and evaluates systems that can recognize, interpret, and simulate human emotion. It is an interdisciplinary field, which includes computer science, psychology, and many others. For years, human emotion has been studied in psychology but recently has become a prominent field in computer science. Largely, the field of affective computing has been focused on analyzing static facial expressions to recognize human emotions, without taking bias (e.g. gender, data bias), context, or temporal information into account. Psychology has shown the difficulty of analyzing emotions without incorporating this type of information. In this dissertation, we have proposed new approaches to recognizing emotions by incorporating both contextual and temporal information, as well as approaches to mitigate data bias. More specifically, this dissertation has the following theoretical and application-based contributions: (1) The first work to recognize context using temporal dynamics from facial action units; (2) We recognize multiple self-reported emotions using facial expression-based videos; (3) A new approach to mitigate data bias in facial action units is proposed; and (4) Multimodal, temporal fusion of physiological signals and action units are used for emotion recognition. This dissertation has a wide range of applications in the fields including, but not limited to, medicine, security, and entertainment.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy (Ph.D.)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Computer Science and Engineering
Grantor
University of South Florida
Year dc:date
2021

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hinduja, Saurabh
Contributors dc:contributor
  • Shaun Canavan, Ph.D.
  • Rangachar Kasturi, Ph.D.
  • Jeffrey F. Cohn, Ph.D.
  • Marvin J. Andujar, Ph.D.
  • Pei-Sung Lin, Ph.D.
  • Elizabeth Schotter, Ph.D.

Subjects

dc:subject × 4

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.usf.edu/etd/8793

Chain of custody

source
Harvested from
University of South Florida
Base URL
digitalcommons.usf.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Hinduja, Saurabh. Analysis of Contextual Emotions Using Multimodal Data. Dissertation thesis, University of South Florida, 2021. https://digitalcommons.usf.edu/etd/8793