Back to search

Cal Poly

Deep Learning for Affect Recognition: A Comparative Study of Physiological Sensor-Based and Facial Image-Based Approaches

Abstract

dc:description.abstract

This thesis investigates deep learning approaches for affect recognition using wearable physiological signals and facial image data. The sensor-based component evaluates stress and affect recognition on the WESAD dataset using wrist-based physiological windows and examines multiple temporal modeling strategies, including convolutional, recurrent, hybrid CNN-LSTM, attention-based, ensemble, and time-frequency approaches. The image-based component evaluates hard-label facial expression recognition on AffectNet+ using pre-trained ResNet-50, EfficientNet-B3, and ConvNeXt-Tiny architectures across Easy, Challenging, and Difficult subsets representing different levels of expression ambiguity. Experimental results show that the proposed Multi-Branch Attention CNN-BiLSTM (MBA-CNN-BiLSTM) model achieves the strongest wearable stress- and affect-recognition performance among the evaluated WESAD models under the random sample-level evaluation setting, reaching 99.91% binary accuracy and 98.10% three-class accuracy while reducing error rate by 5.67 and 11.03 percentage points, respectively, relative to the reproduced 1D CNN baseline. On AffectNet+, the best mean accuracies were 88.82% on the Easy subset using EfficientNet-B3 and 59.48% and 41.83% on the Challenging and Difficult subsets using ConvNeXt-Tiny, corresponding to error-rate reductions of 2.96, 7.86, and 7.49 percentage points relative to the published ResNet-50 Hard-FER reference targets. Together, these results demonstrate that physiological and visual modalities capture complementary aspects of affective state, motivating future multimodal affect-recognition systems that integrate internal physiological responses with external expressive cues.

Degree

thesis:*
Name thesis:degree_name
MS in Electrical Engineering
Discipline thesis:degree_discipline
Electrical Engineering

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Panchal, Ravi
Contributors dc:contributor
  • Ria Kanjilal
  • Electrical Engineering
  • College of Engineering

Subjects

dc:subject × 1

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.calpoly.edu/theses/3341

Chain of custody

source
Harvested from
Cal Poly
Base URL
digitalcommons.calpoly.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Panchal, Ravi. Deep Learning for Affect Recognition: A Comparative Study of Physiological Sensor-Based and Facial Image-Based Approaches. https://digitalcommons.calpoly.edu/theses/3341