{"id":{"repo_id":"calpoly","oai_identifier":"oai:digitalcommons.calpoly.edu:theses-5032"},"canonical_url":"https://search.ndltd.org/etd/calpoly/oai:digitalcommons.calpoly.edu:theses-5032","repository":{"repo_id":"calpoly","name":"Cal Poly","base_url":"https://digitalcommons.calpoly.edu/do/oai/"},"display":{"title":"Deep Learning for Affect Recognition: A Comparative Study of Physiological Sensor-Based and Facial Image-Based Approaches","abstract":"This thesis investigates deep learning approaches for affect recognition using wearable physiological signals and facial image data. The sensor-based component evaluates stress and affect recognition on the WESAD dataset using wrist-based physiological windows and examines multiple temporal modeling strategies, including convolutional, recurrent, hybrid CNN-LSTM, attention-based, ensemble, and time-frequency approaches. The image-based component evaluates hard-label facial expression recognition on AffectNet+ using pre-trained ResNet-50, EfficientNet-B3, and ConvNeXt-Tiny architectures across Easy, Challenging, and Difficult subsets representing different levels of expression ambiguity. Experimental results show that the proposed Multi-Branch Attention CNN-BiLSTM (MBA-CNN-BiLSTM) model achieves the strongest wearable stress- and affect-recognition performance among the evaluated WESAD models under the random sample-level evaluation setting, reaching 99.91% binary accuracy and 98.10% three-class accuracy while reducing error rate by 5.67 and 11.03 percentage points, respectively, relative to the reproduced 1D CNN baseline. On AffectNet+, the best mean accuracies were 88.82% on the Easy subset using EfficientNet-B3 and 59.48% and 41.83% on the Challenging and Difficult subsets using ConvNeXt-Tiny, corresponding to error-rate reductions of 2.96, 7.86, and 7.49 percentage points relative to the published ResNet-50 Hard-FER reference targets. Together, these results demonstrate that physiological and visual modalities capture complementary aspects of affective state, motivating future multimodal affect-recognition systems that integrate internal physiological responses with external expressive cues.","abstract_html":"This thesis investigates deep learning approaches for affect recognition using wearable physiological signals and facial image data. The sensor-based component evaluates stress and affect recognition on the WESAD dataset using wrist-based physiological windows and examines multiple temporal modeling strategies, including convolutional, recurrent, hybrid CNN-LSTM, attention-based, ensemble, and time-frequency approaches. The image-based component evaluates hard-label facial expression recognition on AffectNet+ using pre-trained ResNet-50, EfficientNet-B3, and ConvNeXt-Tiny architectures across Easy, Challenging, and Difficult subsets representing different levels of expression ambiguity. Experimental results show that the proposed Multi-Branch Attention CNN-BiLSTM (MBA-CNN-BiLSTM) model achieves the strongest wearable stress- and affect-recognition performance among the evaluated WESAD models under the random sample-level evaluation setting, reaching 99.91% binary accuracy and 98.10% three-class accuracy while reducing error rate by 5.67 and 11.03 percentage points, respectively, relative to the reproduced 1D CNN baseline. On AffectNet+, the best mean accuracies were 88.82% on the Easy subset using EfficientNet-B3 and 59.48% and 41.83% on the Challenging and Difficult subsets using ConvNeXt-Tiny, corresponding to error-rate reductions of 2.96, 7.86, and 7.49 percentage points relative to the published ResNet-50 Hard-FER reference targets. Together, these results demonstrate that physiological and visual modalities capture complementary aspects of affective state, motivating future multimodal affect-recognition systems that integrate internal physiological responses with external expressive cues.","abstract_has_math":false,"creators":["Panchal, Ravi"],"institution":null,"degree_name":"MS in Electrical Engineering","degree_level":null,"degree_discipline":"Electrical Engineering","degree_department":null,"school":null,"contributors":["Ria Kanjilal","Electrical Engineering","College of Engineering"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T07:12:58Z","subjects":["Signal Processing"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/3341","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.calpoly.edu/theses/3341","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.calpoly.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.calpoly.edu%3Atheses-5032","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Ria Kanjilal","Electrical Engineering","College of Engineering"]},{"key":"dc:creator","label":"Author","values":["Panchal, Ravi"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2026-06-12T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2026-06-01T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical Engineering"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Electrical Engineering"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Signal Processing"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/3341"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["This thesis investigates deep learning approaches for affect recognition using wearable physiological signals and facial image data. The sensor-based component evaluates stress and affect recognition on the WESAD dataset using wrist-based physiological windows and examines multiple temporal modeling strategies, including convolutional, recurrent, hybrid CNN-LSTM, attention-based, ensemble, and time-frequency approaches. The image-based component evaluates hard-label facial expression recognition on AffectNet+ using pre-trained ResNet-50, EfficientNet-B3, and ConvNeXt-Tiny architectures across Easy, Challenging, and Difficult subsets representing different levels of expression ambiguity. Experimental results show that the proposed Multi-Branch Attention CNN-BiLSTM (MBA-CNN-BiLSTM) model achieves the strongest wearable stress- and affect-recognition performance among the evaluated WESAD models under the random sample-level evaluation setting, reaching 99.91% binary accuracy and 98.10% three-class accuracy while reducing error rate by 5.67 and 11.03 percentage points, respectively, relative to the reproduced 1D CNN baseline. On AffectNet+, the best mean accuracies were 88.82% on the Easy subset using EfficientNet-B3 and 59.48% and 41.83% on the Challenging and Difficult subsets using ConvNeXt-Tiny, corresponding to error-rate reductions of 2.96, 7.86, and 7.49 percentage points relative to the published ResNet-50 Hard-FER reference targets. Together, these results demonstrate that physiological and visual modalities capture complementary aspects of affective state, motivating future multimodal affect-recognition systems that integrate internal physiological responses with external expressive cues."]},{"key":"dc:title","label":"Title","values":["Deep Learning for Affect Recognition: A Comparative Study of Physiological Sensor-Based and Facial Image-Based Approaches"]}]}],"canonical_facts":{"dc:contributor":["Ria Kanjilal","Electrical Engineering","College of Engineering"],"dc:creator":["Panchal, Ravi"],"dc:date.available":["2026-06-12T07:00:00Z"],"dc:date.created":["2026-06-01T07:00:00Z"],"dc:description.abstract":["This thesis investigates deep learning approaches for affect recognition using wearable physiological signals and facial image data. The sensor-based component evaluates stress and affect recognition on the WESAD dataset using wrist-based physiological windows and examines multiple temporal modeling strategies, including convolutional, recurrent, hybrid CNN-LSTM, attention-based, ensemble, and time-frequency approaches. The image-based component evaluates hard-label facial expression recognition on AffectNet+ using pre-trained ResNet-50, EfficientNet-B3, and ConvNeXt-Tiny architectures across Easy, Challenging, and Difficult subsets representing different levels of expression ambiguity. Experimental results show that the proposed Multi-Branch Attention CNN-BiLSTM (MBA-CNN-BiLSTM) model achieves the strongest wearable stress- and affect-recognition performance among the evaluated WESAD models under the random sample-level evaluation setting, reaching 99.91% binary accuracy and 98.10% three-class accuracy while reducing error rate by 5.67 and 11.03 percentage points, respectively, relative to the reproduced 1D CNN baseline. On AffectNet+, the best mean accuracies were 88.82% on the Easy subset using EfficientNet-B3 and 59.48% and 41.83% on the Challenging and Difficult subsets using ConvNeXt-Tiny, corresponding to error-rate reductions of 2.96, 7.86, and 7.49 percentage points relative to the published ResNet-50 Hard-FER reference targets. Together, these results demonstrate that physiological and visual modalities capture complementary aspects of affective state, motivating future multimodal affect-recognition systems that integrate internal physiological responses with external expressive cues."],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/3341"],"dc:subject":["Signal Processing"],"dc:title":["Deep Learning for Affect Recognition: A Comparative Study of Physiological Sensor-Based and Facial Image-Based Approaches"],"thesis:degree_discipline":["Electrical Engineering"],"thesis:degree_name":["MS in Electrical Engineering"]},"updated_at":"2026-11-19T07:12:58Z"}