Back to search

Virginia Commonwealth University

BENCHMARKING AND ENHANCING GENERALIZATION IN MULTILINGUAL SPEECH EMOTION RECOGNITION

Abstract

dc:description.abstract

Speech Emotion Recognition (SER) is pivotal in advancing human-computer interaction by enabling machines to understand and respond to human emotions. Despite significant progress with self-supervised learning models, SER systems often struggle with generalization across diverse languages and unseen data distributions, limiting their real-world applicability. This thesis addresses these challenges by first introducing a large-scale benchmark to evaluate the robustness and adaptability of state-of-the-art SER models in both in-domain and out-of-domain settings. The benchmark includes a diverse set of multilingual datasets, emphasizing cross-lingual and out-of-domain evaluations to assess model generalization. Surprisingly, we find that the Whisper model, originally designed for automatic speech recognition, outperforms dedicated self-supervised learning models in cross-lingual SER tasks. Building upon these findings, we propose a novel approach that combines soft labeling and aggressive data augmentation techniques to model temporal emotion shifts in large-scale multilingual speech data. We introduce {\textbf{Eisper}}, a novel training paradigm that leverages the Whisper encoder augmented with Matryoshka Representation Learning to capture hierarchical emotional representations. By aggregating 17 diverse datasets and employing stochastic sample packing, we create a comprehensive dataset encompassing 571 hours of multilingual emotional speech. Extensive experiments demonstrate significant improvements in zero-shot generalization, achieving state-of-the-art performance on out-of-domain datasets and surpassing existing models on the SER Evals benchmark. This thesis contributes to advancing the field of SER by providing valuable insights into model generalization, offering a robust benchmark for future research, and presenting novel methodologies to enhance the generalizability of SER models across diverse languages and data distributions.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
Thesis
Discipline thesis:degree_discipline
Computer Science

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Osman Ismael, Mohamed
  • https://orcid.org/0009-0000-9171-8955
Contributors dc:contributor
  • Tamer Nadeem

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • © The Author

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Virginia Commonwealth University
Base URL
scholarscompass.vcu.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Osman Ismael, Mohamed; https://orcid.org/0009-0000-9171-8955. BENCHMARKING AND ENHANCING GENERALIZATION IN MULTILINGUAL SPEECH EMOTION RECOGNITION. Thesis thesis, https://doi.org/10.25772/H2HS-8726