National Technical University of Athens (NTUA)
Multimodal representation learning with application in sentiment analysis
Abstract
dc:descriptionMultimodal learning is the computational paradigm focused on learning and integrating heterogeneous and interconnected information sources, i.e., modalities. At the core of any multimodal learning system lies multimodal representation learning, which aims to learn representations that both reflect unimodal characteristics and effectively capture complex cross-modal interactions. This thesis examines multimodal representation learning within the context of Multimodal Sentiment Analysis (MSA), a domain that requires effectively combining linguistic, acoustic, and visual information to uncover the complex nature of human sentiment. MSA's significance lies in its role as an exemplary microcosm of broader multimodal learning challenges. By addressing issues ranging from the integration of heterogeneous modalities (from natural signals to symbolic language) to suboptimal network optimization and modality information imbalance, MSA provides a valuable case study through which to examine multimodal representation learning. In the first part of this thesis, we focus on developing novel feature and latent space transformations which are inspired by regularization techniques and aim at increasing the scenarios seen by the model during training. These methods can be applied on top of existing multimodal architectures. We address a wide range of challenges from feature space augmentation, to modality imbalance and generic regularization by generating new multimodal mixtures. Our work covers a wide and diverse range of computational methodologies: a permutation-based feature augmentation method which increases data diversity by "injecting noise" from the underlying sample distribution (SeqAug); a latent space masking strategy that can be applied as regularization and can be further shifted to directly tackle modality imbalance (M3); and a versatile mixing-based regularization approach suitable for multimodal networks, which generates latent space mixtures which enrich the scenarios seen by the model during training (PowMix). These algorithms significantly extend the arsenal of regularization-based feature and latent space transformations for multimodal architectures, and demonstrate that existing networks are often underoptimized and can benefit substantially from proper regularization. In the second part, we explore the idea of deep multimodal fusion and introduce two novel architectural designs, one encoder-based and one encoder-decoder based. The core principle of the deep fusion paradigm is that information integration occurs across multiple network layers, with fused information propagating through the network. The first approach leverages the hierarchical structure of language to fuse information at word, sentence, and sentiment representation levels, maintaining separate computational streams for modality-specific (unimodal encoders) and fused representations (DHF). We empirically verify that this scheme is a powerful fusion paradigm and can be further extended to transformer-based approaches (DeepSER). The second approach revisits the concept of deep fusion from an encoder-decoder architectural perspective. We implement a multimodal language model augmented with novel cross-attention blocks and learnable fusion tokens (DeepMLF). This design naturally allows for depth, scalability and controlled interactions in the fusion process. Furthermore, this design allows for explicitly handling the capacity allocated to multimodal information. These approaches demonstrate that macroscopic architectural considerations such as deep fusion are more crucial than the microscopic core fusion operations.
Degree
thesis:*- Grantor dc:publisher
- National Technical University of Athens (NTUA)
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Georgiou, Efthymios
- Γεωργίου, Ευθύμιος
Subjects
dc:subject × 18- Πολυτροπική μάθηση
- Βαθιά μάθηση
- Τεχνικές ομαλοποίησης
- Σύμμειξη πληροφορίας
- Πολυτροπική Αναγνώριση Συναισθήματος
- Multimodal learning
- Deep learning
- Regularization
- Multimodal fusion
- Multimodal sentiment analysis
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Τεχνητή νοημοσύνη
- Επιστήμη ηλεκτρονικών υπολογιστών
- Computer and Information Sciences
- Natural Sciences
- Artificial Intelligence
- Computer Science
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/58806