Back to search

National Technical University of Athens (NTUA)

Multimodal representation learning with application in sentiment analysis

Abstract

dc:description

Multimodal learning is the computational paradigm focused on learning and integrating heterogeneous and interconnected information sources, i.e., modalities. At the core of any multimodal learning system lies multimodal representation learning, which aims to learn representations that both reflect unimodal characteristics and effectively capture complex cross-modal interactions. This thesis examines multimodal representation learning within the context of Multimodal Sentiment Analysis (MSA), a domain that requires effectively combining linguistic, acoustic, and visual information to uncover the complex nature of human sentiment. MSA's significance lies in its role as an exemplary microcosm of broader multimodal learning challenges. By addressing issues ranging from the integration of heterogeneous modalities (from natural signals to symbolic language) to suboptimal network optimization and modality information imbalance, MSA provides a valuable case study through which to examine multimodal representation learning. In the first part of this thesis, we focus on developing novel feature and latent space transformations which are inspired by regularization techniques and aim at increasing the scenarios seen by the model during training. These methods can be applied on top of existing multimodal architectures. We address a wide range of challenges from feature space augmentation, to modality imbalance and generic regularization by generating new multimodal mixtures. Our work covers a wide and diverse range of computational methodologies: a permutation-based feature augmentation method which increases data diversity by "injecting noise" from the underlying sample distribution (SeqAug); a latent space masking strategy that can be applied as regularization and can be further shifted to directly tackle modality imbalance (M3); and a versatile mixing-based regularization approach suitable for multimodal networks, which generates latent space mixtures which enrich the scenarios seen by the model during training (PowMix). These algorithms significantly extend the arsenal of regularization-based feature and latent space transformations for multimodal architectures, and demonstrate that existing networks are often underoptimized and can benefit substantially from proper regularization. In the second part, we explore the idea of deep multimodal fusion and introduce two novel architectural designs, one encoder-based and one encoder-decoder based. The core principle of the deep fusion paradigm is that information integration occurs across multiple network layers, with fused information propagating through the network. The first approach leverages the hierarchical structure of language to fuse information at word, sentence, and sentiment representation levels, maintaining separate computational streams for modality-specific (unimodal encoders) and fused representations (DHF). We empirically verify that this scheme is a powerful fusion paradigm and can be further extended to transformer-based approaches (DeepSER). The second approach revisits the concept of deep fusion from an encoder-decoder architectural perspective. We implement a multimodal language model augmented with novel cross-attention blocks and learnable fusion tokens (DeepMLF). This design naturally allows for depth, scalability and controlled interactions in the fusion process. Furthermore, this design allows for explicitly handling the capacity allocated to multimodal information. These approaches demonstrate that macroscopic architectural considerations such as deep fusion are more crucial than the microscopic core fusion operations.

Degree

thesis:*
Grantor dc:publisher
National Technical University of Athens (NTUA)
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Georgiou, Efthymios
  • Γεωργίου, Ευθύμιος

Subjects

dc:subject × 18

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/58806

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Georgiou, Efthymios; Γεωργίου, Ευθύμιος. Multimodal representation learning with application in sentiment analysis. National Technical University of Athens (NTUA), 2025. http://hdl.handle.net/10442/hedi/58806