Back to search

Georgia Institute of Technology

Extending Multimodal Large Language Models Beyond Single Modalities

Abstract

dc:description.abstract

Multimodal large language models (MLLMs) aim to unify diverse sensory inputs—such as text, vision, and audio—within a single reasoning framework, enabling more comprehensive and human-like understanding. However, as the number of supported modalities increases, generalist MLLMs often underperform compared to specialist models due to cross-modal interference and imbalanced data quality across modalities. This thesis systematically investigates these challenges and proposes solutions through a series of controlled experiments. The proposed architecture integrates CLIP and BEATs encoders for vision and audio, respectively, with a Vicuna-v1.5 large language model, connected via lightweight two layer MLP projectors. A two-stage training protocol—modality alignment pretraining followed by supervised instruction tuning—is employed using carefully curated datasets for each modality. Experimental results demonstrate that true joint training, especially when paired with high-quality, targeted datasets, significantly mitigates cross-modal interference and enhances performance across both vision and audio tasks. Notably, simple projection layers outperform more complex alternatives like Q-Former in this setup, and instruction-following behaviors learned in well-resourced modalities readily transfer to data-scarce ones under joint training. The findings highlight the importance of training strategy, data quality, and architectural simplicity in building scalable, generalist MLLMs. Future work will extend these strategies to additional modalities and explore adaptive curriculum learning for continual multimodal integration.

Degree

thesis:*
Level thesis:degree_level
Masters
Department dc:contributor.department
Computer Science
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Vasu, Manushree
Advisor dc:contributor.advisor
  • Shi, Humphrey
Committee members dc:contributor.committeemember
  • Hoffman, Judy
  • Kira, Zsolt

Subjects

dc:subject × 2

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/77954

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Vasu, Manushree. Extending Multimodal Large Language Models Beyond Single Modalities. Masters thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/77954