Back to search

Virginia Tech

Towards Unified and Generalizable Multimodal Foundation Models

Abstract

dc:description.abstract

Recent advances in multimodal models have reshaped multimodal learning by leveraging large language models (LLMs) as backbones and integrating them with vision encoders or diffusion models. These approaches have achieved strong performance in either multimodal understanding or multimodal generation. However, no existing system offers a unified framework capable of performing both understanding and generation across flexible input-output modality combinations while also generalizing to unseen, real-world tasks. Unification is essential not only for enabling a single model to perform traditional understanding and generation tasks, but also for enabling cross-modal tasks, such as visual storytelling, report generation, and script creation, that neither understanding nor generation models alone can accomplish. By processing and generating inputs and outputs that span multiple modalities, unified models more closely mirror the way humans naturally acquire and construct knowledge. Generalization, in turn, enables models to adapt to novel tasks in open-world environments under human-specified instructions or principles. Together, unification and generalizability constitute two fundamental pillars for advancing toward general-purpose multimodal intelligence. This dissertation advances unified and generalizable multimodal modeling through novel architectures, post-training paradigms, and reinforcement learning algorithms. It addresses two enduring challenges in multimodal foundation models: (1) the lack of modality unification, and (2) limited generalizability in open-world settings. The contributions are fourfold. First, we introduce Modality-Specialized Synergizers (MOSS), a framework that enables interleaved multimodal generation in pretrained models. Second, we propose an efficient unified architecture that bridges vision-language models and diffusion models, providing a novel pathway for joint understanding and generation. Third, we establish multimodal instruction tuning as a new post-training paradigm to improve zero-shot generalization and robustness. Finally, we extend image understanding to the spatiotemporal domain by developing a novel reinforcement learning algorithm that promotes temporal awareness, enabling vision–language models to reason effectively about videos. Extensive experiments across diverse multimodal benchmarks demonstrate that these approaches significantly enhance unification, generalizability, and overall capability. Collectively, this research strengthens the foundations of multimodal AI and outlines a pathway toward universal models that can understand, reason, and generate across modalities in complex open-world environments.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Xu, Zhiyang
Chair dc:contributor.committeechair
  • Huang, Lifu
Committee members dc:contributor.committeemember
  • Bansal, Mohit
  • Reddy, Chandan K.
  • Ramakrishnan, Narendran
  • Wang, Xuan

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:45740

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Xu, Zhiyang. Towards Unified and Generalizable Multimodal Foundation Models. doctoral thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/142446