Abstract
dc:description.abstractRecent advances in multimodal models have reshaped multimodal learning by leveraging large language models (LLMs) as backbones and integrating them with vision encoders or diffusion models. These approaches have achieved strong performance in either multimodal understanding or multimodal generation. However, no existing system offers a unified framework capable of performing both understanding and generation across flexible input-output modality combinations while also generalizing to unseen, real-world tasks. Unification is essential not only for enabling a single model to perform traditional understanding and generation tasks, but also for enabling cross-modal tasks, such as visual storytelling, report generation, and script creation, that neither understanding nor generation models alone can accomplish. By processing and generating inputs and outputs that span multiple modalities, unified models more closely mirror the way humans naturally acquire and construct knowledge. Generalization, in turn, enables models to adapt to novel tasks in open-world environments under human-specified instructions or principles. Together, unification and generalizability constitute two fundamental pillars for advancing toward general-purpose multimodal intelligence. This dissertation advances unified and generalizable multimodal modeling through novel architectures, post-training paradigms, and reinforcement learning algorithms. It addresses two enduring challenges in multimodal foundation models: (1) the lack of modality unification, and (2) limited generalizability in open-world settings. The contributions are fourfold. First, we introduce Modality-Specialized Synergizers (MOSS), a framework that enables interleaved multimodal generation in pretrained models. Second, we propose an efficient unified architecture that bridges vision-language models and diffusion models, providing a novel pathway for joint understanding and generation. Third, we establish multimodal instruction tuning as a new post-training paradigm to improve zero-shot generalization and robustness. Finally, we extend image understanding to the spatiotemporal domain by developing a novel reinforcement learning algorithm that promotes temporal awareness, enabling vision–language models to reason effectively about videos. Extensive experiments across diverse multimodal benchmarks demonstrate that these approaches significantly enhance unification, generalizability, and overall capability. Collectively, this research strengthens the foundations of multimodal AI and outlines a pathway toward universal models that can understand, reason, and generate across modalities in complex open-world environments.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science & Applications
- Department dc:contributor.department
- Computer Science and#38; Applications
- Grantor dc:publisher
- Virginia Tech
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Xu, Zhiyang
- Chair dc:contributor.committeechair
-
- Huang, Lifu
- Committee members dc:contributor.committeemember
-
- Bansal, Mohit
- Reddy, Chandan K.
- Ramakrishnan, Narendran
- Wang, Xuan
Subjects
dc:subject × 3Rights
dc:rights- Statement dc:rights
-
- In Copyright
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
- vt_gsexam:45740