Back to search

Virginia Tech

Knowledge-Centric Multimodal Intelligence: Understanding, Reasoning, and Generation

Abstract

dc:description.abstract

Generative multimodal foundation models now handle a wide range of tasks spanning understanding, reasoning, and generation. Yet the knowledge they draw on at inference time is bounded by what their finite parameters memorized during training, leading to fabricated content, missed low-frequency details, and incomplete use of available evidence. This dissertation argues that closing the gap between what a model can access and what it actually uses drives capability gains across all three axes. For multimodal understanding, I develop a retrieval-augmented vision-language model that learns to ignore irrelevant retrievals, making knowledge-intensive visual question answering resilient to retriever noise. For reasoning, I first introduce a recursive questioning procedure that surfaces parametric knowledge a single-pass prompt cannot reach. I then address the complementary challenge that knowledge-driven methods alone cannot eliminate reasoning errors, developing self-revision mechanisms that let the model correct its own intermediate outputs in both autoregressive and discrete-diffusion generation paradigms. For multimodal generation, I extend retrieval augmentation into image synthesis, incorporating fine-grained visual evidence that the text prompt alone cannot specify. Across visual question answering, mathematical reasoning, constraint solving, and image generation, these methods outperform approaches that rely on parametric knowledge alone. The broader claim is that more effective utilization of available knowledge drives capability gains beyond what larger models or longer training alone can provide.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Qi, Jingyuan
Chairs dc:contributor.committeechair
  • Huang, Lifu
  • Zhou, Dawei
Committee members dc:contributor.committeemember
  • Lourentzou, Ismini
  • Yao, Danfeng
  • Thomas, Christopher Lee

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Creative Commons Attribution-NonCommercial 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:46876

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Qi, Jingyuan. Knowledge-Centric Multimodal Intelligence: Understanding, Reasoning, and Generation. doctoral thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143546