Boston University
Grounding in multimodal foundation models: evaluating and improving modality-specific reasoning and generation
Abstract
dc:description.abstractMultimodal foundation models aim to integrate information across vision, audio, and language, yet current systems often fail to ground their outputs in the modality that actually contains the relevant evidence. Instead, they tend to rely on dominant priors, particularly textual cues or vision input, even when the task requires reasoning from another modality. This insufficient grounding leads to systematic errors in both multimodal reasoning models and generative systems conditioned on text. This thesis studies the problem of weak modality grounding through two complementary investigations. First, we analyze multimodal large language models (MLLMs) that process video, audio, and text. Using a benchmark designed with controlled modality conflicts and misleading textual context, we show that current models struggle to prioritize the correct modality when answering modality-specific questions. We further introduce an alignment-aware tuning strategy that explicitly trains models to rely on the queried modality and ignore irrelevant signals, improving multimodal grounding and robustness. Second, we examine grounding failures in text-to-image generation, where models often fail to correctly represent objects in specified physical states, particularly cases involving absence or emptiness. We propose a fully automated synthetic data pipeline that generates, verifies, and recaptions training examples focused on object-state grounding. Fine-tuning generative models on this curated dataset improves prompt-image alignment on both existing benchmarks and a newly constructed object-state evaluation set. Together, these studies demonstrate that many multimodal failures arise from weak grounding between instructions and the modalities that should provide supporting evidence. By introducing targeted evaluations and training strategies that reinforce modality-specific grounding, this thesis provides practical methods for improving the reliability of multimodal foundation models.
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Chakka, Venkata Sai Krishna Chaitanya
- Advisor dc:contributor.advisor
-
- Ghadiyaram, Deepti
- ORCID dc:identifier.orcid
Subjects
dc:subject × 7Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53566