Back to search

Boston University

Grounding in multimodal foundation models: evaluating and improving modality-specific reasoning and generation

Abstract

dc:description.abstract

Multimodal foundation models aim to integrate information across vision, audio, and language, yet current systems often fail to ground their outputs in the modality that actually contains the relevant evidence. Instead, they tend to rely on dominant priors, particularly textual cues or vision input, even when the task requires reasoning from another modality. This insufficient grounding leads to systematic errors in both multimodal reasoning models and generative systems conditioned on text. This thesis studies the problem of weak modality grounding through two complementary investigations. First, we analyze multimodal large language models (MLLMs) that process video, audio, and text. Using a benchmark designed with controlled modality conflicts and misleading textual context, we show that current models struggle to prioritize the correct modality when answering modality-specific questions. We further introduce an alignment-aware tuning strategy that explicitly trains models to rely on the queried modality and ignore irrelevant signals, improving multimodal grounding and robustness. Second, we examine grounding failures in text-to-image generation, where models often fail to correctly represent objects in specified physical states, particularly cases involving absence or emptiness. We propose a fully automated synthetic data pipeline that generates, verifies, and recaptions training examples focused on object-state grounding. Fine-tuning generative models on this curated dataset improves prompt-image alignment on both existing benchmarks and a newly constructed object-state evaluation set. Together, these studies demonstrate that many multimodal failures arise from weak grounding between instructions and the modalities that should provide supporting evidence. By introducing targeted evaluations and training strategies that reinforce modality-specific grounding, this thesis provides practical methods for improving the reliability of multimodal foundation models.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chakka, Venkata Sai Krishna Chaitanya
Advisor dc:contributor.advisor
  • Ghadiyaram, Deepti
ORCID dc:identifier.orcid

Subjects

dc:subject × 7

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53566

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Chakka, Venkata Sai Krishna Chaitanya. Grounding in multimodal foundation models: evaluating and improving modality-specific reasoning and generation. masters thesis, Boston University, 2026. https://hdl.handle.net/2144/53566