Back to search

University of Alabama Birmingham

Efficient Multimodal Foundation Model Tuning For Hallucination Mitigation

Abstract

dc:description.abstract

Over the past few years, multimodal foundation models have achieved remarkable progress in perception and understanding. However, two challenges limit their reliability: (1) dependence on offline training, which in most real-world settings requires large volumes of labeled data and, as a result, hinders the model’s ability to adapt to new data or domains; (2) weak cross-modal grounding, which often leads to hallucinated content generation, producing descriptions that are linguistically fluent but inconsistent with the input visual evidence. This dissertation frames hallucination mitigation as an outcome of transitioning from fixed learning (static, offline fine-tuning) to adaptive, feedback-driven lifelong learning. By incorporating test-time adaptation, it seeks to enable multimodal foundation models to evolve through interaction with real data. In this process, an evaluator assesses the overall quality of generated outputs, providing informative feedback that allows the model to adapt its internal representations and progressively reduce hallucinations during inference. This work begins with unimodal visual perception and a human-in-the-loop validation mechanism, where human evaluators assess and refine uncertain predictions to enhance overall reliability. However, this approach can only verify the generated content but is not capable of refining its model based on informative human feedback. This limitation motivates the final hallucination mitigation framework, where AI-based evaluation is transformed into a reward signal that actively updates the foundation model's parameters during inference, enabling continuous model improvement and reduced hallucination. Throughout the subsequent chapters, we progressively unfold the work, tracing its evolution from unimodal perception to multimodal understanding, and from static fine-tuning to feedback-driven, lifelong learning. The research initially explores contrastive learning to enhance intra-modal representations. Next, the study extends contrastive mechanisms to multimodality and hallucination detection. We subsequently evaluate parameter-efficient fine-tuning strategies in practical settings. Finally, the reinforcement learning-based test-time adaptation framework empowers the multimodal foundation model to refine its internal parameters and generated outputs during inference, achieving ongoing enhancement without offline retraining and effectively reducing hallucinations through iterative feedback-driven correction. Together, these studies establish a unified framework that transforms multimodal foundation models from passive generators into adaptive learners, leveraging lifelong learning to thereby achieve effective hallucination mitigation.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy (PhD)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Computer and Information Sciences

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhao, Fei
Contributors dc:contributor
  • <p>Chengcui Zhang</p>
  • <p>Baocheng Geng<br>Brandon Oubre<br>Keren Li<br>Tianyang Wang</p>

Subjects

dc:subject × 3

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
University of Alabama Birmingham
Base URL
digitalcommons.library.uab.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Zhao, Fei. Efficient Multimodal Foundation Model Tuning For Hallucination Mitigation. Dissertation thesis, https://digitalcommons.library.uab.edu/etd-2020s/142