Back to search

University of Minnesota

Towards More Human-like Multi-Modal Understanding and Prediction using Novel Benchmark, Evaluation Frameworks and Modeling Strategies

Abstract

dc:description.abstract

Recent advances in artificial intelligence (AI) have pushed the field to achieve remarkable heights in areas such as image recognition, scanpath prediction, and other multimodal tasks. However, despite these successes, today's AI systems still struggle to emulate the performance of human cognition and behavior. While machines excel at processing vast amounts of data, they often lack the nuanced ability to integrate multimodal inputs, demonstrate transparent reasoning processes, and adapt to ever-changing real-world conditions. Our thesis is motivated by the need to overcome these shortcomings. Current multimodal models frequently struggle when faced with distractions, unexpected environmental changes, or complex multimodal tasks, resulting in systems that are difficult and less ideal to use in real-life. In contrast, human cognition naturally exhibits robustness, clarity in reasoning, and a rich, multi-layered approach to attention. These qualities enable humans to understand and interact with their surroundings in a manner that is both adaptive and transparent. To bridge this gap, our research adopts a comprehensive strategy that focuses on building AI systems with human-like capabilities. We begin by developing novel datasets that capture the intricacies of human attention and reasoning as well as their adaptability in real-world conditions. Our VisualHow dataset, for example, provides detailed annotations that support fine-grained multimodal problem solving, while the IQVA dataset collects attention patterns when answering questions in immersive, real-world environments, offering insights into adaptive, task-driven performance. On the other hand, our ongoing IR-VQA work highlights the problem of multimodal distractions in Large Vision-Language Models (LVLMs) and collects a large-scale benchmark to further understand and evaluate this challenge, which is common in the real world due to the amount of extraneous information. Complementing these datasets, we introduce an advanced evaluation framework that reflects the multi-facetedness of human attention. Specifically, it assesses both the spatial and exploratory dynamics of attention in a scanpath prediction task. By designing multi-dimensional metrics to analyze scanpaths, our evaluation approach ensures that the system's internal operations are aligned with human visual behaviors. Finally, our modeling efforts integrate these insights into robust AI systems. We propose innovative approaches that combine traditional saliency-based methods with reinforcement learning to capture the dynamic, sequential nature of attention, as well as methods that utilize gating mechanisms to address the issue of distractions for LVLMs. Through these efforts, we address key challenges including the alignment of model and human visual behaviors, the management of distractions, and the need for more advanced and human-like reasoning. By uniting novel datasets, comprehensive evaluation framework, and state-of-the-art modeling techniques, this work lays the foundation for AI systems that not only achieve high performance on standard benchmarks but also reason, adapt, and generalize in ways that mirror human intelligence. Overall, this thesis contributes to the development of practical, reliable multimodal systems that bridge the gap between machine capabilities and human-like understanding.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yang, Jinhui

Subjects

dc:subject × 6

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Citation
Yang, Jinhui. (2026). Towards More Human-like Multi-Modal Understanding and Prediction using Novel Benchmark, Evaluation Frameworks and Modeling Strategies. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/282341.

Chain of custody

source
Harvested from
University of Minnesota
Base URL
conservancy.umn.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Yang, Jinhui. Towards More Human-like Multi-Modal Understanding and Prediction using Novel Benchmark, Evaluation Frameworks and Modeling Strategies. 2026. https://hdl.handle.net/11299/282341