Back to search

Boston University

Reasoning in visual environments

Abstract

dc:description.abstract

Real-world visual tasks require a synchrony of perception and reasoning—people not only perceive the visual environment (e.g. the position of all the objects), but also reason in it. For example, they imagine consequences of actions (e.g. what would happen if I ...), manipulate visual symbols and maps (e.g., maybe I need to fit this piece here to ...), and iterate (e.g. based on the result of the previous action, I should now...). Such reasoning, bundled as visual reasoning—about space, time, and affordances—forms a fundamental bedrock of human intelligence. While today's multimodal AI models excel at textual reasoning (e.g. coding) and visual perception, they falter when asked to reason visually. As AI transitions toward embodied and virtual agents assisting humans, closing this gap is critical. The current bottlenecks to close the gap revolve around having access to rich interactive visual data (which is lacking in current pretraining) and training paradigms that allow models to internally reason visually. Hence, this dissertation aims to improve vision-centric reasoning in multimodal AI models along these two complementary axes. On the data side, we develop methods for generating training data with minimal human supervision that explicitly encode visual-spatial knowledge in language. This includes improving models' understanding of novel complex spatial compositions in static images, as well as leveraging physics engines to generate data capturing causal visual interactions when objects and the ego-self are in motion. On the training paradigm side, I develop an approach that encourages models to learn latent reasoning processes underlying visual tasks rather than simply memorizing final answers. Finally, we dive into an application that requires iterative visual reasoning—where an agent needs to iteratively reason and take actions on a visual interface to complete complex digital tasks. We show promising preliminary results on using synthetic environments to improve the performance of agents on real-world benchmarks. Together, these contributions advance visual thinking skills that AI systems need to operate in both the physical and digital worlds.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ray, Arijit
Advisor dc:contributor.advisor
  • Saenko, Kate

Subjects

dc:subject × 7

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53613

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Ray, Arijit. Reasoning in visual environments. doctoral thesis, Boston University, 2026. https://hdl.handle.net/2144/53613