Abstract
dc:description.abstractReal-world visual tasks require a synchrony of perception and reasoning—people not only perceive the visual environment (e.g. the position of all the objects), but also reason in it. For example, they imagine consequences of actions (e.g. what would happen if I ...), manipulate visual symbols and maps (e.g., maybe I need to fit this piece here to ...), and iterate (e.g. based on the result of the previous action, I should now...). Such reasoning, bundled as visual reasoning—about space, time, and affordances—forms a fundamental bedrock of human intelligence. While today's multimodal AI models excel at textual reasoning (e.g. coding) and visual perception, they falter when asked to reason visually. As AI transitions toward embodied and virtual agents assisting humans, closing this gap is critical. The current bottlenecks to close the gap revolve around having access to rich interactive visual data (which is lacking in current pretraining) and training paradigms that allow models to internally reason visually. Hence, this dissertation aims to improve vision-centric reasoning in multimodal AI models along these two complementary axes. On the data side, we develop methods for generating training data with minimal human supervision that explicitly encode visual-spatial knowledge in language. This includes improving models' understanding of novel complex spatial compositions in static images, as well as leveraging physics engines to generate data capturing causal visual interactions when objects and the ego-self are in motion. On the training paradigm side, I develop an approach that encourages models to learn latent reasoning processes underlying visual tasks rather than simply memorizing final answers. Finally, we dive into an application that requires iterative visual reasoning—where an agent needs to iteratively reason and take actions on a visual interface to complete complex digital tasks. We show promising preliminary results on using synthetic environments to improve the performance of agents on real-world benchmarks. Together, these contributions advance visual thinking skills that AI systems need to operate in both the physical and digital worlds.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ray, Arijit
- Advisor dc:contributor.advisor
-
- Saenko, Kate
Subjects
dc:subject × 7Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53613