{"id":{"repo_id":"bu","oai_identifier":"oai:open.bu.edu:2144/53613"},"canonical_url":"https://search.ndltd.org/etd/bu/oai:open.bu.edu:2144/53613","repository":{"repo_id":"bu","name":"Boston University","base_url":"https://open.bu.edu/oai/request"},"display":{"title":"Reasoning in visual environments","abstract":"Real-world visual tasks require a synchrony of perception and reasoning—people not only perceive the visual environment (e.g. the position of all the objects), but also reason in it. For example, they imagine consequences of actions (e.g. what would happen if I ...), manipulate visual symbols and maps (e.g., maybe I need to fit this piece here to ...), and iterate (e.g. based on the result of the previous action, I should now...). Such reasoning, bundled as visual reasoning—about space, time, and affordances—forms a fundamental bedrock of human intelligence. While today's multimodal AI models excel at textual reasoning (e.g. coding) and visual perception, they falter when asked to reason visually. As AI transitions toward embodied and virtual agents assisting humans, closing this gap is critical. The current bottlenecks to close the gap revolve around having access to rich interactive visual data (which is lacking in current pretraining) and training paradigms that allow models to internally reason visually. Hence, this dissertation aims to improve vision-centric reasoning in multimodal AI models along these two complementary axes. On the data side, we develop methods for generating training data with minimal human supervision that explicitly encode visual-spatial knowledge in language. This includes improving models' understanding of novel complex spatial compositions in static images, as well as leveraging physics engines to generate data capturing causal visual interactions when objects and the ego-self are in motion. On the training paradigm side, I develop an approach that encourages models to learn latent reasoning processes underlying visual tasks rather than simply memorizing final answers. Finally, we dive into an application that requires iterative visual reasoning—where an agent needs to iteratively reason and take actions on a visual interface to complete complex digital tasks. We show promising preliminary results on using synthetic environments to improve the performance of agents on real-world benchmarks. Together, these contributions advance visual thinking skills that AI systems need to operate in both the physical and digital worlds.","abstract_html":"Real-world visual tasks require a synchrony of perception and reasoning—people not only perceive the visual environment (e.g. the position of all the objects), but also reason in it. For example, they imagine consequences of actions (e.g. what would happen if I ...), manipulate visual symbols and maps (e.g., maybe I need to fit this piece here to ...), and iterate (e.g. based on the result of the previous action, I should now...). Such reasoning, bundled as visual reasoning—about space, time, and affordances—forms a fundamental bedrock of human intelligence. While today&#x27;s multimodal AI models excel at textual reasoning (e.g. coding) and visual perception, they falter when asked to reason visually. As AI transitions toward embodied and virtual agents assisting humans, closing this gap is critical. The current bottlenecks to close the gap revolve around having access to rich interactive visual data (which is lacking in current pretraining) and training paradigms that allow models to internally reason visually. Hence, this dissertation aims to improve vision-centric reasoning in multimodal AI models along these two complementary axes. On the data side, we develop methods for generating training data with minimal human supervision that explicitly encode visual-spatial knowledge in language. This includes improving models&#x27; understanding of novel complex spatial compositions in static images, as well as leveraging physics engines to generate data capturing causal visual interactions when objects and the ego-self are in motion. On the training paradigm side, I develop an approach that encourages models to learn latent reasoning processes underlying visual tasks rather than simply memorizing final answers. Finally, we dive into an application that requires iterative visual reasoning—where an agent needs to iteratively reason and take actions on a visual interface to complete complex digital tasks. We show promising preliminary results on using synthetic environments to improve the performance of agents on real-world benchmarks. Together, these contributions advance visual thinking skills that AI systems need to operate in both the physical and digital worlds.","abstract_has_math":false,"creators":["Ray, Arijit"],"institution":"Boston University","degree_name":"Doctor of Philosophy","degree_level":"doctoral","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":[],"advisors":["Saenko, Kate"],"committee_chairs":[],"committee_members":[],"year":2026,"date_issued":"2026","date_published":"2026","updated_at":"2026-11-18T20:20:11Z","subjects":["Computer science","AI","computer vision","large language models","multimodal","reasoning","visual reasoning"],"languages":["en_US"],"rights":["Attribution 4.0 International"],"rights_urls":["http://creativecommons.org/licenses/by/4.0/"],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2144/53613","outbound_label":"Handle","outbound_source":"dc:identifier.uri","record_url":"https://hdl.handle.net/2144/53613","record_label":"Repository record","record_kind":"handle","record_source":"dc:identifier.uri","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://open.bu.edu/oai/request?verb=GetRecord&metadataPrefix=dim&identifier=oai%3Aopen.bu.edu%3A2144%2F53613","prefix":"dim"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Saenko, Kate"]},{"key":"dc:creator","label":"Author","values":["Ray, Arijit"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2026-10-05T18:36:16Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2026-10-05T18:36:16Z"]},{"key":"dc:date.issued","label":"Date","values":["2026"]},{"key":"dc:date.updated","label":"Dc Date Updated","values":["2026-10-02T19:15:30Z"]},{"key":"dc:type","label":"Dc Type","values":["Thesis/Dissertation"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["doctoral"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Boston University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science","AI","computer vision","large language models","multimodal","reasoning","visual reasoning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en_US"]},{"key":"dc:rights","label":"Dc Rights","values":["Attribution 4.0 International"]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://creativecommons.org/licenses/by/4.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/2144/53613"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["2026"]},{"key":"dc:description.abstract","label":"Abstract","values":["Real-world visual tasks require a synchrony of perception and reasoning—people not only perceive the visual environment (e.g. the position of all the objects), but also reason in it. For example, they imagine consequences of actions (e.g. what would happen if I ...), manipulate visual symbols and maps (e.g., maybe I need to fit this piece here to ...), and iterate (e.g. based on the result of the previous action, I should now...). Such reasoning, bundled as visual reasoning—about space, time, and affordances—forms a fundamental bedrock of human intelligence. While today's multimodal AI models excel at textual reasoning (e.g. coding) and visual perception, they falter when asked to reason visually. As AI transitions toward embodied and virtual agents assisting humans, closing this gap is critical. The current bottlenecks to close the gap revolve around having access to rich interactive visual data (which is lacking in current pretraining) and training paradigms that allow models to internally reason visually. Hence, this dissertation aims to improve vision-centric reasoning in multimodal AI models along these two complementary axes. On the data side, we develop methods for generating training data with minimal human supervision that explicitly encode visual-spatial knowledge in language. This includes improving models' understanding of novel complex spatial compositions in static images, as well as leveraging physics engines to generate data capturing causal visual interactions when objects and the ego-self are in motion. On the training paradigm side, I develop an approach that encourages models to learn latent reasoning processes underlying visual tasks rather than simply memorizing final answers. Finally, we dive into an application that requires iterative visual reasoning—where an agent needs to iteratively reason and take actions on a visual interface to complete complex digital tasks. We show promising preliminary results on using synthetic environments to improve the performance of agents on real-world benchmarks. Together, these contributions advance visual thinking skills that AI systems need to operate in both the physical and digital worlds."]},{"key":"dc:title","label":"Title","values":["Reasoning in visual environments"]}]}],"canonical_facts":{"dc:contributor.advisor":["Saenko, Kate"],"dc:creator":["Ray, Arijit"],"dc:date.accessioned":["2026-10-05T18:36:16Z"],"dc:date.available":["2026-10-05T18:36:16Z"],"dc:date.issued":["2026"],"dc:date.updated":["2026-10-02T19:15:30Z"],"dc:description":["2026"],"dc:description.abstract":["Real-world visual tasks require a synchrony of perception and reasoning—people not only perceive the visual environment (e.g. the position of all the objects), but also reason in it. For example, they imagine consequences of actions (e.g. what would happen if I ...), manipulate visual symbols and maps (e.g., maybe I need to fit this piece here to ...), and iterate (e.g. based on the result of the previous action, I should now...). Such reasoning, bundled as visual reasoning—about space, time, and affordances—forms a fundamental bedrock of human intelligence. While today's multimodal AI models excel at textual reasoning (e.g. coding) and visual perception, they falter when asked to reason visually. As AI transitions toward embodied and virtual agents assisting humans, closing this gap is critical. The current bottlenecks to close the gap revolve around having access to rich interactive visual data (which is lacking in current pretraining) and training paradigms that allow models to internally reason visually. Hence, this dissertation aims to improve vision-centric reasoning in multimodal AI models along these two complementary axes. On the data side, we develop methods for generating training data with minimal human supervision that explicitly encode visual-spatial knowledge in language. This includes improving models' understanding of novel complex spatial compositions in static images, as well as leveraging physics engines to generate data capturing causal visual interactions when objects and the ego-self are in motion. On the training paradigm side, I develop an approach that encourages models to learn latent reasoning processes underlying visual tasks rather than simply memorizing final answers. Finally, we dive into an application that requires iterative visual reasoning—where an agent needs to iteratively reason and take actions on a visual interface to complete complex digital tasks. We show promising preliminary results on using synthetic environments to improve the performance of agents on real-world benchmarks. Together, these contributions advance visual thinking skills that AI systems need to operate in both the physical and digital worlds."],"dc:identifier.uri":["https://hdl.handle.net/2144/53613"],"dc:language.iso":["en_US"],"dc:rights":["Attribution 4.0 International"],"dc:rights.uri":["http://creativecommons.org/licenses/by/4.0/"],"dc:subject":["Computer science","AI","computer vision","large language models","multimodal","reasoning","visual reasoning"],"dc:title":["Reasoning in visual environments"],"dc:type":["Thesis/Dissertation"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["doctoral"],"thesis:degree_name":["Doctor of Philosophy"],"thesis:institution_name":["Boston University"]},"updated_at":"2026-11-18T20:20:11Z"}