Back to search

Georgia Institute of Technology

From Web to World: Harnessing Foundation Models for Intelligent Robotic Assistants in Real-World Environments

Abstract

dc:description.abstract

In this dissertation, we explore how simulated embodied experience and spatial grounding can enhance foundation models for robotics, bridging the gap between abstract reasoning capabilities and physical robotic interaction. We present three key contributions: (1) Adaptive Skill Coordination (ASC) and Language-guided Skill Coordination (LSC): These approaches address open-vocabulary long-horizon mobile manipulation tasks, demonstrating how simulators can develop fundamental sensorimotor skills. This creates a robust repertoire of capabilities that foundation models can employ for real-world interaction. (2) Vision-Language Frontier Maps (VLFM): This approach combines pre-trained vision-language models with low-level navigation policies trained in simulation. By grounding these models with explicit spatial maps of the environment, VLFM enhances their ability to reason about and navigate in the real world. (3) A novel method for fine-tuning multi-modal large language models using simulated data: This approach enables models to develop reasoning capabilities beyond semantic understanding for navigation tasks. By fine-tuning with diverse simulated scenarios, we demonstrate that models leverage knowledge from both their pre-training on web-scale data and navigation training to achieve superior navigation performance. This research aims to create embodied AI systems that leverage the strengths of foundation models while effectively operating in real-world environments.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yokoyama, Naoki Harrison
Advisor dc:contributor.advisor
  • Ha, Sehoon
Committee members dc:contributor.committeemember
  • Batra, Dhruv
  • Tan, Jie
  • Kalakrishnan, Mrinal
  • Koltun, Vladlen

Subjects

dc:subject × 7

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/77941

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Yokoyama, Naoki Harrison. From Web to World: Harnessing Foundation Models for Intelligent Robotic Assistants in Real-World Environments. Doctoral thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/77941