Georgia Institute of Technology
Leveraging Pretrained Vision and Language Models for Robotic Manipulation
Abstract
dc:description.abstractThe objective of this dissertation is to investigate how pretrained vision and language models can be integrated into robotic manipulation systems, with the goal of leveraging their inherent knowledge and reasoning capacities. It is motivated by the rapid advancement of data-driven approaches for robotic manipulation, which develops trainable policy networks that can perform a wide range of manipulation tasks, spanning from simple grasping to more complicated household (e.g. cooking, cloth folding) and industrial (e.g. assembly, screw driving) tasks. Despite this progress, several challenges remain: (1) Collecting robotic demonstration data is often costly and time consuming; (2) Learned policies are often limited in generalizability; (3) Integrating trainable policies into a working manipulation system remains non-trivial. The difficulties hinder the scalability and generalizability of a learning-based robotic system. On the other hand, recent progress shows the feasibility of scaling vision and language models, where large vision and language models that are pretrained on internet-scale data have exhibited impressive performance and generalizability. Hence, a promising strategy is to leverage the knowledge embedded in pretrained vision and language models for robotic tasks. In this dissertation, we verify the benefit of and explore the strategies for leveraging pretrained vision and language models for robotic systems. We propose a series of methods targeting multiple essential robotic functionalities, including language-conditioned robotic grasping, 6-DoF grasping from visual observations, and spatial understanding for task planning. Furthermore, we study the limitations and corresponding mitigations of Vision-Language-Action models (VLAs), which is a recent paradigm that unifies vision-language understanding and robotic control within a single large model by extending pretrained Vision-Language Models (VLMs).
Degree
thesis:*- Name thesis:degree_name
- Electrical and Computer Engineering, PhD
- Department thesis:degree_department
- Electrical and Computer Engineering
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Chen, Yiye
- Advisor dc:contributor.advisor
-
- Vela, Patricio A.
- Committee members dc:contributor.committeemember
-
- AlRegib, Ghassan
- Kira, Zsolt
- Gan, Lu
- Vishwanath, Sriram
Rights
- Language dc:language.iso
- English
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/81772