Back to search

Georgia Institute of Technology

Leveraging Pretrained Vision and Language Models for Robotic Manipulation

Abstract

dc:description.abstract

The objective of this dissertation is to investigate how pretrained vision and language models can be integrated into robotic manipulation systems, with the goal of leveraging their inherent knowledge and reasoning capacities. It is motivated by the rapid advancement of data-driven approaches for robotic manipulation, which develops trainable policy networks that can perform a wide range of manipulation tasks, spanning from simple grasping to more complicated household (e.g. cooking, cloth folding) and industrial (e.g. assembly, screw driving) tasks. Despite this progress, several challenges remain: (1) Collecting robotic demonstration data is often costly and time consuming; (2) Learned policies are often limited in generalizability; (3) Integrating trainable policies into a working manipulation system remains non-trivial. The difficulties hinder the scalability and generalizability of a learning-based robotic system. On the other hand, recent progress shows the feasibility of scaling vision and language models, where large vision and language models that are pretrained on internet-scale data have exhibited impressive performance and generalizability. Hence, a promising strategy is to leverage the knowledge embedded in pretrained vision and language models for robotic tasks. In this dissertation, we verify the benefit of and explore the strategies for leveraging pretrained vision and language models for robotic systems. We propose a series of methods targeting multiple essential robotic functionalities, including language-conditioned robotic grasping, 6-DoF grasping from visual observations, and spatial understanding for task planning. Furthermore, we study the limitations and corresponding mitigations of Vision-Language-Action models (VLAs), which is a recent paradigm that unifies vision-language understanding and robotic control within a single large model by extending pretrained Vision-Language Models (VLMs).

Degree

thesis:*
Name thesis:degree_name
Electrical and Computer Engineering, PhD
Department thesis:degree_department
Electrical and Computer Engineering
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chen, Yiye
Advisor dc:contributor.advisor
  • Vela, Patricio A.
Committee members dc:contributor.committeemember
  • AlRegib, Ghassan
  • Kira, Zsolt
  • Gan, Lu
  • Vishwanath, Sriram

Rights

Language dc:language.iso
English

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81772

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Chen, Yiye. Leveraging Pretrained Vision and Language Models for Robotic Manipulation. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/81772