Back to search

Cal Poly

Prompt-Driven Tabletop Robotic Manipulation with a Structured LLM Interface and Visual Verification

Abstract

dc:description.abstract

Creating a tabletop robotic manipulation system that connects language goals to robot actions is a challenging problem. Large language models can interpret mission objectives and reason through actions, but when placed directly in control of hardware they can produce hallucinated commands that result in unsafe behavior without the proper safeguards. This thesis presents a constrained LLM-guided manipulation system built around the Quanser Qarm, a four degree of freedom manipulator with a mounted Realsense RGB-D camera and a gripper with readable current. Every decision made by the LLM planner is routed through a gated command interface that restricts the model to legal actions for the current robot state, with visual and current-based verification closing the loop after each physical action. We tested the system across single-color stacking, stack correction, alternate color stack building and external robustness tasks across forty trials, achieving a 65\% autonomous and non-autonomous completion rate. The system is able to adapt to unexpected scene changes including replaced, removed, and added cubes during place sequences. Dominant failures sources were due to object orientation, depth estimation error, and state tracking mismatches that could cascade into poor planner decisions. Through this work you may find useful lessons about integrating LLMs into physical systems, and how reliably an LLM can facilitate an autonomous system through language prompts.

Degree

thesis:*
Name thesis:degree_name
MS in Mechanical Engineering
Discipline thesis:degree_discipline
Mechanical Engineering

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Franco, Tomas
Contributors dc:contributor
  • Mohammad Hasan
  • Mechanical Engineering
  • College of Engineering

Subjects

dc:subject × 7

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.calpoly.edu/theses/3406

Chain of custody

source
Harvested from
Cal Poly
Base URL
digitalcommons.calpoly.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Franco, Tomas. Prompt-Driven Tabletop Robotic Manipulation with a Structured LLM Interface and Visual Verification. https://digitalcommons.calpoly.edu/theses/3406