{"id":{"repo_id":"calpoly","oai_identifier":"oai:digitalcommons.calpoly.edu:theses-5077"},"canonical_url":"https://search.ndltd.org/etd/calpoly/oai:digitalcommons.calpoly.edu:theses-5077","repository":{"repo_id":"calpoly","name":"Cal Poly","base_url":"https://digitalcommons.calpoly.edu/do/oai/"},"display":{"title":"Prompt-Driven Tabletop Robotic Manipulation with a Structured LLM Interface and Visual Verification","abstract":"Creating a tabletop robotic manipulation system that connects language goals to robot actions is a challenging problem. Large language models can interpret mission objectives and reason through actions, but when placed directly in control of hardware they can produce hallucinated commands that result in unsafe behavior without the proper safeguards. This thesis presents a constrained LLM-guided manipulation system built around the Quanser Qarm, a four degree of freedom manipulator with a mounted Realsense RGB-D camera and a gripper with readable current. Every decision made by the LLM planner is routed through a gated command interface that restricts the model to legal actions for the current robot state, with visual and current-based verification closing the loop after each physical action. We tested the system across single-color stacking, stack correction, alternate color stack building and external robustness tasks across forty trials, achieving a 65\\% autonomous and non-autonomous completion rate. The system is able to adapt to unexpected scene changes including replaced, removed, and added cubes during place sequences. Dominant failures sources were due to object orientation, depth estimation error, and state tracking mismatches that could cascade into poor planner decisions. Through this work you may find useful lessons about integrating LLMs into physical systems, and how reliably an LLM can facilitate an autonomous system through language prompts.","abstract_html":"Creating a tabletop robotic manipulation system that connects language goals to robot actions is a challenging problem. Large language models can interpret mission objectives and reason through actions, but when placed directly in control of hardware they can produce hallucinated commands that result in unsafe behavior without the proper safeguards. This thesis presents a constrained LLM-guided manipulation system built around the Quanser Qarm, a four degree of freedom manipulator with a mounted Realsense RGB-D camera and a gripper with readable current. Every decision made by the LLM planner is routed through a gated command interface that restricts the model to legal actions for the current robot state, with visual and current-based verification closing the loop after each physical action. We tested the system across single-color stacking, stack correction, alternate color stack building and external robustness tasks across forty trials, achieving a 65\\% autonomous and non-autonomous completion rate. The system is able to adapt to unexpected scene changes including replaced, removed, and added cubes during place sequences. Dominant failures sources were due to object orientation, depth estimation error, and state tracking mismatches that could cascade into poor planner decisions. Through this work you may find useful lessons about integrating LLMs into physical systems, and how reliably an LLM can facilitate an autonomous system through language prompts.","abstract_has_math":false,"creators":["Franco, Tomas"],"institution":null,"degree_name":"MS in Mechanical Engineering","degree_level":null,"degree_discipline":"Mechanical Engineering","degree_department":null,"school":null,"contributors":["Mohammad Hasan","Mechanical Engineering","College of Engineering"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T07:12:58Z","subjects":["Tabletop robotic manipulation","prompt-driven robotics","visual state verification","structured LLM interfaces","JSON-based robot task planning","Quanser QArm","Electro-Mechanical Systems"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/3406","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.calpoly.edu/theses/3406","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.calpoly.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.calpoly.edu%3Atheses-5077","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Mohammad Hasan","Mechanical Engineering","College of Engineering"]},{"key":"dc:creator","label":"Author","values":["Franco, Tomas"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2026-08-01T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2026-08-01T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Mechanical Engineering"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Mechanical Engineering"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Tabletop robotic manipulation","prompt-driven robotics","visual state verification","structured LLM interfaces","JSON-based robot task planning","Quanser QArm","Electro-Mechanical Systems"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/3406"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Creating a tabletop robotic manipulation system that connects language goals to robot actions is a challenging problem. Large language models can interpret mission objectives and reason through actions, but when placed directly in control of hardware they can produce hallucinated commands that result in unsafe behavior without the proper safeguards. This thesis presents a constrained LLM-guided manipulation system built around the Quanser Qarm, a four degree of freedom manipulator with a mounted Realsense RGB-D camera and a gripper with readable current. Every decision made by the LLM planner is routed through a gated command interface that restricts the model to legal actions for the current robot state, with visual and current-based verification closing the loop after each physical action. We tested the system across single-color stacking, stack correction, alternate color stack building and external robustness tasks across forty trials, achieving a 65\\% autonomous and non-autonomous completion rate. The system is able to adapt to unexpected scene changes including replaced, removed, and added cubes during place sequences. Dominant failures sources were due to object orientation, depth estimation error, and state tracking mismatches that could cascade into poor planner decisions. Through this work you may find useful lessons about integrating LLMs into physical systems, and how reliably an LLM can facilitate an autonomous system through language prompts."]},{"key":"dc:title","label":"Title","values":["Prompt-Driven Tabletop Robotic Manipulation with a Structured LLM Interface and Visual Verification"]}]}],"canonical_facts":{"dc:contributor":["Mohammad Hasan","Mechanical Engineering","College of Engineering"],"dc:creator":["Franco, Tomas"],"dc:date.available":["2026-08-01T07:00:00Z"],"dc:date.created":["2026-08-01T07:00:00Z"],"dc:description.abstract":["Creating a tabletop robotic manipulation system that connects language goals to robot actions is a challenging problem. Large language models can interpret mission objectives and reason through actions, but when placed directly in control of hardware they can produce hallucinated commands that result in unsafe behavior without the proper safeguards. This thesis presents a constrained LLM-guided manipulation system built around the Quanser Qarm, a four degree of freedom manipulator with a mounted Realsense RGB-D camera and a gripper with readable current. Every decision made by the LLM planner is routed through a gated command interface that restricts the model to legal actions for the current robot state, with visual and current-based verification closing the loop after each physical action. We tested the system across single-color stacking, stack correction, alternate color stack building and external robustness tasks across forty trials, achieving a 65\\% autonomous and non-autonomous completion rate. The system is able to adapt to unexpected scene changes including replaced, removed, and added cubes during place sequences. Dominant failures sources were due to object orientation, depth estimation error, and state tracking mismatches that could cascade into poor planner decisions. Through this work you may find useful lessons about integrating LLMs into physical systems, and how reliably an LLM can facilitate an autonomous system through language prompts."],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/3406"],"dc:subject":["Tabletop robotic manipulation","prompt-driven robotics","visual state verification","structured LLM interfaces","JSON-based robot task planning","Quanser QArm","Electro-Mechanical Systems"],"dc:title":["Prompt-Driven Tabletop Robotic Manipulation with a Structured LLM Interface and Visual Verification"],"thesis:degree_discipline":["Mechanical Engineering"],"thesis:degree_name":["MS in Mechanical Engineering"]},"updated_at":"2026-11-19T07:12:58Z"}