{"id":{"repo_id":"usf","oai_identifier":"oai:digitalcommons.usf.edu:etd-10945"},"canonical_url":"https://search.ndltd.org/etd/usf/oai:digitalcommons.usf.edu:etd-10945","repository":{"repo_id":"usf","name":"University of South Florida","base_url":"https://digitalcommons.usf.edu/do/oai/"},"display":{"title":"Task Progress Assessment and Monitoring Using Self-Supervised Learning","abstract":"Robotic manipulation for cooking requires a thorough understanding of the cooking environment. The robot must understand the cooking objects and their states at each intermediate level as the process continues. To understand these states, we need frame-level annotations. To overcome this frame-level dependency, we introduce a self-supervised learning method to obtain the frame-level state representation with ”temporal video alignment” and ”contrastive learning.”In this work, we use self-supervised learning to train a model using multiple videos of the same action being performed in various settings. This model can extract frame-level embedding space and align videos via simple distance-based matching. We show that this learned embedding space can be used to perform state and progress estimation and anomaly detection. Finally, we demonstrate how these embeddings can be used to perform robotic mixing by capturing state progression from offline videos. We use Q-learning and UR5 robotic arm to perform mixing.","abstract_html":"Robotic manipulation for cooking requires a thorough understanding of the cooking environment. The robot must understand the cooking objects and their states at each intermediate level as the process continues. To understand these states, we need frame-level annotations. To overcome this frame-level dependency, we introduce a self-supervised learning method to obtain the frame-level state representation with ”temporal video alignment” and ”contrastive learning.”In this work, we use self-supervised learning to train a model using multiple videos of the same action being performed in various settings. This model can extract frame-level embedding space and align videos via simple distance-based matching. We show that this learned embedding space can be used to perform state and progress estimation and anomaly detection. Finally, we demonstrate how these embeddings can be used to perform robotic mixing by capturing state progression from offline videos. We use Q-learning and UR5 robotic arm to perform mixing.","abstract_has_math":false,"creators":["Bobbala, Sainath Reddy"],"institution":"University of South Florida","degree_name":"MS in Computer Science (M.S.C.S.)","degree_level":"Thesis","degree_discipline":"Computer Science and Engineering","degree_department":null,"school":null,"contributors":["Yu Sun, Ph.D.","Dmitry B. Goldgof, Ph.D.","Shaun Canavan, Ph.D."],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2022,"date_issued":"2022-11-07T08:00:00Z","date_published":"2022-11-07T08:00:00Z","updated_at":"2026-11-19T08:48:12Z","subjects":["Contrastive Learning","Q-Learning","Temporal Cycle Consistency","Computer Sciences"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.usf.edu/etd/9748","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.usf.edu/etd/9748","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.usf.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.usf.edu%3Aetd-10945","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Yu Sun, Ph.D.","Dmitry B. Goldgof, Ph.D.","Shaun Canavan, Ph.D."]},{"key":"dc:creator","label":"Author","values":["Bobbala, Sainath Reddy"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2022-11-07T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science and Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis","M.S.C.S."]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Computer Science (M.S.C.S.)"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of South Florida"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Contrastive Learning","Q-Learning","Temporal Cycle Consistency","Computer Sciences"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.usf.edu/etd/9748"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Robotic manipulation for cooking requires a thorough understanding of the cooking environment. The robot must understand the cooking objects and their states at each intermediate level as the process continues. To understand these states, we need frame-level annotations. To overcome this frame-level dependency, we introduce a self-supervised learning method to obtain the frame-level state representation with ”temporal video alignment” and ”contrastive learning.”In this work, we use self-supervised learning to train a model using multiple videos of the same action being performed in various settings. This model can extract frame-level embedding space and align videos via simple distance-based matching. We show that this learned embedding space can be used to perform state and progress estimation and anomaly detection. Finally, we demonstrate how these embeddings can be used to perform robotic mixing by capturing state progression from offline videos. We use Q-learning and UR5 robotic arm to perform mixing."]},{"key":"dc:title","label":"Title","values":["Task Progress Assessment and Monitoring Using Self-Supervised Learning"]}]}],"canonical_facts":{"dc:contributor":["Yu Sun, Ph.D.","Dmitry B. Goldgof, Ph.D.","Shaun Canavan, Ph.D."],"dc:creator":["Bobbala, Sainath Reddy"],"dc:date":["2022-11-07T08:00:00Z"],"dc:description":["Robotic manipulation for cooking requires a thorough understanding of the cooking environment. The robot must understand the cooking objects and their states at each intermediate level as the process continues. To understand these states, we need frame-level annotations. To overcome this frame-level dependency, we introduce a self-supervised learning method to obtain the frame-level state representation with ”temporal video alignment” and ”contrastive learning.”In this work, we use self-supervised learning to train a model using multiple videos of the same action being performed in various settings. This model can extract frame-level embedding space and align videos via simple distance-based matching. We show that this learned embedding space can be used to perform state and progress estimation and anomaly detection. Finally, we demonstrate how these embeddings can be used to perform robotic mixing by capturing state progression from offline videos. We use Q-learning and UR5 robotic arm to perform mixing."],"dc:identifier":["https://digitalcommons.usf.edu/etd/9748"],"dc:subject":["Contrastive Learning","Q-Learning","Temporal Cycle Consistency","Computer Sciences"],"dc:title":["Task Progress Assessment and Monitoring Using Self-Supervised Learning"],"thesis:degree_discipline":["Computer Science and Engineering"],"thesis:degree_level":["Thesis","M.S.C.S."],"thesis:degree_name":["MS in Computer Science (M.S.C.S.)"],"thesis:institution_name":["University of South Florida"]},"updated_at":"2026-11-19T08:48:12Z"}