{"id":{"repo_id":"usf","oai_identifier":"oai:digitalcommons.usf.edu:etd-10475"},"canonical_url":"https://search.ndltd.org/etd/usf/oai:digitalcommons.usf.edu:etd-10475","repository":{"repo_id":"usf","name":"University of South Florida","base_url":"https://digitalcommons.usf.edu/do/oai/"},"display":{"title":"Knowledge Extraction and Inference Based on Visual Understanding of Cooking Contents","abstract":"In this dissertation, we discuss our work on analyzing cooking content for the ultimate goal ofautomatic robotic manipulation. For a robot to perform a cooking task, it will need to both have an understanding of the scene and utilize prior knowledge. We will explore two main sub-problems: knowledge extraction and inference, and visual understanding of the scene in this dissertation. Visual understanding of a scene, requires algorithms that can visually infer information from a single image or video. Many algorithms in the area of image classification, object detection, or activity recognition can be used in this area. Although great advances has been achieved by the emergence of deep learning, state-of-the art algorithms in this area have limitations. To attempt to overcome this lack of performance, we propose to use structured knowledge representations combined with state of the art deep learning techniques for visual understanding of cooking videos. Besides objects, and motions, we recognize that states of objects are also very important in interpreting the scene and therefore extensively explore the problem of states in visual cooking content. We introduce the state identification challenge in cooking applications and collect a dataset for research in the area of ingredient state analysis. We further look into the problem of simultaneous knowledge extraction from a single image and extracting information about ingredients, their states, the inter-connection between different objects in the scene and the motion-object interconnections. This problem requires an algorithm that can model the correlation of various concepts in a single image simultaneously. Using deep algorithms that can take as input multiple inputs and generate multiple outputs are fit for this problem. Therefore we propose to incorporate auto-regressive self-attention based mechanisms to extract knowledge from a single image. We show that the knowledge acquired from a single image can be used for calorie estimation. We suggest that total knowledge extraction from a single image can be used in future work for task graph inference.","abstract_html":"In this dissertation, we discuss our work on analyzing cooking content for the ultimate goal ofautomatic robotic manipulation. For a robot to perform a cooking task, it will need to both have an understanding of the scene and utilize prior knowledge. We will explore two main sub-problems: knowledge extraction and inference, and visual understanding of the scene in this dissertation. Visual understanding of a scene, requires algorithms that can visually infer information from a single image or video. Many algorithms in the area of image classification, object detection, or activity recognition can be used in this area. Although great advances has been achieved by the emergence of deep learning, state-of-the art algorithms in this area have limitations. To attempt to overcome this lack of performance, we propose to use structured knowledge representations combined with state of the art deep learning techniques for visual understanding of cooking videos. Besides objects, and motions, we recognize that states of objects are also very important in interpreting the scene and therefore extensively explore the problem of states in visual cooking content. We introduce the state identification challenge in cooking applications and collect a dataset for research in the area of ingredient state analysis. We further look into the problem of simultaneous knowledge extraction from a single image and extracting information about ingredients, their states, the inter-connection between different objects in the scene and the motion-object interconnections. This problem requires an algorithm that can model the correlation of various concepts in a single image simultaneously. Using deep algorithms that can take as input multiple inputs and generate multiple outputs are fit for this problem. Therefore we propose to incorporate auto-regressive self-attention based mechanisms to extract knowledge from a single image. We show that the knowledge acquired from a single image can be used for calorie estimation. We suggest that total knowledge extraction from a single image can be used in future work for task graph inference.","abstract_has_math":false,"creators":["Babaeian Jelodar, Ahmad Babaeian"],"institution":"University of South Florida","degree_name":"Doctor of Philosophy (Ph.D.)","degree_level":"Dissertation","degree_discipline":"Computer Science and Engineering","degree_department":null,"school":null,"contributors":["Yu Y. Sun, Ph.D.","Shaun Canavan, Ph.D.","Heather Culbertson, Ph.D.","John Licato, Ph.D.","Kyle Reed, Ph.D."],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2021,"date_issued":"2021-07-01T07:00:00Z","date_published":"2021-07-01T07:00:00Z","updated_at":"2026-11-19T08:47:49Z","subjects":["Calorie Estimation","Cooking Video Understanding","Ingredient Recognition","Knowledge Representation","Video Understanding","Deep Learning","Computer Sciences"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.usf.edu/etd/9278","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.usf.edu/etd/9278","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.usf.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.usf.edu%3Aetd-10475","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Yu Y. Sun, Ph.D.","Shaun Canavan, Ph.D.","Heather Culbertson, Ph.D.","John Licato, Ph.D.","Kyle Reed, Ph.D."]},{"key":"dc:creator","label":"Author","values":["Babaeian Jelodar, Ahmad Babaeian"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2021-07-01T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science and Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation","Ph.D."]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy (Ph.D.)"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of South Florida"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Calorie Estimation","Cooking Video Understanding","Ingredient Recognition","Knowledge Representation","Video Understanding","Deep Learning","Computer Sciences"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.usf.edu/etd/9278"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["In this dissertation, we discuss our work on analyzing cooking content for the ultimate goal ofautomatic robotic manipulation. For a robot to perform a cooking task, it will need to both have an understanding of the scene and utilize prior knowledge. We will explore two main sub-problems: knowledge extraction and inference, and visual understanding of the scene in this dissertation. Visual understanding of a scene, requires algorithms that can visually infer information from a single image or video. Many algorithms in the area of image classification, object detection, or activity recognition can be used in this area. Although great advances has been achieved by the emergence of deep learning, state-of-the art algorithms in this area have limitations. To attempt to overcome this lack of performance, we propose to use structured knowledge representations combined with state of the art deep learning techniques for visual understanding of cooking videos. Besides objects, and motions, we recognize that states of objects are also very important in interpreting the scene and therefore extensively explore the problem of states in visual cooking content. We introduce the state identification challenge in cooking applications and collect a dataset for research in the area of ingredient state analysis. We further look into the problem of simultaneous knowledge extraction from a single image and extracting information about ingredients, their states, the inter-connection between different objects in the scene and the motion-object interconnections. This problem requires an algorithm that can model the correlation of various concepts in a single image simultaneously. Using deep algorithms that can take as input multiple inputs and generate multiple outputs are fit for this problem. Therefore we propose to incorporate auto-regressive self-attention based mechanisms to extract knowledge from a single image. We show that the knowledge acquired from a single image can be used for calorie estimation. We suggest that total knowledge extraction from a single image can be used in future work for task graph inference."]},{"key":"dc:title","label":"Title","values":["Knowledge Extraction and Inference Based on Visual Understanding of Cooking Contents"]}]}],"canonical_facts":{"dc:contributor":["Yu Y. Sun, Ph.D.","Shaun Canavan, Ph.D.","Heather Culbertson, Ph.D.","John Licato, Ph.D.","Kyle Reed, Ph.D."],"dc:creator":["Babaeian Jelodar, Ahmad Babaeian"],"dc:date":["2021-07-01T07:00:00Z"],"dc:description":["In this dissertation, we discuss our work on analyzing cooking content for the ultimate goal ofautomatic robotic manipulation. For a robot to perform a cooking task, it will need to both have an understanding of the scene and utilize prior knowledge. We will explore two main sub-problems: knowledge extraction and inference, and visual understanding of the scene in this dissertation. Visual understanding of a scene, requires algorithms that can visually infer information from a single image or video. Many algorithms in the area of image classification, object detection, or activity recognition can be used in this area. Although great advances has been achieved by the emergence of deep learning, state-of-the art algorithms in this area have limitations. To attempt to overcome this lack of performance, we propose to use structured knowledge representations combined with state of the art deep learning techniques for visual understanding of cooking videos. Besides objects, and motions, we recognize that states of objects are also very important in interpreting the scene and therefore extensively explore the problem of states in visual cooking content. We introduce the state identification challenge in cooking applications and collect a dataset for research in the area of ingredient state analysis. We further look into the problem of simultaneous knowledge extraction from a single image and extracting information about ingredients, their states, the inter-connection between different objects in the scene and the motion-object interconnections. This problem requires an algorithm that can model the correlation of various concepts in a single image simultaneously. Using deep algorithms that can take as input multiple inputs and generate multiple outputs are fit for this problem. Therefore we propose to incorporate auto-regressive self-attention based mechanisms to extract knowledge from a single image. We show that the knowledge acquired from a single image can be used for calorie estimation. We suggest that total knowledge extraction from a single image can be used in future work for task graph inference."],"dc:identifier":["https://digitalcommons.usf.edu/etd/9278"],"dc:subject":["Calorie Estimation","Cooking Video Understanding","Ingredient Recognition","Knowledge Representation","Video Understanding","Deep Learning","Computer Sciences"],"dc:title":["Knowledge Extraction and Inference Based on Visual Understanding of Cooking Contents"],"thesis:degree_discipline":["Computer Science and Engineering"],"thesis:degree_level":["Dissertation","Ph.D."],"thesis:degree_name":["Doctor of Philosophy (Ph.D.)"],"thesis:institution_name":["University of South Florida"]},"updated_at":"2026-11-19T08:47:49Z"}