Back to search

University of South Florida

Knowledge Extraction and Inference Based on Visual Understanding of Cooking Contents

Abstract

dc:description

In this dissertation, we discuss our work on analyzing cooking content for the ultimate goal ofautomatic robotic manipulation. For a robot to perform a cooking task, it will need to both have an understanding of the scene and utilize prior knowledge. We will explore two main sub-problems: knowledge extraction and inference, and visual understanding of the scene in this dissertation. Visual understanding of a scene, requires algorithms that can visually infer information from a single image or video. Many algorithms in the area of image classification, object detection, or activity recognition can be used in this area. Although great advances has been achieved by the emergence of deep learning, state-of-the art algorithms in this area have limitations. To attempt to overcome this lack of performance, we propose to use structured knowledge representations combined with state of the art deep learning techniques for visual understanding of cooking videos. Besides objects, and motions, we recognize that states of objects are also very important in interpreting the scene and therefore extensively explore the problem of states in visual cooking content. We introduce the state identification challenge in cooking applications and collect a dataset for research in the area of ingredient state analysis. We further look into the problem of simultaneous knowledge extraction from a single image and extracting information about ingredients, their states, the inter-connection between different objects in the scene and the motion-object interconnections. This problem requires an algorithm that can model the correlation of various concepts in a single image simultaneously. Using deep algorithms that can take as input multiple inputs and generate multiple outputs are fit for this problem. Therefore we propose to incorporate auto-regressive self-attention based mechanisms to extract knowledge from a single image. We show that the knowledge acquired from a single image can be used for calorie estimation. We suggest that total knowledge extraction from a single image can be used in future work for task graph inference.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy (Ph.D.)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Computer Science and Engineering
Grantor
University of South Florida
Year dc:date
2021

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Babaeian Jelodar, Ahmad Babaeian
Contributors dc:contributor
  • Yu Y. Sun, Ph.D.
  • Shaun Canavan, Ph.D.
  • Heather Culbertson, Ph.D.
  • John Licato, Ph.D.
  • Kyle Reed, Ph.D.

Subjects

dc:subject × 7

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.usf.edu/etd/9278

Chain of custody

source
Harvested from
University of South Florida
Base URL
digitalcommons.usf.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Babaeian Jelodar, Ahmad Babaeian. Knowledge Extraction and Inference Based on Visual Understanding of Cooking Contents. Dissertation thesis, University of South Florida, 2021. https://digitalcommons.usf.edu/etd/9278