Back to search

University of Thessaly (UTH)

A deep learning approach to 2D/3D object affordance understanding

Abstract

dc:description

A long-standing challenge in the computer vision field is to recognize the perceived objects and leverage their rich visual information. In fact, objects constitute key elements for a wide variety of real-world applications; from scene understanding and industry automation to security and robotics. Significant steps have been made towards 2D/3D object detection and recognition over the last few years that were complemented by the rapid advancements in processing units technology. However, robust object understanding remains an open challenge since recent works focus mostly on the appearance attributes of the objects, such as shape and texture, and omit any information about their functionalities. In this dissertation we develop models and techniques that allow us to understand and exploit these functionalities, also known as object “affordances”, i.e. the set of actions that humans can perform while interacting with the object. In particular, first we investigate the impact of object affordances to RGB-D object recognition through the “function from motion” perspective, where the affordance information is extracted by observing human-object interactions. Motivated by the research findings of cognitive neuroscience, we are the first to apply the so-called “sensorimotor” learning theory in computer vision, using end-to-end deep neural networks to fuse the object appearance (sensory) and affordance (motor) information and improving object recognition in RGB-D videos. Second, we develop an encoder-decoder model that is able to localize and segment the object parts that support specific human-object interactions. Rather than relying on object-specific information, such as bounding boxes and class labels, our model is able to learn to focus in the interaction spot through processing spatio-temporal information and predict affordance segmentation masks both in RGB-D videos and static images. Lastly, we introduce SOR3D, the first large-scale RGB-D dataset that consists of human-object interaction sequences and facilitates affordance-related research. The corpus is publicly available and includes various object and action related annotations, ranging from video-level object and action class labels to frame-level affordance heatmaps and segmentation masks. Extensive experiments on the introduced SOR3D dataset demonstrate the efficacy of the proposed models in the aforementioned tasks. From the presented results, we observe that: a) the utilization of object affordance information leads to improved object recognition, and b) object affordance localization and segmentation in videos and static images can be achieved without the need for extra object-related information, such as object class and location.

Degree

thesis:*
Grantor dc:publisher
University of Thessaly (UTH)
Year dc:date
2020

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Thermos, Spyridon
  • Θερμός, Σπυρίδων

Subjects

dc:subject × 12

Rights

dc:rights
Statement dc:rights
  • BY_NC_ND
Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/47421

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Thermos, Spyridon; Θερμός, Σπυρίδων. A deep learning approach to 2D/3D object affordance understanding. University of Thessaly (UTH), 2020. http://hdl.handle.net/10442/hedi/47421