University of Crete (UOC)
Human action prediction and forecasting based on visual data
Abstract
dc:descriptionThe ability to observe human movements and predict their actions is a developmental skill acquired by humans early in life. When witnessing a person performing a task, we can easily forecast their subsequent actions based on contextual cues and past experiences. In this work, we aim at developing such abilities for machines, focusing on the tasks of vision-based action prediction, action anticipation and next-active-object prediction. Action prediction is defined as the inference of an action label while the action is still ongoing. Such a capability is useful for early response and further action planning. We consider the problem of action prediction in scenarios involving humans interacting with objects. We formulate an approach that builds time series representations of the performance of the humans and the objects. Such a representation of an ongoing action is then compared to prototype actions. This is achieved by a Dynamic Time Warping (DTW)-based time series alignment framework which identifies the best match between the ongoing action and the prototype ones. We predict actions in trimmed and untrimmed action sequences with the use of the DTW algorithm. In the same vein, for the prediction of actions we propose two new alignment algorithms called OBE-S-DTW and OE-S-DTW that show superior results on the task of action prediction compared to DTW. Following, we propose a graph-based methodology for the visual prediction of human-object interactions in videos. Rather than forecasting the human and object motion, we aim at predicting (a) the class of the on-going human-object interaction and (b) the class(es) of the next active object(s) (NAOs), i.e., the object(s) that will be involved in the interaction in the near future as well as the time the interaction will occur. Finally, we address the problem of action anticipation by taking into consideration the history of all executed actions throughout long, procedural activities. A novel approach noted as Visual-Linguistic Modeling of Action History (VLMAH) is proposed that fuses the immediate past in the form of visual features as well as the distant past based on a cost-effective form of linguistic constructs (semantic labels of the nouns, verbs, or actions). Our approach generates accurate near-future action predictions during procedural activities by leveraging information on the long- and short-term past. The proposed methods constitute solutions for the problems of action prediction and anticipation and next-active-object prediction. The aforementioned methodologies have been evaluated on challenging datasets and showcase results superior to the current state-of-art.
Degree
thesis:*- Grantor dc:publisher
- University of Crete (UOC)
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Manousaki, Victoria
- Μανουσάκη, Βικτωρία
Subjects
dc:subject × 21- Πρόβλεψη ανθρώπινων δραστηριοτήτων
- Πρόγνωση ανθρώπινων δραστηριοτήτων
- Πρόβλεψη επόμενου ενεργού αντικειμένου
- Ταίριασμα γράφων
- Νευρωνικά δίκτυα
- Ευθυγράμμιση χρονολογικών σειρών
- Αλληλεπίδραση ανθρώπου-αντικειμένου
- Action prediction
- Action forecasting
- Activity forecasting
- Next-active object prediction
- Graph matching
- Dynamic time warping
- Deep neural networks
- Human-object interaction
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Τεχνητή νοημοσύνη
- Computer and Information Sciences
- Natural Sciences
- Artificial Intelligence
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/56106