University of Glasgow
Explicit object-centric video prediction with deep learning models
Abstract
dc:description.abstractVideo prediction is a crucial task for intelligent agents such as robots and autonomous vehicles, it enables them to anticipate and act early on time-critical incidents. Many state of-the-art video prediction methods typically model the dynamics of a scene jointly and implicitly and seeing it as a single entity, without any explicit decomposition into separate objects. This is sub-optimal, as every object in a dynamic scene has their own pattern of movement, typically somewhat independent of others. Therefore, we hypothesize that explicit modelling of moving objects is crucial for video prediction in limited data and compute scenarios. We first investigate video prediction with multiple moving and interacting objects in a static camera setting within the context of a latent-transformer as the video predictor. We conduct detailed and carefully-controlled experiments on both synthetic and real-world datasets; our results show that decomposing a dynamic scene leads to higher quality predictions compared with models of a similar capacity that lack such decomposition. We then investigate the trajectory prediction of occluded objects and scenes with background motion which is a common phenomena in real-world scenarios. We introduce explicit motion information, depth map and point flow, to assist the prediction model we proposed previously. We investigate this approach in both synthetic and real-world scenarios. The experimental results shows that with the integration of explicit motion information, the predicted trajectory of dynamic objects is more accurate. We finally investigate the case of deformable objects such as scenes in garment manipulation tasks. We introduced a diffusion variant of our proposed video prediction model to better handle the motion prediction of fully deformable objects because of its continuous nature compared to transformer based architectures. By testing it on a garment manipulation dataset, we find that our diffusion-based variant outperformed our transformer-based models. Our findings suggest that for video prediction models to accurately model motion patterns inside a dynamic scene, scaling up holistic models are inefficient and recourse consuming. In contrast, decomposition of objects and modeling with their explicit motion information can be a better and more efficient alternative compared to monolithic models with same capacity. Furthermore, this setting implies that it can be more useful in closed-world settings like robotic manipulation tasks where limited objects are in the scene.
Degree
thesis:*- Level dc:type.qualificationlevel
- PhD
- Grantor dc:publisher.institution
- University of Glasgow
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Sulaiman, Yiliyasi
Subjects
dc:subject × 1Rights
- Language dc:language
- en
Identifiers
dc:identifier.*- DOI dc:identifier.doi
- https://doi.org/10.5525/gla.thesis.85705