University of North Texas
Probabilistic Temporal Correspondence Learning for Self-Supervised Video Alignment
Abstract
dc:descriptionVideo alignment, the task of establishing temporal correspondences between frames of videos that depict the same activity, is a fundamental building block for applications ranging from action recognition to rehabilitation monitoring and robotic imitation learning. Self-supervised methods based on temporal cycle-consistency make fine-grained alignment possible without frame-level annotations, but they rest on assumptions that break down in realistic settings: they presume a single one-to-one correspondence for every frame, they assume that every frame is alignable, and they rely on local features that cannot disambiguate visually identical phases of repetitive actions. This dissertation introduces Probabilistic Temporal Cycle Consistency (PTCC), a self-supervised framework that addresses these limitations. First, PTCC replaces the unimodal softmax of standard cycle-consistency with a Gaussian mixture model, representing frame correspondences as a multi-modal distribution so that one frame can match every valid repetition of an action. Second, a parameter-free bidirectional verification mechanism identifies and down-weights non-alignable outlier frames, such as background or idle segments, without additional supervision. Third, a Transformer-based temporal encoder gives each frame a global temporal receptive field, allowing the model to distinguish repetitions that are separated in time yet visually similar. We evaluate PTCC on the Penn Action and Multiview Pouring datasets using six complementary metrics. PTCC establishes a new state of the art on both, achieving 82.65% phase alignment accuracy and a Kendall's Tau of 0.99 on Penn Action and reducing the cycle consistency error by over 97% relative to standard temporal cycle-consistency. Ablation and qualitative studies confirm the contribution of each component.
Degree
thesis:*- Grantor dc:publisher
- University of North Texas
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Liu, Siyuan 1984-
- Contributors dc:contributor
-
- Yang, Qing
- Guo, Xuan
- Yuan, Xiaohui
- Morozov, Kirill
Subjects
dc:subject × 3Rights
dc:rights- Statement dc:rights
-
- Public
- Liu, Siyuan 1984-
- Copyright
- Copyright is held by the author, unless otherwise noted. All rights Reserved.
- Language dc:language
- English
Identifiers
dc:identifier.*- Identifier
-
local-cont-no: submission_5065
ark: ark:/67531/metadc2652554