Abstract
dc:description.abstractThis dissertation presents a unified exploration of representation learning from 3D pose and motion data, across three different learning paradigms: supervised, contrastive, and self-supervised. The central theme is to develop models that are robust, generalizable, and interpretable across sparse and dense motion signals, with applications spanning animal behavior analysis and human rehabilitation. We begin with OptiPose, a supervised model that leverages the self-attention mechanism to refine noisy 3D pose sequences. By learning spatial and temporal constraints over joint trajectories, OptiPose treats each pose as a token and adjusts pose sequences to produce smooth and anatomically consistent movements. To support and evaluate this model, we introduce Rodent3D, a multimodal video dataset captured with synchronized thermal and RGB-D cameras. Rodent3D provides annotated 3D keypoints and serves as a new benchmark for 3D animal pose estimation and refinement. Next, we present ExeChecker, a contrastive learning framework for evaluating human rehabilitation exercises. Trained on triplets of correct and incorrect movements, ExeChecker learns discriminative embeddings that highlight joints responsible for viierrors. This approach enables interpretable, joint-level feedback for exercise assessment. The accompanying ExeCheck dataset, developed in collaboration with physical therapists, contains paired RGB-D recordings of common exercises executed correctly and incorrectly, and therefore enables practical evaluation in physiotherapy scenarios. Finally, we propose MAE-JEPA, a hybrid self-supervised framework that combines masked autoencoding (MAE) with joint embedding prediction architectures (JEPA). While JEPA have demonstrated strong performance on dense video data, they tend to underperform on sparse skeletal data. By integrating coordinate reconstruction and latent-space prediction– weighted by learned uncertainty– MAE-JEPA adapts its learning dynamics to data density, improving representation quality without the need for hand-crafted supervision or augmentation schemes. Together, these three studies form a cohesive effort to model and understand 3D movement under varying supervision levels and data characteristics, offering insight into how learning objectives, architectures, and masking strategies shape pose representation quality and advancing practical tools for behavior science, clinical rehabilitation, and beyond.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Gu, Yiwen
- Advisors dc:contributor.advisor
-
- Betke, Margrit
- Plummer, Bryan
- ORCID dc:identifier.orcid
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53364