Back to search

Boston University

Toward learning representations for movements

Abstract

dc:description.abstract

This dissertation presents a unified exploration of representation learning from 3D pose and motion data, across three different learning paradigms: supervised, contrastive, and self-supervised. The central theme is to develop models that are robust, generalizable, and interpretable across sparse and dense motion signals, with applications spanning animal behavior analysis and human rehabilitation. We begin with OptiPose, a supervised model that leverages the self-attention mechanism to refine noisy 3D pose sequences. By learning spatial and temporal constraints over joint trajectories, OptiPose treats each pose as a token and adjusts pose sequences to produce smooth and anatomically consistent movements. To support and evaluate this model, we introduce Rodent3D, a multimodal video dataset captured with synchronized thermal and RGB-D cameras. Rodent3D provides annotated 3D keypoints and serves as a new benchmark for 3D animal pose estimation and refinement. Next, we present ExeChecker, a contrastive learning framework for evaluating human rehabilitation exercises. Trained on triplets of correct and incorrect movements, ExeChecker learns discriminative embeddings that highlight joints responsible for viierrors. This approach enables interpretable, joint-level feedback for exercise assessment. The accompanying ExeCheck dataset, developed in collaboration with physical therapists, contains paired RGB-D recordings of common exercises executed correctly and incorrectly, and therefore enables practical evaluation in physiotherapy scenarios. Finally, we propose MAE-JEPA, a hybrid self-supervised framework that combines masked autoencoding (MAE) with joint embedding prediction architectures (JEPA). While JEPA have demonstrated strong performance on dense video data, they tend to underperform on sparse skeletal data. By integrating coordinate reconstruction and latent-space prediction– weighted by learned uncertainty– MAE-JEPA adapts its learning dynamics to data density, improving representation quality without the need for hand-crafted supervision or augmentation schemes. Together, these three studies form a cohesive effort to model and understand 3D movement under varying supervision levels and data characteristics, offering insight into how learning objectives, architectures, and masking strategies shape pose representation quality and advancing practical tools for behavior science, clinical rehabilitation, and beyond.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Gu, Yiwen
Advisors dc:contributor.advisor
  • Betke, Margrit
  • Plummer, Bryan
ORCID dc:identifier.orcid

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53364

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Gu, Yiwen. Toward learning representations for movements. doctoral thesis, Boston University, 2025. https://hdl.handle.net/2144/53364