University of British Columbia
M-NeRF : model-based human reconstruction from scratch with mirror-aware neural radiance fields
Abstract
dc:descriptionHuman motion capture either requires multi-camera systems or is unreliable using single-view input due to depth ambiguities. Meanwhile, mirrors are readily available in urban environments and can take the role of additional views. When picturing a person in front of a mirror, the mirror image provides a second view of the person using only a single camera. Prior work has hence exploited this additional constraint to improve 3D human pose reconstruction. Going beyond existing mirror approaches, we utilize mirrors for learning a complete body model, including shape and appearance. Our main contribution is extending articulated neural radiance fields (NERFs) to include a notion of a mirror and making it sample-efficient. We integrate this into an entire system that succeeds without any 3D annotation by automatically calibrating the camera, estimating mirror orientation, and subsequently lifting 2D keypoint detections to 3D skeleton pose that is used to condition the mirror-aware NeRF. We empirically demonstrate the benefit of learning a body model and accounting for mirror-occlusion in challenging mirror scene setups. We show continuous improvements on time-varying articulated 3D joint estimation, reconstruct the body geometry from only mirror images and 2D detections, and synthesize novel views from unobserved viewpoints.
Degree
thesis:*- Name thesis:degree_name
- Master of Science - MSc
- Level thesis:degree_level
- master's
- Discipline thesis:degree_discipline
- Computer Science
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ajisafe, Daniel Abidemi
- Contributors dc:contributor
-
- Rhodin, Helge
Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/83704