Back to search

University of Crete (UOC)

Achieving total 3D human capture with MocapNETs

Abstract

dc:description

The goal of the presented thesis was to investigate and develop a novel, fast, portable, robust and accurate plug and play 3D Human Capture module that receives RGB images captured in-the-wild and regresses the 3D body configuration of any depicted person in the scene. The proposed architecture was built from scratch using first principles and taking advantage of recent advancements in Neural Networks, taking its final form as an ensemble of neural networks. We identified and bridged gaps between state-of-art deep learning methods and well-established model-based vision methodologies predating CNNs. Its name, “MocapNET” was coined to concisely describe it as it became the first neural network-based method in the literature to directly regress Motion Capture (Mocap) output in an end-to-end fashion. To improve accuracy and address personalization aspects, a novel real-time generative optimization algorithm was also developed named “Hierarchical Coordinate Descent” and tailored to the conditionally independent encoders of the MocapNET ensemble complementing their output. The ambition and scope of the retrieved 3D output gradually broadened as the method successfully generalized to more articulated structures during the course of its development. The total 3D capture solution presented includes upper body, lower body, hands, face and gaze. With the term 3D Human Capture we refer not only to positions in a 3D space but rather, the full kinematic solution of the skeleton. The method performs in real-time and its output is natively compatible with 3D editing software due to its BVH container. This makes it globally unique and among a select very few methods that can successfully tackle all these sub-problems that traditionally were sub-fields of the broader computer vision research. The 3D human pose estimation solution developed can be used in devices such as mobile phones, AR/VR headsets, self-driving cars, smart devices, home and factory robots etc, endowing them with capabilities to perceive, compare and enumerate human body poses, which would ultimately facilitate understanding of human behavior. The thesis attempts to carefully document all the aspects of the method including 2D shape descriptors, NN design, PCA compression to allow usage on mobile devices and the various attempts that shaped the method to its final version.

Degree

thesis:*
Grantor dc:publisher
University of Crete (UOC)
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Qammaz, Ammar
  • Γκαμάζ, Αμμάρ

Subjects

dc:subject × 17

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/56972

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Qammaz, Ammar; Γκαμάζ, Αμμάρ. Achieving total 3D human capture with MocapNETs. University of Crete (UOC), 2024. http://hdl.handle.net/10442/hedi/56972