Georgia Institute of Technology
Improving Out-of-Distribution Generalization in Human-Centric Multimodal Vision
Abstract
dc:description.abstractProgress in human-centric multimodal vision has been rapid on in-distribution benchmarks, yet the strongest off-the-shelf systems continue to fail once the test corpus differs from the training corpus. Cross-dataset error roughly doubles for the strongest 2D-to-3D lifters; retrieval accuracy falls toward single digits when test captions come from a different corpus; and segmenters trained on a billion masks still fail to disambiguate a click when a single word could. None of the conventional model-side levers (more labels, more capacity, more augmentation, stronger domain adaptation) closes this residual gap on its own. Read structurally, the residual gap reflects an information-availability limitation rather than a modeling one: in-distribution, distributional priors substitute for information the input does not carry and for consistency the labels do not have, so in-distribution benchmarks make ill-posed tasks appear well-posed. Under distribution shift these priors no longer transfer, and the residual gap becomes a diagnostic: its response to scaling identifies whether the input underdetermines the target or the supervision carries variation the input cannot verify. This dissertation identifies and addresses these two failures. The signal-side failure is addressed by DAISeg, which adds the semantic content missing from click-conditioned segmentation and gains 3–4.5 mIoU on novel classes, and by AugLift, which adds a compact ordinal-depth prior to 2D-to-3D pose lifting and reduces cross-dataset error by 10.1% averaged across four architectures. The noise-side failure is addressed by IPC-Edit, which filters and composes noisy public supervision channels and reaches 68.5% identity preservation at roughly 10× less paired data than closed-corpus formulations, and by MoCHA, which removes annotator-specific caption variation the input cannot verify and more than doubles cross-dataset retrieval R@1 (9.41 → 25.95). Two structural properties recur across the four projects. First, each fix is additive with model-side scaling: it modifies what the model observes while scaling modifies the model itself, so the two compose rather than compete, and the same reasoning predicts stacking with model-side improvements that do not yet exist. Second, foundation models are used as offline label sources rather than runtime dependencies: MoCHA's LLM canonicalizer distills into a compact student that matches its retrieval performance head-to-head. The four fixes are consistently label- or data-efficient in their settings: they add no new per-class supervision or 3D pose capture, and where paired supervision is used they match cross-dataset baselines with substantially less of it.
Degree
thesis:*- Name thesis:degree_name
- Robotics, PhD
- Department thesis:degree_department
- Mechanical Engineering
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Warner, Nikolai
- Advisor dc:contributor.advisor
-
- Essa, Irfan
- Committee members dc:contributor.committeemember
-
- Kira, Zsolt
- Sadhwani, Apaar
- Hoffman, Judy
- Ploetz, Thomas
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/82205