Back to search

West Virginia University

Multimodal Representation Learning for Face Understanding: From Caption Supervision to Foundation Model Adaptation

Abstract

dc:description.abstract

The rapid advancement of intelligent surveillance systems and the increasing demand for reliable biometric identification in border security, public safety, and digital forensics require robust face understanding under unconstrained conditions, including low resolution, pose variation, and occlusion. While Vision Transformer (ViT)-based foundation models have greatly improved visual representation learning, their patch-based tokenization and lack of spatial inductive bias limit their ability to capture fine-grained details in low-resolution inputs. This dissertation investigates multimodal representation learning for face understanding through natural language supervision, large-scale face-caption pre-training, and parameter-efficient foundation model adaptation. It hypothesizes that textual supervision provides complementary semantic cues that improve the robustness, interpretability, and generalizability of facial representations across diverse downstream tasks. To achieve this goal, the dissertation introduces a unified set of multimodal learning frameworks that address several fundamental challenges, including modality heterogeneity and cross-modal alignment, the scarcity and noise of face-caption data, generic and non-discriminative caption generation, and reward hacking in reinforcement learning-based optimization. First, a caption-guided face recognition framework is proposed that utilizes natural language supervision to improve recognition performance in low-resolution surveillance scenarios. Second, a face-caption pre-training framework is developed to learn universal facial representations with strong transferability across both cross-modal understanding and face analysis tasks. Third, an attribute-driven face captioning framework is introduced with a novel reward optimization strategy that generates identity-discriminative descriptions, substantially improving text-based face retrieval. Collectively, these contributions establish multimodal representation learning through caption supervision as a scalable and generalizable framework for face understanding. Extensive experiments on multiple face verification, retrieval, and captioning benchmarks, substantiate these contributions: the caption-guided recognition framework improves 1:1 verification rate by up to 1.8 percentage points (TAR@FAR=10−6) over state-of-the-art text-guided face recognition methods; the face-caption pre-training framework improves zero-shot text-to-face retrieval (R@5) by up to 2.1 percentage points over strong facial representation learning baselines while reaching 92.12% facial attribute recognition accuracy on CelebA; and the attribute-driven captioning framework improves zero-shot text-to-face self-retrieval (R@10) by 5.0–15.8 percentage points over the strongest reward-based baseline across three out-of-domain benchmarks.

Degree

thesis:*
Name thesis:degree_name
PhD
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Lane Department of Computer Science and Electrical Engineering

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hasan, Md Mahedi
Contributors dc:contributor
  • Jeremy Dawson
  • Matthew Valenti
  • Gianfranco Doretto
  • Prashnna Gyawali
  • Jacqueline Speir

Subjects

dc:subject × 8

Identifiers

dc:identifier.*
Identifier
<p><a href="https://orcid.org/0000-0002-6337-3571">https://orcid.org/0000-0002-6337-3571</a></p>
https://researchrepository.wvu.edu/etd/13478

Chain of custody

source
Harvested from
West Virginia University
Base URL
researchrepository.wvu.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Hasan, Md Mahedi. Multimodal Representation Learning for Face Understanding: From Caption Supervision to Foundation Model Adaptation. Dissertation thesis, https://doi.org/10.33915/etd.13478