Back to search

Boston University

Toward equitable and affect-enhanced face analysis and generation

Abstract

dc:description.abstract

Research in face analysis and generation has made substantial progress, enabling applications in content generation, human behavior understanding, and human-computer interaction. However, significant challenges remain, including: (1) persistent racial and gender biases in both discriminative and generative face models due to imbalanced datasets; (2) limited fine-grained expressivity in face generation, particularly in producing stylized faces with nuanced facial expressions; and (3) insufficient integration of facial affective and multimodal behavioral signals for effective prediction of human behavior, especially in educational contexts. This dissertation addresses these gaps by introducing novel datasets and frameworks. First, we develop KenyanFace, a carefully curated face dataset designed to address racial and skin color biases inherent in existing face datasets and models. Next, we enhance diffusion-based text-to-image (T2I) generative frameworks to produce stylized cartoon faces with nuanced, fine-grained facial expressions. Furthermore, we present a series of datasets and models for effectively integrating facial cues with multimodal behavioral signals to predict complex human behaviors such as student engagement and learning outcomes. We collect and annotate datasets comprising facial videos and learning log data recorded during student interactions with an intelligent tutoring system. Leveraging these datasets, we propose several novel affective models that accurately predict student engagement and learning outcomes. Specifically, we introduce models utilizing (1) affective transfer learning methods to effectively incorporate facial affective features, and (2) multimodal approaches that integrate both facial features and learning log data. Building upon these predictive models, we further develop algorithms for personalized selection of educational content tailored to students' real-time emotional and cognitive states, thereby enhancing adaptive interventions and improving learning outcomes.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yu, Hao
Advisor dc:contributor.advisor
  • Betke, Margrit
ORCID dc:identifier.orcid

Subjects

dc:subject × 1

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53394

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Yu, Hao. Toward equitable and affect-enhanced face analysis and generation. doctoral thesis, Boston University, 2025. https://hdl.handle.net/2144/53394