Abstract
dc:description.abstractResearch in face analysis and generation has made substantial progress, enabling applications in content generation, human behavior understanding, and human-computer interaction. However, significant challenges remain, including: (1) persistent racial and gender biases in both discriminative and generative face models due to imbalanced datasets; (2) limited fine-grained expressivity in face generation, particularly in producing stylized faces with nuanced facial expressions; and (3) insufficient integration of facial affective and multimodal behavioral signals for effective prediction of human behavior, especially in educational contexts. This dissertation addresses these gaps by introducing novel datasets and frameworks. First, we develop KenyanFace, a carefully curated face dataset designed to address racial and skin color biases inherent in existing face datasets and models. Next, we enhance diffusion-based text-to-image (T2I) generative frameworks to produce stylized cartoon faces with nuanced, fine-grained facial expressions. Furthermore, we present a series of datasets and models for effectively integrating facial cues with multimodal behavioral signals to predict complex human behaviors such as student engagement and learning outcomes. We collect and annotate datasets comprising facial videos and learning log data recorded during student interactions with an intelligent tutoring system. Leveraging these datasets, we propose several novel affective models that accurately predict student engagement and learning outcomes. Specifically, we introduce models utilizing (1) affective transfer learning methods to effectively incorporate facial affective features, and (2) multimodal approaches that integrate both facial features and learning log data. Building upon these predictive models, we further develop algorithms for personalized selection of educational content tailored to students' real-time emotional and cognitive states, thereby enhancing adaptive interventions and improving learning outcomes.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Yu, Hao
- Advisor dc:contributor.advisor
-
- Betke, Margrit
- ORCID dc:identifier.orcid
Subjects
dc:subject × 1Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53394