Georgia Institute of Technology
Towards Human-Centric Perception: Grounding Human Behavior in Multimodal Context
Abstract
dc:description.abstractPerceiving and understanding human behavior with computer vision is a core challenge for developing AI systems that can effectively interact with and assist people in everyday life. Modeling human behavior is challenging because it requires not only visually recognizing behaviors like gaze, gesture, and movement, but also grounding them in the context in which they occur. Human behavior is shaped by intent and higher-level goals, the surrounding physical environment, social interactions with other people, and additional modalities such as speech and language, making it inherently multimodal and situated. This thesis explores how to model human behavior in context by addressing three core needs: (1) datasets that capture naturalistic human interactions in everyday environments, enabling new behavior modeling tasks, (2) multimodal methods that ground behavior by leveraging information across multiple modalities including vision, audio, and language, and (3) robust methods for recognizing behavioral cues that leverage advances in foundation models to encode context. First, we present contributions to large-scale multimodal egocentric datasets that capture social interactions and human object interactions during activities. Second, we propose a modeling approach and dataset for the novel task of identifying targets of selective auditory attention during social conversations in noisy environments from egocentric video and audio. Third, we present a method for efficiently adapting vision-language retrieval models to represent new concepts and recognize them in different contexts. Fourth, we propose a framework for estimating gaze targets in scenes using the representation from a visual foundation model. Finally, we extend this framework to forecasting gaze behavior in egocentric video.
Degree
thesis:*- Name thesis:degree_name
- Computer Science, PhD
- Department thesis:degree_department
- Interactive Computing
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ryan, Fiona
- Advisors dc:contributor.advisor
-
- Hoffman, Judy
- Rehg, James M.
- Committee members dc:contributor.committeemember
-
- Hays, James
- Kira, Zsolt
- Josef Sivic
Rights
- Language dc:language.iso
- English
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/81715