Back to search

Georgia Institute of Technology

Towards Human-Centric Perception: Grounding Human Behavior in Multimodal Context

Abstract

dc:description.abstract

Perceiving and understanding human behavior with computer vision is a core challenge for developing AI systems that can effectively interact with and assist people in everyday life. Modeling human behavior is challenging because it requires not only visually recognizing behaviors like gaze, gesture, and movement, but also grounding them in the context in which they occur. Human behavior is shaped by intent and higher-level goals, the surrounding physical environment, social interactions with other people, and additional modalities such as speech and language, making it inherently multimodal and situated. This thesis explores how to model human behavior in context by addressing three core needs: (1) datasets that capture naturalistic human interactions in everyday environments, enabling new behavior modeling tasks, (2) multimodal methods that ground behavior by leveraging information across multiple modalities including vision, audio, and language, and (3) robust methods for recognizing behavioral cues that leverage advances in foundation models to encode context. First, we present contributions to large-scale multimodal egocentric datasets that capture social interactions and human object interactions during activities. Second, we propose a modeling approach and dataset for the novel task of identifying targets of selective auditory attention during social conversations in noisy environments from egocentric video and audio. Third, we present a method for efficiently adapting vision-language retrieval models to represent new concepts and recognize them in different contexts. Fourth, we propose a framework for estimating gaze targets in scenes using the representation from a visual foundation model. Finally, we extend this framework to forecasting gaze behavior in egocentric video.

Degree

thesis:*
Name thesis:degree_name
Computer Science, PhD
Department thesis:degree_department
Interactive Computing
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ryan, Fiona
Advisors dc:contributor.advisor
  • Hoffman, Judy
  • Rehg, James M.
Committee members dc:contributor.committeemember
  • Hays, James
  • Kira, Zsolt
  • Josef Sivic

Rights

Language dc:language.iso
English

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81715

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Ryan, Fiona. Towards Human-Centric Perception: Grounding Human Behavior in Multimodal Context. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/81715