Back to search

Georgia Institute of Technology

Learning Vision and Language Cues for Video Understanding in Egocentric and Instructional Videos

Abstract

dc:description.abstract

We perceive the world through a combination of senses: such as sound, smell, and vi- sion, to learn from and interact with our surroundings.Among these, vision and hearing are the primary sources of information gathering, especially through reading and listening. Ef- fectively utilizing and combining these senses is key to developing intelligent systems that can operate in and understand complex environments. A critical challenge hindering effec- tive vision-language learning is an understanding of why and how to effectively integrate language for improved video understanding. In this dissertation, we leverage the language modality to learn effective video repre- sentations across a range of tasks, including action recognition, forecasting, and summa- rization. The key ideas developed in this thesis are (i) Vision-Language supervision for action understanding, and (ii) Leveraging language for video summarization. In Vision-Language supervision for action understanding, we generate rich action de- scriptions and leverage information from multiple modalities to recognize and anticipate future actions in videos. We also discover the extent to which language contributes in un- derstanding actions in videos, through effective cross-modal supervision between the vision and language modalities. Finally in Leveraging language for video summarization, we generate text outputs for every input modality, and evaluate the performance of foundational models on video sum- marization task. By using text as the primary mode of input, we evaluate how the text representations perform on video summarization. Building on this, we propose a hierarchi- cal framework that incorporates multi-granular language cues and evaluate its effectiveness for video summarization.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Beedu, Apoorva
Advisors dc:contributor.advisor
  • Essa, Irfan
  • Romberg, Justin
Committee members dc:contributor.committeemember
  • Ploetz, Thomas
  • Hoffman, Judy
  • Heck, Larry
  • Xu, Wei

Subjects

dc:subject × 5

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/80175

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Beedu, Apoorva. Learning Vision and Language Cues for Video Understanding in Egocentric and Instructional Videos. Doctoral thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/80175