Back to search

Cornell University

Learning to Represent and Recognize Multimodal Videos

Abstract

dc:description.abstract

In today's digital landscape, the staggering growth of video resources has resulted in a wealth of visual, auditory, and textual information readily available on the internet. To fully harness the potential of learning from multimodal videos, it is crucial to develop efficient techniques for processing and analyzing this information. This dissertation delves into two primary areas: label-efficient representation learning from videos and multimodal video recognition. We leverage the advances in large-scale deep learning to fully exploit the potential of internet videos. Label-efficient representation learning from videos is motivated by the fact that internet videos often lack high-quality labels. We first propose a contrastive learning-based framework to learn features in an unsupervised manner. This approach pulls together feature representations from the same video and pushes apart those from different videos. We then investigate how to learn more refined temporal features for time-sensitive tasks, such as temporal event classification and detection, and more precise spatial features for location-aware tasks, including tracking and detection. Lastly, we explore the use of stronger transformer backbones to integrate multimodal data and form a unified, robust representation. Regarding multimodal recognition for videos, we initially examine the challenging case of fine-grained video recognition to determine if different modalities can assist each other in ambiguous scenarios. We create a new expert-curated audiovisual benchmark and discover that multimodal fusion significantly benefits performance. We subsequently investigate open-vocabulary multimodal video recognition and propose a framework that can effectively classify videos from any given category.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Qian, Rui
Chair dc:contributor.chair
  • Belongie, Serge
Committee members dc:contributor.committeemember
  • Hariharan, Bharath
  • Lee, Clarence

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16219401
Dc Identifier Other
ProQuest Submission ID: 13689
ProQuest Publication ID: 30491677

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Qian, Rui. Learning to Represent and Recognize Multimodal Videos. Doctor of Philosophy thesis, Cornell University, 2023. https://hdl.handle.net/1813/114738