Back to search

Boston University

Grounding language in image, video and audio modalities

Abstract

dc:description.abstract

The sub-field of multimodal learning in artificial intelligence involves the problem of jointly learning representations of the audio, language and visual modalities for the purpose of automated applications including virtual assistants, chatbots and robotics. In particular, learning a multimodal representation that facilitates grounding of language in multiple modalities is of paramount importance in AI applications such as video captioning and embodied navigation. Similar to how humans communicate using natural language, endowing machines with the ability to perform grounded language reasoning can help them to interact more effectively with diverse and complex real-world scenarios. In light of this, this thesis addresses open challenges in language grounding across three different axes, such as learning effective multimodal representations from noisy data as well as pretrained unimodal representations, spatiotemporal grounding in videos and grounding language in a new modality that is unseen during training. An abstract formulation of grounding language in other modalities involves encoding data samples from different modalities into a common latent space. Along the first theme of grounding language in images, multimodal approaches often assume that the language modality has a literal relation to other modalities but this is not realistic in domains such as news and Wikipedia. To address this, we introduce an approach that uses a pretrained vision-language model (VLM) to associate the illustrative relationships of multiple complementary images with different parts of long text sequences. In domains where the language modality has a literal relation to data from other modalities, conventional approaches often learn multimodal representations by learning transformation functions to project pretrained unimodal representations but they fail to encode fine-grained context between different modalities. As such, we introduce our work along the second theme of grounding language in videos and audios. We propose a novel language-conditioned visual graph to hierarchically fusecontextual information between video and language representations to address the task of weakly-supervised video moment retrieval. While the abovementioned approaches are effective, their learnt multimodal representations are often global in nature, where each data sample is encoded as a single vector. Representing each video as a single vector makes it challenging to ground language spatiotemporally within a video clip. In light of this, we propose two approaches to learn multimodal representations for grounding language spatiotemporally not only in videos but their corresponding audios as well. Additionally, the problem of encoding a video as a single vector is compounded as the length of the video increases. This results in a loss of fine-grained visual information that is important for grounding language effectively. Thus, we adapt pretrained multimodal-Large Language models (LLMs) to ground language in long temporal and visual information holistically for long-form video question answering. Finally, we briefly transition from learning multimodal representations that ground language in static visual scenes to the third theme of active perception to identify interesting directions for future work. The open-ended generative capability of multimodal-LLMs provides a promising avenue for grounding language in dynamic and evolving scenes. Existing multimodal approaches that ground language in action policies for interacting with their environment often learn these policies by relying on fine-grained step-by-step annotations for training supervision, which limits their generalizability to unseen tasks and environments. Given the demonstrated effectiveness of in-context learning, we begin by studying the effectiveness of using pretrained LLMs for high-level task planning under the few-shot setting with a few examples from the train set. Based on the preliminary results as well as the observations in our previous work, we wrap up this thesis by drawing insights to lay the grounds for future directions in research on multimodal learning and video understanding.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tan, Reuben
Advisors dc:contributor.advisor
  • Saenko, Kate
  • Plummer, Bryan A.

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53119

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Tan, Reuben. Grounding language in image, video and audio modalities. doctoral thesis, Boston University, 2024. https://hdl.handle.net/2144/53119