Abstract
dc:description.abstractThe sub-field of multimodal learning in artificial intelligence involves the problem of jointly learning representations of the audio, language and visual modalities for the purpose of automated applications including virtual assistants, chatbots and robotics. In particular, learning a multimodal representation that facilitates grounding of language in multiple modalities is of paramount importance in AI applications such as video captioning and embodied navigation. Similar to how humans communicate using natural language, endowing machines with the ability to perform grounded language reasoning can help them to interact more effectively with diverse and complex real-world scenarios. In light of this, this thesis addresses open challenges in language grounding across three different axes, such as learning effective multimodal representations from noisy data as well as pretrained unimodal representations, spatiotemporal grounding in videos and grounding language in a new modality that is unseen during training. An abstract formulation of grounding language in other modalities involves encoding data samples from different modalities into a common latent space. Along the first theme of grounding language in images, multimodal approaches often assume that the language modality has a literal relation to other modalities but this is not realistic in domains such as news and Wikipedia. To address this, we introduce an approach that uses a pretrained vision-language model (VLM) to associate the illustrative relationships of multiple complementary images with different parts of long text sequences. In domains where the language modality has a literal relation to data from other modalities, conventional approaches often learn multimodal representations by learning transformation functions to project pretrained unimodal representations but they fail to encode fine-grained context between different modalities. As such, we introduce our work along the second theme of grounding language in videos and audios. We propose a novel language-conditioned visual graph to hierarchically fusecontextual information between video and language representations to address the task of weakly-supervised video moment retrieval. While the abovementioned approaches are effective, their learnt multimodal representations are often global in nature, where each data sample is encoded as a single vector. Representing each video as a single vector makes it challenging to ground language spatiotemporally within a video clip. In light of this, we propose two approaches to learn multimodal representations for grounding language spatiotemporally not only in videos but their corresponding audios as well. Additionally, the problem of encoding a video as a single vector is compounded as the length of the video increases. This results in a loss of fine-grained visual information that is important for grounding language effectively. Thus, we adapt pretrained multimodal-Large Language models (LLMs) to ground language in long temporal and visual information holistically for long-form video question answering. Finally, we briefly transition from learning multimodal representations that ground language in static visual scenes to the third theme of active perception to identify interesting directions for future work. The open-ended generative capability of multimodal-LLMs provides a promising avenue for grounding language in dynamic and evolving scenes. Existing multimodal approaches that ground language in action policies for interacting with their environment often learn these policies by relying on fine-grained step-by-step annotations for training supervision, which limits their generalizability to unseen tasks and environments. Given the demonstrated effectiveness of in-context learning, we begin by studying the effectiveness of using pretrained LLMs for high-level task planning under the few-shot setting with a few examples from the train set. Based on the preliminary results as well as the observations in our previous work, we wrap up this thesis by drawing insights to lay the grounds for future directions in research on multimodal learning and video understanding.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tan, Reuben
- Advisors dc:contributor.advisor
-
- Saenko, Kate
- Plummer, Bryan A.
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53119