Georgia Institute of Technology
Audio-Visual Scene-Aware Dialog: A Step Towards Multimodal Conversational Agents
Abstract
dc:description.abstractDeveloping conversational agents has long been a goal in artificial intelligence, with extensive applications in human-computer interaction and virtual assistance. In an effort to advance natural conversation with these agents, this dissertation introduces the Audio Visual Scene-Aware Dialog task. This task involves generating coherent responses to questions about a dynamic scene, leveraging video and audio inputs alongside the dialog history. To facilitate benchmarking of this task, the Audio Visual Scene-Aware Dialog (AVSD) dataset is introduced. AVSD is a challenging task in visual-and-language joint representation learning. It has gained significant attention in recent years, alongside other tasks such as text-based video retrieval and video summarization, with notable progress driven by transformer-based language encoders. However, existing approaches often underutilize the visual features, leading to biases towards textual information. To address this limitation, this dissertation introduces a novel framework for training the multimodal networks, integrating 3D-CNN network and transformer-based networks to enhance the extraction of semantic representations from videos and improve performance in both generative and retrieval tasks. Finally, this dissertation leverages recent advancements in ego-centric pretraining and the introduction of ego-centric datasets to enhance multimodal representation learning. It introduces a novel framework, the Hierarchical Multimodal Attention Network (HMAM). HMAM adopts a hierarchical approach to jointly learn audiovisual and textual representations, effectively capturing the sequence of events within a video and improving understanding across modalities.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Computer Science
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Alamri, Huda Abdulhadi D Abdulhadi
- Advisor dc:contributor.advisor
-
- Essa, Irfan
- Committee members dc:contributor.committeemember
-
- Chernova, Sonia
- Starner, Thad
- Ghanem, Bernard
Subjects
dc:subject × 4Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/75258