Back to search

Georgia Institute of Technology

Augmenting Large Language Models for Situated Multimodal Conversations in Scientific Documents

Abstract

dc:description.abstract

There has been a rapid increase in the number of scientific articles published in recent times, particularly in the field of computing. For example, the number of electronic pre-prints submitted to ArXiv has increased 30-fold in the last 30 years. Simultaneously, staying current with peer-reviewed work is essential for scientific research. To address this issue, this dissertation outlines research done towards the development of language-model-based methods for situated and multimodal interactive conversations in scientific documents. First, the problem is broken down into five research threads based on the taxonomy of research in multimodal machine learning. These are multimodal representation, translation, fusion, alignment, and co-learning. Next, proofs-of-concept for individual research threads are developed and their efficacy is demonstrated on standard benchmark datasets. Finally, a new dataset called Conversational Papers is collected and open-sourced specifically for situated and multimodal interactive conversations in scientific documents. A series of baseline methods are presented to address the dataset and the overall challenge of situated and multimodal interactive conversations in scientific documents.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sundar, Anirudh
Advisor dc:contributor.advisor
  • Heck, Larry
Committee members dc:contributor.committeemember
  • Anderson, David
  • Muthukumar, Vidya
  • Davenport, Mark
  • Xu, Wei

Subjects

dc:subject × 4

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/77846

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Sundar, Anirudh. Augmenting Large Language Models for Situated Multimodal Conversations in Scientific Documents. Doctoral thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/77846