Back to search

University of British Columbia

Multimodal understanding of long documents : from topic modeling to question answering

Abstract

dc:description

Long multimodal documents, which contain text, images, and other types of content, are common in real-world settings but remain difficult for natural language processing (NLP) models to process. These documents pose challenges in both understanding their content and training models when labeled data is limited. This thesis presents two contributions that address these problems from different angles. First, we introduce CEMTM, a topic modeling method designed for long documents that include both text and images. Instead of relying on bag-of-words or treating different modalities separately, CEMTM uses contextual embeddings and cross-modal alignment to produce more coherent and meaningful topics. It performs well across several datasets and offers better topic diversity and interpretability. Second, we present FM²DS, a pipeline for generating synthetic training data for multimodal multihop question answering (MMQA). FM²DS uses prompting and document retrieval to create realistic question answering (QA) examples, and applies knowledge distillation to transfer reasoning ability from a large teacher model to a smaller multimodal model. This approach makes it possible to train competitive QA systems with only a few examples, reducing the need for large annotated datasets. Together, these two methods support more effective processing of long multimodal documents: CEMTM for exploring and summarizing content, and FM²DS for enabling downstream MMQA systems in low-resource settings. We evaluate both approaches across multiple tasks and demonstrate a substantial performance improvement.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Abaskohi, Amirhossein
Contributors dc:contributor
  • Carenini, Giuseppe

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/92096

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Abaskohi, Amirhossein. Multimodal understanding of long documents : from topic modeling to question answering. master's thesis, University of British Columbia, 2025. http://hdl.handle.net/2429/92096