Back to search

Georgia Institute of Technology

Extracting Knowledge with Multimodal and Multilingual Intelligent Systems

Abstract

dc:description.abstract

Recent advancements in large language models (LLMs) have revolutionized natural language processing and vision-language tasks. While demonstrating emergent capabilities, these models present challenges in responsible development, particularly in visual world knowledge, privacy concerns, and multilingual capabilities. This thesis addresses these challenges through three main contributions. First, we introduce InfoSeek, a vision-language benchmark assessing models’ ability to leverage world knowledge for answering queries about visual entities. To improve performance on this challenging task, we develop multimodal retrieval-augmented generation systems to acquire knowledge from external resources. Inspired by the visual knowledge we found from InfoSeek, we raise an emergent privacy concern of multimodal LLM to reveal geolocation information of user posted images. We then present PrivQA, a benchmark evaluating models’ ability to follow access control instructions and prevent private information disclosure. Our findings reveal biases and vulnerabilities in current privacy protection mechanisms, especially in adversarial settings. Third, we propose three approaches to enhance multilingual capabilities and improve information extraction (IE) in low-resource languages: TransFusion, a framework leveraging English translations to enhance multilingual performance; EasyProject, a simplified method for creating synthetic multilingual IE data; and a model selection algorithm predicting multilingual model performance on unseen languages. These contributions aim to develop and benchmark methods for extracting knowledge with multimodal and multilingual intelligent systems, addressing key challenges in emergent visual knowledge, privacy concerns, and multilingual capabilities.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Computer Science
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chen, Yang
Advisors dc:contributor.advisor
  • Ritter, Alan
  • Xu, Wei
Committee members dc:contributor.committeemember
  • Goyal, Kartik
  • Hu, Hexiang
  • Chang, Ming-Wei

Subjects

dc:subject × 3

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/75688

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Chen, Yang. Extracting Knowledge with Multimodal and Multilingual Intelligent Systems. Doctoral thesis, Georgia Institute of Technology, 2024. https://hdl.handle.net/1853/75688