Back to search

Boston University

Understanding visual and linguistic content by structured representations

Abstract

dc:description.abstract

Understanding visual and linguistic content is essential in different vision and language tasks (e.g., text-to-image editing, article comprehension). While large-scale models like the GPT-Series or Stable Diffusion exhibit impressive performance in these areas, their effectiveness in certain domains can be limited due to an inaccurate or incomplete understanding of input content. For instance, in text-to-image editing, state-of-the-art models (e.g., Imagic, Dreambooth) typically interpret input images in a global manner. In this case, they may fail to fully capture the complex details of scenes involving multiple objects and their relationships. To spur research in this field, this dissertation explores the use of structured representations in visual and linguistic content, aiming for a comprehensive and precise understanding of semantic information. We first introduce scene-graph representations and attribute-excluded features to text-to-image editing, where scene graphs can indicate multiple object interactions and attribute-excluded features can disentangle desired content from text-irrelevant content. We then discuss the applications of structured representations in comprehending linguistic content. We show that splitting long articles into simpler segments (e.g., metadata, named entities, paragraphs) and processing articles based on these segments helps our models analyze articles more accurately. In addition, we present structured representations in broader applications including video-based movie genre analysis and fashion compatibility prediction. To summarize, this work discusses the use of structured representations to understand visual and linguistic content. When directly understanding the entire input is challenging, segmenting the input into simpler parts and processing these parts hierarchically can yield improved performance. The enhancements achieved through structured representations demonstrate their value across multiple applications, making this field worth further exploration.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhang, Zhongping
Advisor dc:contributor.advisor
  • Plummer, Bryan A.

Subjects

dc:subject × 1

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53122

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Zhang, Zhongping. Understanding visual and linguistic content by structured representations. doctoral thesis, Boston University, 2024. https://hdl.handle.net/2144/53122