Boston University
Understanding visual and linguistic content by structured representations
Abstract
dc:description.abstractUnderstanding visual and linguistic content is essential in different vision and language tasks (e.g., text-to-image editing, article comprehension). While large-scale models like the GPT-Series or Stable Diffusion exhibit impressive performance in these areas, their effectiveness in certain domains can be limited due to an inaccurate or incomplete understanding of input content. For instance, in text-to-image editing, state-of-the-art models (e.g., Imagic, Dreambooth) typically interpret input images in a global manner. In this case, they may fail to fully capture the complex details of scenes involving multiple objects and their relationships. To spur research in this field, this dissertation explores the use of structured representations in visual and linguistic content, aiming for a comprehensive and precise understanding of semantic information. We first introduce scene-graph representations and attribute-excluded features to text-to-image editing, where scene graphs can indicate multiple object interactions and attribute-excluded features can disentangle desired content from text-irrelevant content. We then discuss the applications of structured representations in comprehending linguistic content. We show that splitting long articles into simpler segments (e.g., metadata, named entities, paragraphs) and processing articles based on these segments helps our models analyze articles more accurately. In addition, we present structured representations in broader applications including video-based movie genre analysis and fashion compatibility prediction. To summarize, this work discusses the use of structured representations to understand visual and linguistic content. When directly understanding the entire input is challenging, segmenting the input into simpler parts and processing these parts hierarchically can yield improved performance. The enhancements achieved through structured representations demonstrate their value across multiple applications, making this field worth further exploration.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Zhang, Zhongping
- Advisor dc:contributor.advisor
-
- Plummer, Bryan A.
Subjects
dc:subject × 1Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53122