Boston University
From coarse to fine-grained concept based discrimination for phrase detection
Abstract
dc:description.abstractPhrase Detection is a vision and language task where the goal is to determine if a phrase is relevant to an image and localize it, if applicable. The task has many important downstream applications such as Assistive Robotics where a Robot needs to detect/localize an object in an environment. However, training discriminative Phrase Detection models is difficult due to two main challenges 1) sparse training labels: ground truth regions are not exhaustively labeled with all applicable phrases which makes determining negative phrases challenging 2) the training distribution is heavily imbalanced; a small subgroup of the phrases constitute most of the training data. We address these problems through a novel coarse style discrimination: Negative Coarse Concepts (NCC). The method involves grouping visually coherent phrases (concepts) and using the concepts as negative samples which effectively expose the model to a wide and diverse portion of the distribution while minimizing the chance of false negatives. Furthermore, we supplement the coarse discrimination method with a fine grained module (FGM) which effectively discriminates between mutually exclusive tokens of diverse groups such as colors, sizes, etc. Finally, we combine these two novel modules into one model: CFCD-Net, which improves the state-of-the-art performance of Phrase Detection models on Flickr30K Entities and RefCOCO+ Datasets by 1.5-2 points. We further demonstrate how these improvements directly translate to improvements in the downstream task of Binary Image Selection (BISON).
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Qraitem, Maan
- Advisor dc:contributor.advisor
-
- Plummer, Bryan A.
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/48442