Back to search

Boston University

Overcoming visual data scarcity with synthetic examples

Abstract

dc:description.abstract

It is not news that deep networks with large numbers of parameters can easily fit to visual recognition tasks on large labeled datasets and generalize reasonably well. This task however becomes hard with few training data. Also challenging are scenarios where generalization is expected across test distributions that are further apart or more diverse than train. An example of this may be learning to recognize objects in different styles of images : paintings, line-drawings, real images etc. having only seen them in any single one of these modalities during training. A partial panacea to lack of plentiful training data from the target distribution has been pre-training the weights of the deep model on large image datasets curated from the web. This has its own expenses in labeling costs and violation of copyrights and privacy. Additionally, it is not always well suited to all target tasks. In this thesis, through a range of such problems with a shortage of ample training data, we attempt to make up for them via synthetically generated examples. We first investigate a common problem of learning across domains with a source domain (or distribution) with many labeled training examples and a target domain, where only a few labeled training examples but several unlabeled examples are available. Commonly referred to as semi-supervised domain adaptation (SSDA), this problem has previously seen solutions aligning output features across domains. We demonstrated that foregoing explicit feature alignment, the lack of training data could be effectively tackled in this scenario by pre-training on artificial image orientation labels, and using label-preserving image transforms based on standard heuristics as augmentations to available data. Next, we tackle the challenges of pre-training on a large labeled image dataset curated from the web, such as ImageNet. To overcome the annotation costs and potential privacy/copyright violations previously mentioned, we replaced this data with images from a graphics simulator. In doing so, we also leveraged the additional control in parameters like lighting conditions, object materials, camera positions, etc. that the simulator provides to optimize them for a particular downstream task. For efficiency in this optimization, we learned a meta-network called Task2Sim, that could generate values for these parameters given a target task representation. In the next part of the thesis, we delve into foundation models, which are trained on web scale data allowing them to incorporate a major portion of all visual concepts known to humans. We first show that in a scenario of extreme data scarcity, synthetic examples from text-to-image generative models can help learn effective representations for cross-domain retrieval. In the final chapter, we show that multimodal large language model capabilities can be enhanced with preference tuning on synthetic data. In a fully automated manner with large language models, we can generate hard negative captions from existing image-caption data, that can be used to teach MLLMs concepts involved in compositional reasoning.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Mishra, Samarth
Advisors dc:contributor.advisor
  • Saenko, Kate
  • Saligrama, Venkatesh
ORCID dc:identifier.orcid

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53251

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Mishra, Samarth. Overcoming visual data scarcity with synthetic examples. doctoral thesis, Boston University, 2025. https://hdl.handle.net/2144/53251