Back to search

Georgia Institute of Technology

Robustness under distribution shifts in computer vision

Abstract

dc:description.abstract

In conventional machine learning, the test distribution is often assumed to be the same as the training distribution. However, this assumption has become less justified in the deep learning era. Deep learning models have been increasingly deployed in the real world, in which the distribution of the input data can be very different from that of the training data. For example, a model is more likely to predict instances seen more often in its training data, leading to statistically biased priors (prior shifts); a model can perform drastically worse if the input changes in its appearance slightly and still be very confident in wrong predictions (covariate shifts). In this thesis, we study the causes of deep learning models' suboptimal performance under these two distribution shifts and propose mitigation strategies to improve their robustness to the shifts. Prior Shifts: improving recall without sacrificing precision. Prior shifts can arise when i) the training class distribution does not match the true (test) class distribution due to the inherent difficulty in obtaining samples from certain classes, or ii) the distribution of classes does not reflect their relative importance. For example, the natural world is inherently imbalanced such that a collected dataset may exhibit a long-tailed distribution. The iNaturalist dataset, an image collection of 8,142 species, is an imbalanced dataset, with an imbalance ratio of 500. Similarly, in semantic segmentation, the pixel percentage of pedestrians does not reflect their importance in prediction, e.g., for safety. The imbalance ratio can be as large as 1×10^4 between the largest and the smallest categories, such as in the Sythia self-driving dataset. However, naively correcting the imbalance will produce excessive false positive predictions, i.e., improving the recall but sacrificing the precision of small classes. This thesis studies in-training optimization and post-training calibration to improve recall while maintaining precision. Covariate Shifts: learning the new without forgetting the old. Covariate shifts can happen when i) an input image is corrupted by noise such as Gaussian noise in low-light photographyand ii) when the same object appears in an unfamiliar environment or a different variation from the training samples such as paintings and real photos of animals. For example, users can ask a classification model trained from real images of cats to classify a sketch of cats. Similarly, a segmentation neural network can encounter weather conditions not seen during training. Improving the robustness of deep learning models against covariate shifts is critical for safety and reliability reasons. To obtain this robustness, one promising approach is to inherit robust knowledge from a foundation model pre-trained on a large dataset by fine-tuning. However, fine-tuning such models to other tasks often results in losing the pre-trained robustness as the new model overfits the new data and forgets the pre-trained features. This thesis studies constrained fine-tuning strategies to preserve the robustness acquired by the foundation model when fine-tuning it to downstream tasks.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tian, Junjiao
Advisor dc:contributor.advisor
  • Kira, Zsolt
Committee members dc:contributor.committeemember
  • Hoffman, Judy
  • Garg, Animesh
  • Oh, Jean
  • Tran, Dustin

Subjects

dc:subject × 2

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/75665

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Tian, Junjiao. Robustness under distribution shifts in computer vision. Doctoral thesis, Georgia Institute of Technology, 2024. https://hdl.handle.net/1853/75665