Georgia Institute of Technology
Robustness under distribution shifts in computer vision
Abstract
dc:description.abstractIn conventional machine learning, the test distribution is often assumed to be the same as the training distribution. However, this assumption has become less justified in the deep learning era. Deep learning models have been increasingly deployed in the real world, in which the distribution of the input data can be very different from that of the training data. For example, a model is more likely to predict instances seen more often in its training data, leading to statistically biased priors (prior shifts); a model can perform drastically worse if the input changes in its appearance slightly and still be very confident in wrong predictions (covariate shifts). In this thesis, we study the causes of deep learning models' suboptimal performance under these two distribution shifts and propose mitigation strategies to improve their robustness to the shifts. Prior Shifts: improving recall without sacrificing precision. Prior shifts can arise when i) the training class distribution does not match the true (test) class distribution due to the inherent difficulty in obtaining samples from certain classes, or ii) the distribution of classes does not reflect their relative importance. For example, the natural world is inherently imbalanced such that a collected dataset may exhibit a long-tailed distribution. The iNaturalist dataset, an image collection of 8,142 species, is an imbalanced dataset, with an imbalance ratio of 500. Similarly, in semantic segmentation, the pixel percentage of pedestrians does not reflect their importance in prediction, e.g., for safety. The imbalance ratio can be as large as 1×10^4 between the largest and the smallest categories, such as in the Sythia self-driving dataset. However, naively correcting the imbalance will produce excessive false positive predictions, i.e., improving the recall but sacrificing the precision of small classes. This thesis studies in-training optimization and post-training calibration to improve recall while maintaining precision. Covariate Shifts: learning the new without forgetting the old. Covariate shifts can happen when i) an input image is corrupted by noise such as Gaussian noise in low-light photographyand ii) when the same object appears in an unfamiliar environment or a different variation from the training samples such as paintings and real photos of animals. For example, users can ask a classification model trained from real images of cats to classify a sketch of cats. Similarly, a segmentation neural network can encounter weather conditions not seen during training. Improving the robustness of deep learning models against covariate shifts is critical for safety and reliability reasons. To obtain this robustness, one promising approach is to inherit robust knowledge from a foundation model pre-trained on a large dataset by fine-tuning. However, fine-tuning such models to other tasks often results in losing the pre-trained robustness as the new model overfits the new data and forgets the pre-trained features. This thesis studies constrained fine-tuning strategies to preserve the robustness acquired by the foundation model when fine-tuning it to downstream tasks.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Electrical and Computer Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tian, Junjiao
- Advisor dc:contributor.advisor
-
- Kira, Zsolt
- Committee members dc:contributor.committeemember
-
- Hoffman, Judy
- Garg, Animesh
- Oh, Jean
- Tran, Dustin
Subjects
dc:subject × 2Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/75665