Back to search

Georgia Institute of Technology

Information dynamics of self supervised learning

Abstract

dc:description.abstract

Self-supervised learning (SSL) refers to a family of training strategies that allow task-agnostic representation learning across multifarious data modalities. This ability of SSL algorithms to overcome the need for task-specific empirical risk minimization is key for the construction of billion-parameter foundation models. The general premise behind SSL is to pre-train a backbone (foundation) model on a large corpus of unlabeled data with the intent of forming robust representations that are transferable to potentially any downstream task. Despite the empirical success of these methods, current SSL algorithms still face numerous generalization challenges. This includes specific issues such as poor transferability to data settings with localized fine-grained features, degradation of features over long training horizons, and reduced performance due to the presence of noise or other domain-specific data imbalances. In this dissertation, we argue that these generalization challenges are largely a result of the suboptimal design of standard SSL optimization objectives. Current SSL algorithms utilize a fixed objective throughout training that does not adapt to the emergence of training dynamics unique to individual data settings. In general, very few works have analyzed the underlying training dynamics that influences the formation of SSL representations. Additionally, these works have not explored how to exploit these training dynamics in order to inform the design of SSL optimization algorithms. The goals of this dissertation are to 1) identify the components that reflect quality SSL representation spaces, 2) use these components to characterize SSL training dynamics in terms of distinct phases based on local and global relationships as well as info-theoretic representational dynamics, 3) utilize these insights to develop SSL algorithms that adapt to these identified dynamics in a dataset specific manner, and 4) demonstrate that our developed SSL algorithms result in state of the art performance across a wide range of application settings that include medicine, autonomous vehicle sensors, subsurface imaging, and large scale natural image datasets.

Degree

thesis:*
Name thesis:degree_name
Electrical and Computer Engineering, PhD
Department thesis:degree_department
Electrical and Computer Engineering
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Kokilepersaud, Kiran
Advisor dc:contributor.advisor
  • AlRegib, Ghassan
Committee members dc:contributor.committeemember
  • Wang, May
  • Aghazadeh Mohandesi, Amirali
  • Davenport, Mark
  • Frakes, David
ORCID person:identifier.orcid

Rights

Language dc:language.iso
English

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81692

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Kokilepersaud, Kiran. Information dynamics of self supervised learning. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/81692