Back to search

University of Waterloo

Approaches and Techniques to Enhance Efficiency and Performance of Non-Contrastive Self-Supervised Learning Methods

Abstract

dc:description.abstract

Self-supervised learning (SSL) methods have gained considerable attention in recent years due to their ability to learn useful representations of data without relying on labels during training. These methods have revolutionized research across various domains, including Natural Language Processing, Computer Vision, and Graph Deep Learning. SSL methods can be classified into three categories: Generative, Predictive, and Dual-Encoder. Among these, Dual-Encoder methods have gained significant popularity in many applications as, in contrast with generative methods, they do not require training a powerful decoder. Also, in contrast with predictive SSL methods, they do not need a careful design of the pre-training task. However, Dual-Encoder techniques suffer from collapse in representation space, since mapping all the samples into the same point in the embedding space is the trivial solution to their optimization problem. Based on their mechanisms for preventing this issue, those can be further divided into two classes: Contrastive and Non-Contrastive. Contrastive techniques prevent representation collapse using negative sampling, while Non-Contrastive techniques employ asymmetries such as stop-gradient or information maximization through Covariance/Cross-Covariance matrices whitening. This thesis aims to enhance two classes of Non-Contrastive Dual-Encoder SSL methods - Asymmetry-Based and Covariance-Based. The proposed improvements are as follows: • Covariance-Based methods: This thesis proposes techniques to enhance the effi- ciency and performance of Covariance-Based methods. Specifically, the invariance loss term’s efficiency is improved through various data sampling techniques, and the covariance whitening term’s efficiency is enhanced through random dimension selection, LSH bucketing, Nystrom approximation, and random projection. These techniques are thoroughly tested and validated. • Asymmetry-Based methods: This thesis enhances the performance of Asymmetry- Based methods by preventing rank degradation and early termination of the update procedure for the target network. Overall, this thesis proposes novel techniques to enhance the efficiency and performance of Non-Contrastive Dual-Encoder SSL methods. Proposed ideas tested and validated for benchmark static graph datasets. However, those are applicable to other applications and modalities as well.

Degree

thesis:*
Grantor dc:publisher
University of Waterloo
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Saheb Pasand, Ali
Advisor dc:contributor.advisor
  • Ghodsi, Ali

Subjects

dc:subject × 2

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/10012/19244

Chain of custody

source
Harvested from
University of Waterloo
Base URL
dspacemainprd01.lib.uwaterloo.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Saheb Pasand, Ali. Approaches and Techniques to Enhance Efficiency and Performance of Non-Contrastive Self-Supervised Learning Methods. University of Waterloo, 2023. http://hdl.handle.net/10012/19244