Back to search

University of British Columbia

Symmetric implicit geometries under label-imbalances : from cross-entropy to supervised contrastive loss

Abstract

dc:description

Class imbalance in machine learning has long been a significant research challenge. As models grow, reaching interpolation capacity (where models fully fit the training data), traditional methods like loss reweighting or data resampling become less effective. While new methods address imbalanced learning in the over-parameterized regime, their impact on deep learning is unclear. Recent studies reveal the Neural Collapse (NC) phenomenon in over-parameterized networks, where features from penultimate layers collapse into class-representative vectors. These vectors align with last layer classifiers, forming symmetric geometry with maximal class separation. Initially observed experimentally, NC has been analyzed theoretically using the Unconstrained Feature Model (UFM), focusing on the last layer and assuming networks produce optimized outputs, mainly on balanced datasets. In this thesis, we extend the UFM framework to analyze Cross-Entropy (CE) and Supervised Contrastive Loss (SCL) in imbalanced data. We introduce the SELI framework to characterize classifier/feature geometry under CE loss, explaining symmetric geometry on balanced datasets and predicting geometry in imbalanced cases. This leads to our identification of the Minority Collapse phenomenon, where minority class classifiers collapse, reducing margins and classification performance. We explore logit-adjusted CE loss variations that improve accuracy by favoring minority margins. We study resulting geometries and propose optimal hyperparameters for achieving symmetric geometries, enhancing minority margins. For SCL loss, adding a non-negative activation function like ReLU restores symmetry in geometry regardless of imbalance. Our analysis reveals batching's importance in contrasting features, leading to minimum requirements for NC feature collapse. We suggest batching strategies aiding SCL convergence to symmetric geometry and propose using prototype vectors to fine-tune geometry in SCL models without non-negativity constraints. Our findings are supported by extensive experiments on standardized vision datasets.

Degree

thesis:*
Name thesis:degree_name
Master of Applied Science - MASc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Electrical and Computer Engineering
Grantor dc:publisher
University of British Columbia
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Vakilian, Vala
Contributors dc:contributor
  • Thrampoulidis, Christos

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/89113

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Vakilian, Vala. Symmetric implicit geometries under label-imbalances : from cross-entropy to supervised contrastive loss. master's thesis, University of British Columbia, 2024. http://hdl.handle.net/2429/89113