Back to search

Georgia Institute of Technology

Mathematical Guarantees of Near-Optimal Cross-Entropy Loss through Neural Collapse

Abstract

dc:description.abstract

Neural Collapse is a recently observed geometric structure that emerges in the final layer of neural network classifiers. Specifically, Neural Collapse states that at the terminal phase of neural networks training, 1) the intra-class variability of last layer features tends to zero, 2) the class feature means form a simplex Equiangular Tight Frame (ETF) up to scaling, 3) last-layer class features and weights becomes equal up the scaling, and 4) classification behavior collapses to the nearest class center (NCC) decision rule. This thesis investigates the emergence of Neural Collapse for unbiased neural network classifiers when the training cross-entropy loss is near-optimal. Theoretically, we show that, within a small neighborhood of the optimal cross-entropy loss, with $C$ target classes, neural network classifiers with batch normalization and weight decay achieve intra-class feature cosine similarity near one and inter-class cosine similarity near $-\frac{1}{C-1}$, which justifies multiple observations of Neural Collapse under realistic conditions guaranteed by neural network training. Our theorems also imply that higher weight decay values imply more significant Neural Collapse and that batch normalization with affine transformation along with weight decay is critical to the emergence of Neural Collapse, which is supported by experiments. We also empirically investigate the effect of various parameters on the inter- and intra-class cosine similarity of representations obtained.

Degree

thesis:*
Level thesis:degree_level
Masters
Department dc:contributor.department
Computer Science
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Pan, Leyan
Advisor dc:contributor.advisor
  • Vempala, Santosh S.
Committee members dc:contributor.committeemember
  • Singla, Sahil
  • Chen, Yongxin

Subjects

dc:subject × 2

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/72092

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Pan, Leyan. Mathematical Guarantees of Near-Optimal Cross-Entropy Loss through Neural Collapse. Masters thesis, Georgia Institute of Technology, 2023. https://hdl.handle.net/1853/72092