Back to search

Cal Poly

Johnson–Lindenstrauss Projections as Architectural Components in Deep Neural Networks

Abstract

dc:description.abstract

Modern deep neural networks are heavily overparameterized, with learned representations occupying lower-dimensional subspaces than their ambient dimensions suggest. This thesis investigates whether fixed random Johnson-Lindenstrauss (JL) projections can exploit this redundancy by functioning as architectural bottlenecks that reduce model size and training cost while preserving task performance. We systematically evaluate JL-based compression across multiple architectures (convolutional networks, multilayer perceptrons, vision transformers, autoregressive language models) and tasks (reconstruction, classification, language modeling). Experiments reveal that compression tolerance depends critically on placement strategy and compression ratio. Linear probe analysis quantifies intrinsic dimensionality at different network depths, revealing that CIFAR-10 final representations require only 5-10 dimensions despite ambient dimensions of 256-512. Intermediate representations require approximately 50 dimensions to maintain transformation capacity. This 25-100x gap between intrinsic and ambient dimensionality explains why aggressive compression is possible and provides principled targets for layer-specific compression strategies. The success of completely random, non-learned projections in preserving task performance demonstrates that neural networks learn robust geometric structures where distance preservation is sufficient for representation quality. This work establishes random projections as viable architectural components and opens a design space for efficient neural networks through principled dimensionality reduction.

Degree

thesis:*
Name thesis:degree_name
MS in Computer Science
Discipline thesis:degree_discipline
Computer Science

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ray, Charles Q
Contributors dc:contributor
  • Paul Anderson
  • Computer Science
  • College of Engineering

Subjects

dc:subject × 4

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.calpoly.edu/theses/3229

Chain of custody

source
Harvested from
Cal Poly
Base URL
digitalcommons.calpoly.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Ray, Charles Q. Johnson–Lindenstrauss Projections as Architectural Components in Deep Neural Networks. https://digitalcommons.calpoly.edu/theses/3229