{"id":{"repo_id":"calpoly","oai_identifier":"oai:digitalcommons.calpoly.edu:theses-4887"},"canonical_url":"https://search.ndltd.org/etd/calpoly/oai:digitalcommons.calpoly.edu:theses-4887","repository":{"repo_id":"calpoly","name":"Cal Poly","base_url":"https://digitalcommons.calpoly.edu/do/oai/"},"display":{"title":"Johnson–Lindenstrauss Projections as Architectural Components in Deep Neural Networks","abstract":"Modern deep neural networks are heavily overparameterized, with learned representations occupying lower-dimensional subspaces than their ambient dimensions suggest. This thesis investigates whether fixed random Johnson-Lindenstrauss (JL) projections can exploit this redundancy by functioning as architectural bottlenecks that reduce model size and training cost while preserving task performance. We systematically evaluate JL-based compression across multiple architectures (convolutional networks, multilayer perceptrons, vision transformers, autoregressive language models) and tasks (reconstruction, classification, language modeling). Experiments reveal that compression tolerance depends critically on placement strategy and compression ratio. Linear probe analysis quantifies intrinsic dimensionality at different network depths, revealing that CIFAR-10 final representations require only 5-10 dimensions despite ambient dimensions of 256-512. Intermediate representations require approximately 50 dimensions to maintain transformation capacity. This 25-100x gap between intrinsic and ambient dimensionality explains why aggressive compression is possible and provides principled targets for layer-specific compression strategies. The success of completely random, non-learned projections in preserving task performance demonstrates that neural networks learn robust geometric structures where distance preservation is sufficient for representation quality. This work establishes random projections as viable architectural components and opens a design space for efficient neural networks through principled dimensionality reduction.","abstract_html":"Modern deep neural networks are heavily overparameterized, with learned representations occupying lower-dimensional subspaces than their ambient dimensions suggest. This thesis investigates whether fixed random Johnson-Lindenstrauss (JL) projections can exploit this redundancy by functioning as architectural bottlenecks that reduce model size and training cost while preserving task performance. We systematically evaluate JL-based compression across multiple architectures (convolutional networks, multilayer perceptrons, vision transformers, autoregressive language models) and tasks (reconstruction, classification, language modeling). Experiments reveal that compression tolerance depends critically on placement strategy and compression ratio. Linear probe analysis quantifies intrinsic dimensionality at different network depths, revealing that CIFAR-10 final representations require only 5-10 dimensions despite ambient dimensions of 256-512. Intermediate representations require approximately 50 dimensions to maintain transformation capacity. This 25-100x gap between intrinsic and ambient dimensionality explains why aggressive compression is possible and provides principled targets for layer-specific compression strategies. The success of completely random, non-learned projections in preserving task performance demonstrates that neural networks learn robust geometric structures where distance preservation is sufficient for representation quality. This work establishes random projections as viable architectural components and opens a design space for efficient neural networks through principled dimensionality reduction.","abstract_has_math":false,"creators":["Ray, Charles Q"],"institution":null,"degree_name":"MS in Computer Science","degree_level":null,"degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Paul Anderson","Computer Science","College of Engineering"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T07:12:51Z","subjects":["Deep learning","model compression","random projections","Johnson-Lindenstrauss"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/3229","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.calpoly.edu/theses/3229","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.calpoly.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.calpoly.edu%3Atheses-4887","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Paul Anderson","Computer Science","College of Engineering"]},{"key":"dc:creator","label":"Author","values":["Ray, Charles Q"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2026-03-10T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2026-03-01T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Computer Science"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Deep learning","model compression","random projections","Johnson-Lindenstrauss"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/3229"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Modern deep neural networks are heavily overparameterized, with learned representations occupying lower-dimensional subspaces than their ambient dimensions suggest. This thesis investigates whether fixed random Johnson-Lindenstrauss (JL) projections can exploit this redundancy by functioning as architectural bottlenecks that reduce model size and training cost while preserving task performance. We systematically evaluate JL-based compression across multiple architectures (convolutional networks, multilayer perceptrons, vision transformers, autoregressive language models) and tasks (reconstruction, classification, language modeling). Experiments reveal that compression tolerance depends critically on placement strategy and compression ratio. Linear probe analysis quantifies intrinsic dimensionality at different network depths, revealing that CIFAR-10 final representations require only 5-10 dimensions despite ambient dimensions of 256-512. Intermediate representations require approximately 50 dimensions to maintain transformation capacity. This 25-100x gap between intrinsic and ambient dimensionality explains why aggressive compression is possible and provides principled targets for layer-specific compression strategies. The success of completely random, non-learned projections in preserving task performance demonstrates that neural networks learn robust geometric structures where distance preservation is sufficient for representation quality. This work establishes random projections as viable architectural components and opens a design space for efficient neural networks through principled dimensionality reduction."]},{"key":"dc:title","label":"Title","values":["Johnson–Lindenstrauss Projections as Architectural Components in Deep Neural Networks"]}]}],"canonical_facts":{"dc:contributor":["Paul Anderson","Computer Science","College of Engineering"],"dc:creator":["Ray, Charles Q"],"dc:date.available":["2026-03-10T07:00:00Z"],"dc:date.created":["2026-03-01T08:00:00Z"],"dc:description.abstract":["Modern deep neural networks are heavily overparameterized, with learned representations occupying lower-dimensional subspaces than their ambient dimensions suggest. This thesis investigates whether fixed random Johnson-Lindenstrauss (JL) projections can exploit this redundancy by functioning as architectural bottlenecks that reduce model size and training cost while preserving task performance. We systematically evaluate JL-based compression across multiple architectures (convolutional networks, multilayer perceptrons, vision transformers, autoregressive language models) and tasks (reconstruction, classification, language modeling). Experiments reveal that compression tolerance depends critically on placement strategy and compression ratio. Linear probe analysis quantifies intrinsic dimensionality at different network depths, revealing that CIFAR-10 final representations require only 5-10 dimensions despite ambient dimensions of 256-512. Intermediate representations require approximately 50 dimensions to maintain transformation capacity. This 25-100x gap between intrinsic and ambient dimensionality explains why aggressive compression is possible and provides principled targets for layer-specific compression strategies. The success of completely random, non-learned projections in preserving task performance demonstrates that neural networks learn robust geometric structures where distance preservation is sufficient for representation quality. This work establishes random projections as viable architectural components and opens a design space for efficient neural networks through principled dimensionality reduction."],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/3229"],"dc:subject":["Deep learning","model compression","random projections","Johnson-Lindenstrauss"],"dc:title":["Johnson–Lindenstrauss Projections as Architectural Components in Deep Neural Networks"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_name":["MS in Computer Science"]},"updated_at":"2026-11-19T07:12:51Z"}